NESTED LEARNING FOR MULTI-GRANULAR TASKS
Authors: Raphaël Achddou
J. Matias Di Martino
Guillermo Sapiro
Venue:  (2020)
Keywords: Nested learning

Abstract: Standard deep neural networks (DNNs) used for classification are trained in an end-to-end fashion for very specific tasks - object recognition, face identification, character recognition, etc. This specificity often leads to overconfident models that generalize poorly to samples that are not from the original training distribution. Moreover, they do not allow to leverage information from heterogeneously annotated data, where for example, labels may be provided with different levels of granularity. Finally, standard DNNs do not produce results with simultaneous different levels of confidence for different levels of detail, they are most commonly an all or nothing approach. To address these challenges, we introduce the problem of nested learning: how to obtain a hierarchical representation of the input such that a coarse label can be extracted first, and sequentially refine this representation to obtain successively refined predictions, all of them with the corresponding confidence. We explicitly enforce this behaviour by creating a sequence of nested information bottlenecks. Looking at the problem of nested learning from an in formation theory perspective, we design a network topology with two important properties. First, a sequence of low dimensional (nested) feature embeddings are enforced. Then we show how the explicit combination of nested outputs can improve both robustness and finer predictions. Experimental results on CIFAR-10, MNIST, and FASHION-MNIST demonstrate that nested learning outperforms the same network trained in the standard end-to-end fashion. Since the network can be naturally trained with mixed data labeled at different levels of nested details, we also study what is the most efficient way of annotating data, when a fixed training budget is given and the cost of labels increases with the levels in the nested hierarchy.

Under review as a conference paper at ICLR 2020
NESTED LEARNING FOR MULTI-GRANULAR TASKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Standard deep neural networks (DNNs) used for classification are trained in an end-to-end fashion for very specific tasks object recognition, face identification, character recognition, etc. This specificity often leads to overconfident models that generalize poorly to samples that are not from the original training distribu- tion. Moreover, they do not allow to leverage information from heterogeneously annotated data, where for example, labels may be provided with different levels of granularity. Finally, standard DNNs do not produce results with simultaneous dif- ferent levels of confidence for different levels of detail, they are most commonly an all or nothing approach. To address these challenges, we introduce the problem of nested learning: how to obtain hierarchical representation of the input such that a coarse label can be extracted first, and sequentially refine this representa- tion to obtain successively refined predictions, all of them with the corresponding confidence. We explicitly enforce this behaviour by creating a sequence of nested information bottlenecks. Looking at the problem of nested learning from in- formation theory perspective, we design a network topology with two important properties. First, sequence of low dimensional (nested) feature embeddings are enforced. Then we show how the explicit combination of nested outputs can im- prove both robustness and finer predictions. Experimental results on CIFAR- 10, MNIST, and FASHION-MNIST demonstrate that nested learning outperforms the same network trained in the standard end-to-end fashion. Since the network can be naturally trained with mixed data labeled at different levels of nested details, we also study what is the most efficient way of annotating data, when fixed train- ing budget is given and the cost of labels increases with the levels in the nested hierarchy.
INTRODUCTION
Deep learning is providing remarkable computational tools for the automatic analysis and under- standing of complex high-dimensional problems [Esteva et al. (20 SSanhhl et al. (2016); Parkhi J. Despite its tremendous value and versatility, methods based on Deep Neural Networks (DNNs) tend to be overconfident about their predictions and limited to the task and data they have been trained on (Hein et al.) (2018); Guuo et al. (2017): Nguyen et al. (2014)). This happens, among other reasons, because the standard approach to train DNN models consists in optimizing its perfor- mance over a specific dataset and for specific task in an end-to-end fashion /Szegedy et al.) (2014). Standard DNNs are not designed to be trained with data of different quality and to simultaneously provide results at multiple granularities.
al. (2015))
Take as an example the case illustrated in Figure I (left); for high quality facial images, we may be able to infer the person's age group and identity; whereas for poor resolution or occluded examples, only sub-set of these nested predictions may be achievable. We expect the network to automatically understand what can and cannot predict, and this obtained with the framework proposed in this paper (Figure [] right). Moreover, nested learning allows us to leverage training information from diverse datasets, with varying granularity and quality of labels, and combine this information into single model.
On the other hand, when heterogeneous data with different quality and granularity of annotations (as in the example illustrated in Figure [] (left)) is provided for training, low quality samples with coarse labels can help us to understand the structure of the coarser distributions (person, under 50) while simultaneously data with finer labels can contribute to the coarse and fine tasks. This will be formalized later in the paper with tools from information theory.

---

Under review as a conference paper at ICLR 2020
Our prediction:
Vehicle
High
Car/Truck
High
Truck
High
Standard DNN:
Truck
High
Our prediction:
99%
99%
70%
Vehicle
High
Under 50
90%
Under 50
95%
Under 50
60%
Low
Car/Truck
High
Under 90%
Under
70%
Low
Truck
John B5%
Low
Low
Low
Standard DNN:
Common EXVNs beshauker
Destred behazior
Low
John
90%
Martin
9996
Rephael
99%
George
95%
Figure 1: On the left, an illustration of set nested predictions and their associated confidence given an input image of a face. The top block illustrates a desired behavior. Depending on the quality of the input data, one may be able to provide up to a certain level of prediction. This nested learning is the problem addressed in this paper. The bottom block illustrates how standard DNN-based models behave when they trained in end-to-end fashion to perform specific tasks such as face recognition. Clearly the traditional network over-confident in its predictions (potentially wrong for the last cases), and provides an all or nothing response instead of responding only what it can for the given input quality. On the right, we see a real example with results from the proposed framework; while a sharp image gets all the nested levels with high confidence in our proposed system, a low-quality one is getting the first two levels with confidence, while the finer one is correct but low confidence as expected. (Additional examples are presented in Figure in the supplementary material.)
Recently, Alsallakh et al. (2017) showed that convolutional neural networks (CNNs) naturally tend to learn hierarchical high-level features that discriminate groups of classes in the early layers, while the deeper layers develop more specialized feature detectors. We explicitly enforce this behaviour by creating sequence of nested information bottlenecks. Looking at the problem of nested learning from an information theory perspective, we design a network topology with two important prop- erties. First, sequence of low dimensional (nested) feature embeddings are enforced for each level in the taxonomy of the labels. This encourages generalization by foreing information bottle- necks (Tishby et al. 1999), SSwartz-ZZv & Tishby (017)]. Second, skipped connections allow finer embeddings to access information of the input that may be useful for finer classification but not in- formative on coarser categories Ronneberger et al. (2015). Additionally, we show how the explicit calibration and combination of nested outputs can improve the finer predictions and improve robust- ness. Finally, having the flexibility of merging data with different levels of granularity inspired us to study which is the most efficient way of annotating data given fixed budget that takes into account that more detailed training data is more expensive to annotate. The source code associated to this work is open source
The main contributions of this paper are: (I) We introduce the concept of explicit nested learning, where a given level in the hierarchy strictly contains the previous one and strictly adds information; (2) We provide a deep learning architecture that can be trained with data from all levels of the nested structure (all levels of labels and data quality), each one affecting the corresponding component of the network; (3) We provide a model with multiple outputs, one per level of the nested hierar- chy, each one with its own confidence; the user does not need know the "quality" of the data beforehand, the output confidences provide that information.
2 RELATED WORK
The problem of adapting DNNs models and training protocols encourage nested learning shares similarities with other popular problems in machine learning such as Multi-Task Learning (MTL). Though our work is related to MTL because of the similar training challenges, most of the MTL
methods tackle the task in a parallel way, without imposing a task hierarchy the architecture [Ranjan et al. (2016), Kokkinos (2017), Bilen & Vedaldi (2016)] The idea of learning hierarchical representations to improve classification performance has been exploited prior the proliferation of
DNNs, e.g.. [Zweig& & Weinshall (2007), Fergus et al. (2010), Zhao et al. (201T). Liu et al. (201
Some of these ideas have been incorporated into deep learning methods [Alsalakh et al. (2017).
Wehrmann et al. (2018), [eng et al. (2014), (Srivastava & Salakhutdinov (2013)]. and exploited in specific applications (Clar et al. (201)), Xuehong Mao et al. (2016). , Seo & shik Shin 2019))
2

---

Under review as a conference paper at ICLR 2020
Kim et al. (2018) proposed nested sparse network architecture with the emphasis on having a resource-awes aware versatile architecture to meet (simultaneously) diverse resource requirements. Wehrmann et al. (2018) proposed neural network architecture capable of simultancously opti- mizing local and global loss functions to exploit local and global information while penalizing hier- archical violations. Triguero & Vens (2016) investigated different alternatives to label hierarchical multi-label problems by selecting one or multiple thresholds to map output scores to hicrarchical pre- dictions, focusing on performance measures such the H-loss, HMC-loss and the micro mivro-averaged F-measure. Yan et al. (2015) introduced hierarchical deep CNNs (HD-CNNs) which consists of em- bedding CNNs into a two-level category hierarchy. They propose to distinguish a coarse class using an initial classifier and then refine the classification into a second level for each individual coarse category.
Although the works listed above are important, relevant, and related the work presented here. there are notable differences between them and what we propose. For example, while Kim et al.) (2018) propose a nested architecture providing different (potentially nested) outputs, they do not study how to combine these outputs into refined single prediction, nor provide a reliable confidence measure associate them. Furthermore, the architecture they propose has key differences with ours, while they propose a nested hierarchy in a end-to-end fashion (i.e., features associated to the coarse and fine levels are shared from the top the bottom of the network), we enforce sequential information bottleneck. As we show in the following sections this sequence of coarse to fine low dimensional representations facilitate robust calibration and combination of nested outputs. Yan iet al. (2015) study the problem of nested learning for two nested levels of granularity, and optimize for a final fine prediction. Their design is specific for a two-level category hierarchy while our work generalize to any number of nested levels, and we simultancously can train and test up to an arbitrary level of gramularity. Another important difference is that the focus of their work is on the implementation details and performance of their two-hierarchy network versus traditional end-to-end learning. As they mention in the conclusion of their work, future work should aim to extend their ideas to more than two hierarchical levels and to contextualize their empirical results into a theoretical framework. Our work takes steps these two specific directions. This is also fundamental difference with the works developed by Triguero & Vens (2016) and Wehrmann et al. (0118). Moreover, all these approaches suppose that every sample is annotated for all the granularity levels. In contrast with them, we can efficiently train (and predict) our model on dataset that provide only coarse labels, intermediate levels, or fine labels. Our approach is therefore more general in that regard. Also, we show that if testing conditions shift from the ones on training. we can still provide relatively confident coarser labels while avoiding overconfident (erroneous) fine predictions (see, e.g., Figure [ right; and Figure ij the supplementary material). Also, the questions of neural network overconfidence and output readability are never addressed in those works, which is crucial for both results interpretation, and output combination. Finally, because our solution can leverage information from datasets with different granularity, we are able analyze how different proportions of training coarse and fine data affects models cost, performance and robustness.
3 NESTED LEARNING
Let us assume we want to classify the popular hand written digits of MNIST & Cortes (2010) An input image can be represented as realization x of the random variable X We denote as X the alphabet of X. Associated to x, there is a label y that corresponds to the actual number the person writing the character wanted represent. The label y is realization of the random variable Y. In this illustrative case, Y can take 10 different values: ) = {0, 1...., 9}. Of course Y and X are not independent random variables. Generally, Y precedes X, and the problem of classification can be stated as the problem of inferring y from observed sample x. i.e., Y X + Ý. Y denotes new random variable (estimated from X) which approximate Y. More precisely, a common practice is to find a mapping X + Y such that the probability P(Ý + Y|X) is minimized.
Nested classification. In this work focus on the inherent hierarchical structure most classification problems have. For example, imagine now that we have hand written characters including numbers, lower case letters, and capital letters. It would be intuitive to first attempt classify these characters into three categories: numbers, lower case letters, and capital letters. Then, depending on this coarse
Cappttll letters will be used to denote random variables and lower case letters to denote the value of particular realization.
3

---

Under review as a conference paper at ICLR 2020
classification we can perform finer classification, i.e., classifying the numbers into 0 9 classes, the letters into a 2, and so forth.
Of course, we could have an arbitrary number of nested random variables associated to different levels of labels granularity. Here, subscripts indicate the granularity of the label, for example, Y;-1 is the closest coarse level of Y, Y represents the random variable associated to each A value in the closest coarse node, i.e., Y," represents Y; given that y:-1 k, k E V
Definition 3.1. We define Yi,..., Yn as a discrete sequence of nested labels if (YY = 0
Vi [1, n 1]. H denotes the standard definition of entropy for discrete random variables.
Definition 3.2. A discrete sequence of nested labels Y1, Yn is strictly nested if HY;((V-1)) < HYY)) Vi n).
X
Y1
R(G(N)
HOSIX)
Y22
Y2
Y22
Figure 2: On the left we illustrate the taxonomy of example of strictly nested labels. First handwritten characters are classified "number" "letter," and then these categories and letters. X, Y1 and Y2 denote random variables representing the input, the coarse label, and the fine label respectively. Y1*h represents the random variable associated to cach value k in the coarser node, i.c., Y, given that M/-1 k. k E V/-1-- The diagram in the center, illustrate the entropy of fine and coarse level, and how having information about coarser level may reduce the entropy of the fine level. The right diagram illustrates the reduction of uncertainty on the labels given the input, and how the uncertainty on the fine labels can be reduced even further if input information and coarse information are combined.
refined into specific numbers
The core of this work is to formulate classification problems such that the information of the input is extracted in a hierarchical way. To this end, intermediate (coarse) predictions Y; are jointly learned. The two key components of the proposed approach are: (i) nested information bottlenecks and (ii) combination of the predicted coarse and fine labels. The analysis provided this section is agnostic to most of the implementation details (which are addressed in coming sections).
Hierarchical information bottlenecks and the role of skipped connections. We assume the random variable X contains information about the sequence of strictly nested labels Y, i.e., XXY < HY;), as illustrated in Figure 2 More precisely, HY; XX > HY (X) and XX) < HY X). To exploit these properties, we standard DNN layers (convolu- tional, pooling, normalization, and activation layers). As illustrated in Figure 3 we begin by guid- ing the network to find low dimensional feature representation fi such that (X)) < (X) while, (X),, Yi) is close to / [X, Y1) where ((..) stands for the standard mutual information. (DNNs are remarkably efficient at compressing and extracting the mutual information between high dimensional inputs and target labels (Shwartz-Ziv & Tishby (2017), Moshkovitz & Tishby (011)))
favorid infirrsaiin bu (lenecc))
Leura the information the ispat about the csunse libel.
Leam complementary information, to represntation reftre prodiction.
Keen retining the embboddnng the in anmularity the label.
Figure 3: Illustrative scheme of the proposed framework. From left to right, the input data X, first set of layers that extract from X a feature representation fi, which leads to Vi (estimation of the coarse label Vi). fi is then jointly exploited in addition with complementary information of the input. This leads to a second representation fa from which a finer classification is obtained. The same idea is repeated until the fine level of classification is achieved. It is important to highlight that this high level deseription of the proposed model can be implemented in multiple ways. In the following sections we present our own implementation and provide details of the specific architecture we chose for experimental validation.
4

---

Under review as a conference paper at ICLR 2020
The second step consists in learning the complementtrr information such that when combined with the representation f1, allows us to achieve a second representation f2 from which the second hierar- chical label Y2 can be inferred. To this end, skipped connections play a critical role as will be further detailed next. As we discussed before,
T(X, Y,) = H(X) H(X > >H(I))++)
<1(X)
On the other hand, we want each feature embedding fs to compress the information of X while (X),, Yi) = [(X, Y.). Equation [ means that the finer the classification the more informa- tion from X we need. Notice that while in many DNNs, skipped connections have proved to help on the networks compactness and to mitigate vanishing gradients SZegedy et al. (2017). in the present work they are included for more fundamental reason. (See the experiments presented in the supplementary material Section F.2 for complementary empirical evidence and validation.) If we do not consider skipped connections, X + f((X) + (X+ forms Markov chain where [(x,/++1X < ((x)) (data-processing inequality) which contradicts Equation [ Sec- tion 4 presents experiments illustrating the impact of skipped connections nested learning, and complements the discussion started here.
Combination of nested outputs. We will present in the following sections experimental evidence showing that nested learning leads to an improvement in performance and robustness. In addition, the explicit combination of nested predictions, since the network simultaneously produces all the outputs Y; (with corresponding confidence), can improve the accuracy and robustness even further.
Our motivation is to explicitly refine the fine prediction leveraging the information of all the coarser outputs, i.c., (Y,,.... .... Yi) + Y. Let define sę(g) the network output score associated to the event Y; = q. In general. if (q) > s; (www) most likely P(Y, = g) P(Y; = w). but P(Y; = q) _ sg). In other words, score value of 0.3 does not mean the sample belongs to this class with 30% probability. This can be addressed by calibrating the outputs, which consists of mapping output scores into estimation of the class probability (g) + Py (q). as defined and thoroughly explained in Zadrozny & Elkan (2002). Py. (q) denotes the calibrated output of the network which approximates P(Y, = q). (We will precise how calibration performed in the following section.) Then, we can use the estimated probability associated to a fine label P'y to compute the conditional probability P(Y: y (Y:-1 = k). This is achieved by re- normmalzzing the finer labels associated to the same coarse label, i.e.,
P.. (1)
Ps P (4)
Σ EwEy Py. (w)
where y denotes the set of labels at granularity level i that share with y the same coarser label kg- Finally, the estimated conditional probability is combined with the prior of the coarser prediction to recompute the fine prediction Pr (g) = Prape (g) (kg).
3.1 IMPLEMENTATION CHALLENGES AND DETAILS
Training. Let (z) = (fi(1, (8)))..))) g: my = be the function coded by our net- work, where m denotes the number of granularity levels and as before x an input sample. Each sub-function gi corresponds to the output of granularity i (computed from the feature bottleneck f.). G depends on parameters (f))ş=1..,, which are common to the sub-fune functions of coarser granulari- ties, and some granularity-speciff parameters 7). The general framework of the architecture follows Figure 3 meaning trunk of convolutionnal filters with parameters 0 and fully connected layers for each intermediate outputs with parameters 1).
Training this kind of model with disparity of samples per granularity is hard, and naively sampling random batches of training data leads to a noisy gradient computation Kokkinos (2017). (See the supplementary material Section F.3 for complementary experimental validation and analysis.) In order overcome this issue, we organize the training samples and train the network in a cascaded manner. The dataset D is organized in subsets of samples labeled up to granularity for 1, m. Formally we can write D= y) with the set of inputs and y the set of labels. We consider that x = (xi)i=1,... mm and = (y.. where D; = (x) represents the subset of data for which the label is known up to the granularity level i.
5

---

Under review as a conference paper at ICLR 2020
Having the dataset partitioned in this fashion naturally leads to cascaded training of the network. We train the model to solve a sequence of optimization problems using (X1.1) as the training exam- ples at each step. We can write this sequence (P.) as: (P.): min 1y= Σラ= azCn, (Y,, Y2), where Cn is the n-categorical cross-entropy and a are the weights for each individual loss. We first start train the network on coarse labels and gradually add finer labels and start optimizing deeper parameters in an iterative way. We support our cascaded training protocol against a more standard one with experimental results presented in the supplementary (Section (Additional details are presented as supplementary material Section
B
Scores calibration. As shown by Hein et al. 2018)) deep neural networks with ReLU activations tend to produce overconfident results for samples which are out of the training distribution. In order to mitigate this and have meaningful ouiputs that can be combined, we consider two step calibration method. The first step consists in adding a "rejection" class for each level of granularity. Synthetic samples associated to this class are generated from an uniform distribution. By training the network with this supplementary class we mitigate the problem of overconfidence for sample that are far from the training distribution. However, keeping fixed coverage on the input space would require a number of samples that grows exponentially with the dimension of the input space.
To overcome this problem, we think of the network as a sequence of encoders in the latent spaces g: (()) (defined as the penultimate layer prior each prediction), and fine-tune the last layer adding synthetic out-of-distribution samples from an uniform distribution in the latent space. The latent space has a much smaller dimension than the original input space, and therefore is tractable to synthesise samples with a uniform coverage of the latent space. (Additional details are provided in the supplementary material Section B)
The second calibration step is classic temperature scaling introduced in Guo et al. (2017). This technique consists in scaling the output of the fully-connected layer before the softmax activation by an optimal temperature parameter. Given x the input data and g the function coded by neural network before the softmax activation o, the new calibrated output given by 9 = ( The temperature parameter I' is tuned so that the mean confidence of the predictions matches the empir-
ical accuracy, more precisely, we want to minimize, [IP(Ý = Y = p) p whereas before, y
denotes the network prediction of Y, and is the empirical confidence associated to it. The previous expression can be approximated, using a validation partition of the data, by computing the Expected Calibration Error (ECE) ECE = ンjー [Bil | B3) con (B,)), Nacini et al. 2015). This mea- sure takes the weighted average between the accuracy and confidence on N bins Bz. 1...N. denotes the total number of samples, and [By] the number of samples on the bin Bj.
EXPERIMENTS AND DISCUSSION
We consider three sets of publicly available datasets for experimental evaluation: the handwritten digits from MNIST (LeC'n & Cortes (2010)) the small clothes images from Fashion-MNIST (Xiao) jet al.) (2017). ). and CIFARIO (Krizhevsky et al.). First we study how the actual nested nature of the labels affects nested learning. Then we compare end-to-end training versus the proposed nested approach. In a third group of experiments we evaluate different combination methods to leverage all the predictions into a refined fine prediction. Then we evaluate the impact of skipped connec- tions. Finally, we study how the ratio of coarse and fine labels affects the network's performance. Additional experiments are presented in the supplementary material, section A F
Hierarchical structure versus random grouping. We group the MNIST samples into two groups of nested labels: "VG" and "RG." both with three labels of granularity. VG corresponds to hierar- chy that groups visual similarities; the coarse class groups digits into {{3, 8, 5, 0, 6), (9,
2}}, the intermediate class into {13, 8, 5), {0, 6). {9, 4, 7). {1, 2}}. and of course the fine class into {0} (9). Additional information on how we grouped the labels for the three datasets is pre- sented in Appendix C RG consists of a 3 level (coarse/middle//ingg grouping based on the order of N. Empirical results shows that grouping the labels based on visual similarities leads to better results in terms of accuracy, intuitively supporting the idea of nested learning. (See the results presented in Table 4 provided in the supplementary materials.)
Standard end-to-end training versus nested learning. Let us define |D,| as the number of training samples that are annotated up to the level of granularity i. To understand if adding more samples with

---

Under review as a conference paper at ICLR 2020
coarse annotation helps improving the performance on the fine task, compared models trained exclusively with fine data DA = Dy and models trained fine data plus coarsely annotated data B = D3 + D2 + D1. Table T shows the accuracy for the fine, middle, and coarse outputs for the standard end-to-end network versus the same architecture trained using nested learning and additional coarse and middle data. If we compare the lines for which D3- = 20%, we observe that adding coarse and middle granularity samples improves the accuracy even for the fine task. Moreover, it also improves the robustness of the model. We tested this models when test data is distorted and shifted from the conditions at training. Distortion I to 4 correspond to four levels (increasing the severity) of "turbulence-like" image distortion, the implementation of this distortion is inspired on the work of Meinhardt--lopis & Micheli (2014) (details are provided in the supplementary material, Section D). Again looking at Table I we that the model trained with additional coarse and middle samples is more robust and less overconfident. A similar pattern is observed for MNIST and fashion-MNIST datasets (see for example, tables 5 and 6 in the supplementary material). Complementing these results, Section in the supplementary material compared the proposed nested architecture with a Multi-Task learning approach.
method
Original
Distortion
Distortion 2
Distortion 3
Distortion
Bound on Std
Coarse (end to end), D3 = 20%
96.0/ 97.6
8 94,0
82.5/
80.2/92.9
77.0/ /930
Coarse (end-to-end). D3 32%
06.8/98.2
86.2 / 93.9
82.1/93.
78.3
75.1 /92.9
+0.3
Coarse (nested) Ours, D1.2.3 = 20%
96.5 996.7
87.8 92.5
84.9/9155
814//909
78.2 90.6
Middle (end-to-end). D3 20%
84.1 89.88
65.2 79.6
76.7/76.0
48.9/74.9
42.6 75,9
Middle (end to end), D3 = 32%
87.5 /92.8
65.5 81.5
76.3
78.8
41.3 79.4
1:0.4
Middle (nested) Ours, .1.3.3 20%
85.2/85.0
65.4 73.5
58.1
59.3/69.5
43.9 69.7
Fine (end to end), D3 201%
75.9 88.4
50.3 /
42.8 /64,9
34.2/65.5
28.4/ 73.4
(end-to-end), D2 32%
81.0 88.4
52.0 73.3
32.8/7199
26.8
40.5
Fine (nested) Ours, D1 20%
77./7777
62,6/62.6
43.2 /59.8
35.7/57.7
29.1/57.7
Accuracy and mean confidence /ooee//Coo// for Cifarl0 datasct. Coarse, fine, and middle indicate the accuracy at cach level of the label. End-to-end, denotes the model trained with exclusively data annotated for the fine label. We compare two end-to-end models trained with different amounts of data with fine labels. We first set D3- 20%. which we inerease afterwards to D3 32%. On the other hand, "nested" denotes the same architecture, trained with coarse middle and fine labeled data. In this experiment
Table I:
set DD = D2 = D3- = 20%. We repeated the distortion generation 10 times (for the levels of distortion), the last column "Bound on Std" shows the maximum standard deviation obtained across "Distortion 1" to "Distortion 4," therefore the value on the last column can he interpreted as a bound on the variability of the reported results. (Similar results are reported for MNIST and fashion-MNIST datasets, see tables 5 and 6 in the supplementary material.)
The previous discussion is interesting as it shows that including additional coarse data tends to help also the discrimination of the fine task. However one may argue that the comparison is unfair, as one model sees more data than the other. That is a very interesting point and we address it in following experiments where we study how the proportion of fine and coarse granularity data affect performance (for a fixed budget and different cost models). For now, let us observe what happens if we increase the amount of fine data from D3 20% to 32% (which assuming a linear cost model equals the budget of training with Dj = D2 D3 = 20%). As expected, (see Table [ the performance on clean test data improves for the end-to-end model. However, (see columns Distortion 1-4) it generalizes less to unseen (distorted) test data and also becomes significantly more overconfident.
Output combination. Kuncheva (2004) presented many useful combination methods for both one- hot encoding classifiers and continuous scores. We compare some of those combination methods (e.g., Mcan, Product, and Majority Vote) to the one that we designed specifically for multiple nested outputs. (Details and numerical results are provided in the supplementary material, Section E and Table We observed that different combination methods perform similar on test data that matches the train data, while the proposed method outperforms the others when test samples are distorted.
The role of skipped connections. Skipped connections are included in order to allow information flow from the input to the finer feature bottlenecks. Table 2 shows the accuracy for two networks with the same structure, trained on the same data, one including skipped connections (SC) and the other-one not. Section [2 in the supplementary material complements these results, and provides empirical measurements of the mutual information between different components of the model with and without skipped connections.
7

---

the MNIST test set Under review as a conference paper at ICLR 2020
Method
Original
Distortion
Distortion 2
Distontio
Distortion
Coarse (without SC)
99.6
96.3
91.3
85.3
79.2
Course (with SC)
99.7
97.2
81.4
Middle (without SC)
99.3
92.1
79.4
66.8
58.4
Middle SC) Ours
99.5
95.1
87.3
79.2
65.5
Fine (without SC)
98.9
88.2
72.4
56.4
44.0
Fine (with SC) Ours
99.2
94.2
84,9
69.5
53.2
Table 2: Comparison the same network structure, trained on the same coarse, middle, and fine data, with without skipped connections. 20% fine middle and coarse samples of MNIST dataset where selected for training. As the previous experiments, Distortion 1-4 correspond to test distorted samples with turbulence- like distortion (described in the supplementary material).
Working a budget. As discussed before, establishing fair comparison between models trained using more fine or coarse data not trivial. To address this problem, we investigated three cost models: linear, convex, and concave. For the linear model, we assume the cost associated to annotate one sample (I, y:) is proportional to [D Analogously, for the convex/concave cost model, assume the cost of annotating a sample (x, 81) is proportional to g(Da)) with 9(-) being strictly convex/concave. For each budget and cost model, we created hundreds of train sets with different amounts of coarse, middle, and fine samples. Figure shows shows the accuracy for different budgets and cost models. It interesting to observe that for a convex cost model, increasing the number of coarse annotations produces better results for the same budget; while as expected, the opposite is observed for a concave cost model. We present additional experiments in the Figure 10 in the supplementary material.
Linear model
Concave model
Convex model
349
Budget
Budget
Budget
Figure 4: Accuracy the MNIST data when different ratios of coarse, middle, and fine samples are selected for training. Plots report the accuracy on the prediction of the finest category (ten classes: 0 9). If we define n1, n2 and ny the number of samples for which we know only the coarse, middle, and fine label respectively, the budget associated to a training set Bag ma.na = n mig((Di function associated to labeling a coarse, middle, and fine sample. In this experiment, we tested three models for a linear model where the cost is lincar, a concave model (we chose g(u) x log(x)), and a convex model (we chose g(u) e"). The colors represent the proportion of coarse samples in the training set. Blue represents proportion of coarse samples, and red larger proportion of fine labels. (Additional results are presented Figure i0 the supplementary material.)
n229( Daal) + maga()))). g represents cost.
CONCLUSION
In this work we introduced the concept of nested learning, which improves classification accuracy and robustness. Moreover, it allows to leverage information from datasets annotated with different levels of granularity. Additionally, experiments suggest that nested models have a very desired be- haviour, c.g., they gradually break as the quality of the test data deteriorates. We showed that imple- menting nested learning using a hierarchy of information bottlenecks provides a natural framework to also enforce calibrated outputs, where each level comes with its confidence value.
Given a fixed budget and model, we studied what is the economically most efficient strategy for labeling data. Furthermore, experimental results show that if the amount of fine training samples constant, then adding samples with only coarse annotation increases the performance and bustness for the fine task. To recap, the introduced nested learning framework performs as expected from own human experience, where for good data we can provide high level inference with high confidence; and when the data is not so good, we can still provide with high confidence some level of inference on it.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Bilal Alsallakh, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. Do convolutional neural networks learn class hierarchy? CoRR, 2017.
Michael Behrisch, Benjamin Bach, Nathalie Henry Riche, Tobias Schreck, and Jean-Daniel Fekete. Matrix reordering methods for table and network visualization. Comput. Gruph. Forum, 35(3): 693-716, June 2016. ISSN 0167-7055. doi: 10. 10.1111/cgf. 12935.
potttps://de. .org/
of 12935
Ishmael Belghazi, Sai Rajeswar, Aristide Baratin, R. Devon Hjelm, and Aaron C. Courville. MINE: mutual information neural estimation. CoRR, wwwpss/...0000000 1.0022, 2018a.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062 2018b.
Hakan Bilen and Andrea Vedaldi. Integrated perception with recurrent multi-task neural networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 235-243, 2016.
Tyler Clark, Alexander Wong, Masoom A Haider, and Farzad Khalvati. Fully deep convolutional neural networks for segmentation of the prostate gland in diffusion-weighted mr images. In In- ternational Conference Image Analysis and Recognition, pp. 97- 104. Springer, 2017.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 48 64, Cham, 2014. Springer International Publishing.
Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Scbastian Thrun. Dermatologist-lewwe classification of skin cancer with deep neural networks. Nature, 542:1 15-, January 2017. URL http: doi. org/o0.g.o..oooooooooooooooo.... 1038 103///atttee22a0/ 2rl121
Rob Fergus, Hector Bernal, Yair Weiss, and Antonio Torralba. Semantic label sharing for learning with many categories. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios (eds.), Computer Vision ECCV 2010, pp. 762-775, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-11 August 2017, 2017.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how mitigate the problem. CoRR, abs/1812.05720,, 2018
Eunwoo Kim, Chanho Ahn, and Songhwai Oh. Nestednet: Learning nested sparse structures in deep neural networks. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR), June 2018.
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. The IEEE Conference on Com- puler Vision and Pattern Recognition (CVPR), July 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). URL http::////wwt http///wwwccco Eoronto...............o......... wwweekriiiiaaee prri//cfaa. mtml
Ludmila I. Kuncheva. Combining Pattern Classifiers: Methods and Algorithms. Wiley, 2004.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Baoyuan Liu, Fereshteh Sadeghi, Marshall Tappen, Ohad Shamir, and Ce Liu. Probabilistic label trees for efficient large scale image classification. pp. 843-850, 06 2013. doi: 10.1 109/CVPR. 2013.114

---

Under review as a conference paper at ICLR 2020
Ping Luo. Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. 2018.
Enric Meinhardt-Llopis and Mario Micheli. Implementation of the Centroid Method for the Correc- tion of Turbulence. Image Processing On Line, 4:187-195, 2014. doi: n0.5200//poo..214.. 105.
Michal Moshkovitz and Naftali Tishby. Mixing complexity and its applications to neural networks. CoRR, 0is//7000,7799 2017.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CoRR, abs/1412.1897, 2014. URL http: 1897
7/arxi //arxiv.org/an
Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6):1191- 1253, 2003.
Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In British Machine Vision Conference, 2015.
Rajeev Ranjan, Vishal M. Patel, and Rama Chellappa. Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition, 2016. (Rttttt///aaxttt
org/ aas///b//////// /1603. C1249
O. Ronneberger, P.Fischer, and Brox. U-net:
Convolutional networks for biomedical image seg- mentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 9351 of LNCS, pp. 234 241, 2015.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Advances in Neural Information Processing Systems, pp. 2483- 2493, 2018
Yian Seo and Kyung shik Shin. Hierarchical convolutional neural networks for fashion image clas- sification. Expert Systems with Applications, 116:328 339, 2019. ISSN 0957-4174. doi:
0.0..02.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. CoRR, atss//7030000 10, 2017.
Fabio Spanhol, Luiz Soares de Oliveira, Caroline Petitjean, and Laurent Heutte. Breast cancer histopathological image classification using convolutional neural networks. 07 2016. doi: 10.
1100//CCNN.2011.77755111
Nitish Srivastava and Ruslan Salakhutdinov. Discriminative transfer learning with tree- based priors. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26. pp. 2094 2102. Curran Associates, Inc., 2013. URL http: wwarrrsn.iis... hwwp.cccppp..p.oooo pdf
502
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty- FiirttyFFirtaa AAAI Confer- ence on Artificial Intelligence, 2017.
Naftali Tishby, Fernando C. Pereira, and William Bialek, The information bottleneck method. pp- 368-377, 1999.
10

---

Under review as a conference paper at ICLR 2020
Isaac Triguero and Celine Vens. Labelling strategies for hierarchical multi-label classification tech- niques. Pattern Recognition, 56:170-183, 2016.
Jonatas Wehrmann, Ricardo Cerri, and Rodrigo Barros. Hierarchical multi-label classification net- works. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Con- ference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp- 5075-5084, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms, 2017.
Xuehong Mao, S. Hijazi, R. Casas, P. Kaul, R. Kumar, and C. Rowen Hierarchical enn for traffic sign recognition. In 2016 IEEE Intelligent Vehicles Symposium (/V), pp. 130-135, June 2016.
Zhicheng Yan, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Robinson Piramuthu. HD-CNN: hierarchical deep convolutional neural network for image classification. CoRR, abs/1410.0736. 2015.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pp. 694-699, ACM, 2002.
Bin Zhao, Fei Li, and Eric Xing. Large-scale category structure aware image cat- egorization. In J. Shawe-Taylon R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24, pp. 1251 1259. Curran Associates, Inc.. 2011. URL http::/ ccpp.ec/
4347 [ag-://gsee--eeooooortotteeeeaaa pdf
A. Zweig and D. Weinshall. Exploiting object hierarchy: Combining models from different category levels. In 2007 IEEE 11th International Conference on Computer Vision, pp. 1-8, Oct 2007. doi: 10.1 11110//CCV.20077449900666

---

Under review as a conference paper at ICLR 2020
Supplementary Material
A ADDITIONAL OUTPUT EXAMPLES
l'animal', 0.998) intermediate (hhorseddeer, 0.86)
coarse ours: ('animal', 0.994)
intermediate ours: ('horse/deer, 0.594)
fine ('horse., 0.853)
fine ours: ('horse', 0.574)
fine end-to-end: ('horse', 0.943)
fine end-to-end: ('dog' 0.954)
coarse ours: 0.999)
coarse ours: ('animal:, 0.994)
intermediate ours: (cat/dog 0.733)
ours: ('dog' 0.68)
intermediate ours: kat/dog 0.808)
fine ours: ('dog" 0.616)
('dog',
fine end-to-end: ('bird', 0.575)
coarse ours: l'vehicle', 0.997)
coarse ours: ('vehicle', 0.776)
intermediate ours: airplane/ship 0.967)
intermediate ours: airplane/ship 0.678)
fine ours: ('ship'. 0.966) fine end-to end: ('ship', 0.993)
fine ('ship', 0.781) end-to-end: ('cat', 0.992)
('animal', 0.954)
coarse ('animal,, 0.654)
intermediate ours: (bird/frog. 0.925)
intermediate (bird/frog. 0.543) ('bird. 0.387) ('ship", 0.929)
fine ours: ('bird', 0.911)
fine end to end: ('bird', 0.999)
('animal,, 0.999)
('animal', intermediate ours: (bird/frog. 0.889) fine ours: ('bird', 0.89) fine end-to end-to-end: ('cat', 0.575)
('bir/ffrog, 0.988)
fine ours: ('bird', 0.968)
fine end-to-end: ('bird', 1.0)
coarse ours: ('animal', 0.947)
coarse ours: ('animal', 0.998)
intermediate ours: kct//dgg 0.752)
intermediate ours: (cat/dog, 0.933)
fine ours: ('cat', 0.453)
fine ours: ('cat', 0.523)
fine end-to-end: ('cal', 0.532)
fine end-to-end: ('bird', 0.532)
coarse ours: l'vehicle', 0.99)
coarse ours: ('vehicle', 0.77)
intermediate ours: airglane/ship 0.916)
intermediate ours: airplane/ship 0..782)
ours: l'airplane', 0.865)
fine ours: ('airplane', 0.422)
('airplane', 0.941)
fine end-to-end: ('bird'. 0.741)
Figure 5: These results complement the example illustrated Figure 1 The output of a standard (end-to-end) DNN and our proposed nested learning version are compared. On the left we show clean images from the test of CIFAR- 10 dataset, on the right, the same examples but blured. Next to each image our prediction (for the fine, middle, and coarse level) and the prediction of a standard (end-to-end) DNN are displayed. Both DNN share the same architecture and their performance is compared on Table (rows corresponding to "(end-to-end) D3 32% and "(nested) our 21,2,3 = 20%"). As shown Table the performance of both networks is similar on clean data (i.c., data that match the train distribution), but our approach can provide more accurate middle and coarse predictions when the input data is corrupted, moreover, we are significantly less overconfident on the prediction of out-of-distribution test samples.
B IMPLEMENTATION DETAILS
B. ARCHITECTURE
The architecture of our model is presented in Figure We evaluate the same architecture for MNIST and Pashion-MNIS
the images have the same and number of channels. For CIFARIO, the architecture of our model is very similar but with an increased depth of the convolutional filters. Classifying images from CIFAR10 is indeed harder problem than classifying images from MNIST or Fashion-MNIST" and therefore, it requires a model with more parameters. The architecture we is adaptation of the U-Net, designed fit our proposed framework of nested information
12

---

Under review as a conference paper at ICLR 2020
bottlenecks. The U-Net is indeed good starting point, it meets most of the criteria that presented in Section 3 First, it consists of a convolutional network that enforces a bottleneck repre- sentation. Second, it presents skip connections that allow information of the input to flow to deeper components of the network.
We also added Match-Normalization (BN) layers after every convolutional filters in order to achieve a faster convergence. BN very helpful mitigate the vanishing gradient phenomenon. Also, since it introduces randomness during training. BN acts regularization. Classic L1 regularization was shown to be similar BN Luo et al. (2018), while including BN layers also improves training speed and stability as explained by Santurkar et al. (2018).
Max-pooling Convolution with kernel of size 3 Batch Normalization
Up-sampling
Skipped-connectione
12
12
12
Bx1
0 8
1
3
4 5
7
GAP
18x1
16x3x3
32x3x3
0123
Global
Average
12x1
Pooling (GAP)
0 1
Figure 6: Architecture of our CNN for both Fashion-MMn MNIST and MNIST. This model is adaptation of the U-Net network (Ronneberger al. 2015)) designed to fit nested learning framework. The blue boxes repre- sent the feature extracted by convolutional layers. We perform global average pooling rather than flattening decrease the number parameters. After the global average pooling, the feature vector normalized with respect to the L2 norm (instance normalization layer). The normalized features are followed by fully connected layers to compute the final output. The model used to test CIFAR10 set is very similar but has convolutional layers with more kernels handle this (sightly more complex) task.
B.2 TRAINING
We train the models in an iterative way. First we optimize the weights for the coarse prediction with the samples that only coarsely annotated and freeze the remaining weights. Then, optimize the weights to the intermediate output with the samples that are coarsely and intermediately annotated. Finally train the whole network, with the samples that are coarsely, intermediately. and finely annotated. (In general, this process has as many steps as levels of granularity.) Each training step is performed using ADAM optimizer with different learning rates: 2 10-3 for the first step, x 10-3 for the second, and x 10-4 for the last. We stop the training of each step when stagnation of the validation loss is observed.
13

---

Under review as a conference paper at ICLR 2020
B.3 CALIBRATION
The calibration consists of two main steps. First "rejection" class is modeled using uniform distribution on the latent space. This models duttto--istiiittion samples and mitigates overconfident predictions portions of the feature space where training data is observed. The second step consists of temperature scaling to convert output scores into approximations class probabilities.
The "rejection" class. For every level of granularity and for every sample of the training dataset, we store the normalized outputs of the global averaging layer (GAP) in a dataset D,. The samples have size and are normalized with respect to the L2 norm, therefore, they live in 5 1), the unitary sphere Rs centered in zero. We randomly sample n, new instances from an uniform distribution in t (0, 1). These samples (associated to new "rejection" class) are aggregated to D; and the fully connected layers fine tuned Naturally, the larger [Da and 8,, the larger n, should be. We n, DD (s()) where S(s;) the area of the hypersphere of unitary radius in the 8;- dimensional space. Figure illustrates for a one dimensional toy example, how the proposed ideas provide and efficient solution reduce outputs overconfidence out-of-distribution input samples.
(a)
(b)
(c)
One dimensional toy example that illustrates the problem prediction overconfidence for input samples that are far from the training distribution. In this example, two classes ("1" and "0") are considered and we assume that the input ~ X is onc-dimensional. (a) illustrates the empirical distribution of cach class (on green P uniform distribution from which sample synthetic training instances associates the "rejection" class. Figure (b) shows the confidence output associated the class "1" and "0" different values only on the original data (standard approach). Figure (c) illustrates the output of the same DNN trained with the samples associated to the classes "1" and "0", plus the synthetic samples from the "rejection" class.
Figure 7:
1) and yellow P(X 0)). In addition, we illustrate (blue distribution) the
for model trained
Temperature scaling, Temperature scaling improves calibration by converting arbitrary score val- ues into an approximation of class probabilities. As explained in Section 3.1 we minimize the empirical ECE metric over the validation set, composed of 5000 unseen samples for all the dataset we tested our method on. To find the optimal temperature T. we compute the ECE over 50 values between 1 and 3 and select the one for which the ECE minimized. If T' the model is already well calibrated, On all our experiments, the minimal ECE value was always reached for '' values strictly lower than 3.
"To this end, used ADAM optimizer with learning of 10
14

---

Under review as a conference paper at ICLR 2020
C LABEL GROUPING
we showed that nested learning performs better if the taxonomy of the labels has a real nested structure (e.g., one based on visual similarity). To group fine labels into meaningful nested structure, we first train shallow neural network and classify images into the fine classes. Then we used the confusion matrix M associated this auxiliary classifier to establish which classes are closer to each other.
In Section 4
For MNIST and Fashion-MNNI for example, we wanted to group the labels in coarse categories which also contained 2 intermediate categories. To this end, we find the 10-permutation applied to both the rows and columns of M, such that the non-diagonal 5 5 matrices of .M had the lowest possible Ll-norm. We iterate this process to find the intermediate categories. It is computationally hard go through all the permutations, therefore, (0166) to perform matrix reordering with a reduced complexity.
follow the ideas proposed by Behrisch et al.
Figure presents the groups of labels obtained for MNIST, Fashion- MNIST, and CIFARIO. Our results show natural and intuitive taxonomies, see, e.g., how MNIST digits are grouped according to natural oripntdr similarity, with and in the same intermediate class for example.
Coarse: 0
Coarse:
Tops
Accessories
Internadaaie
3
8
5
9
4
7
Heawy topu
Outhers
Interneditte11
Intomediate::
6
2
Taxonomy obtained for MNIST
(b) Taxonomy obtained for Fashion-MNN MNNIST
Vehicles
Animals
dog
Deer or horse
Beat plane
Ooer
Horbe
Bird Frog
(c) Taxonomy obtained for CIFARIO
Figure Nested groups labels obtained by minimizing the non-diagonal components on the confusion matrix of an auxiliary simple classifier.
D PERTURBATIONS
Selecting realistic and meaningful perturbations to DNN models is non trivial problem. For example, adding Gaussian noise mainly affects the high frequency components of the input images and we observed that both standard (end-to-end) and nested networks were not severely affected by this type of perturbation. this work we focus on structural deformations inspired by a model of turbulence. The pseudo-code of this perturbation is presented in Algorithm I and was inspired in
15

---

Under review as a conference paper at ICLR 2020
the work of Meenhardt-LLopis & Micheli (2014). Figure illustrates the distortion of an example image from MNIST dataset for different levels of turbulence intensity.
9
9
Figure 9: Example of different perturbations applied during testing time. From left right: the original sample, and the distorted version with parameters (S, (1,0.8), (S,T) (1,1.0. (8,1) (1,1.3), and (S. T) = (1, 1.5) respectively.
Algorithm 1: Turbulence distortion
Data: the input: I E " the parameters (S,T) E R
Result:
the distorted image diist
Creating a vector field (u,v) for the distortion:
u, normal moise
( (w, h)), normal noise((w, h))
u, goussian__ fflterr. S), qquusian__fitter(t. S)
T 2, ut] X std(w)
T
Interpolate the image with the obtained vector field:
T(8,7) bilinear interpolate(T, u, v)
E COMBINATION METHODS
Combining multiple classifiers is standard approach in machine learning. However, most ap- proaches combine similar outputs and are not designed for the specific problem of nested learning. We compared some standard combination methods and calibration-based combination strategy.
Combination methods can be classified into two categories. First those that combine or vote dis- crete classification results, where classifier outputs considered one-hot encoding vectors. For example, the Majority Vote (MA) which consists aggregating the decision of multiple classifiers and selecting the candidate that receives classifiers continuous outputs. As we show the following, the second class of methods are more suitable for the combination of nested outputs.
votes. The second category of methods combine
For example, suppose that given an input sample x, the classifier outputs an estimations of the probability associated to each coarse, middle, and fine label: Py, Py. and Py.- In order to apply standard combination methods, we need to transform the coarse and intermediate predictions into fine prediction and vice-versa. transform the probability associated granularity, we can assume the finer classes associated to each coarse class are equally probable, i.e., the probability associated label. On the other hand, to aggregate probabilities associates to a finer level into coarser level, we can simply add those probabilities associated the same coarse node.
coarse label into a finer
coarse node, is divided equally into the finer nodes associated this
Once probabilities associated to coarse levels are propagated to the fine levels and vice-versa, we can combine them using the mean or the product rule as deseribed in Kuncheval (2004). Table B shows the result of combining nested outputs with standard combination methods and our strategy described in Section 3.]
F ADDITIONAL EXPERIMENTS
16

---

Under review as a conference paper at ICLR 2020
Distortion
Without comb.
Ours
Coarse & Fine
Mean
Product
MV
Original
98.8
98.6
98.7
98.7
98.7
98.5
Distortion
82.4
Distortion 4
45.8
50.6
49.6
48.6
49.6
47.5
Table 3: Comparison of the fine accuracy for different combination techniques. MNIST dataset with D3 D2 D3
The model is trained on
10%
Convex model
1.0
0.9
Acc on the finest category
0.8
0.7
0.6
pestortation
orggial
0.2
Hide
ride bettom
0.1
10
12
14
Budget
Figure 10: Accuracy on the classification the fine label MNIST data. These results complement results presented in Figure 4 Figure and for three cost models (linear, concave, and convex). In this figure, we show the performance of each individual model. For each model we display the accuracy on the clean data (dots) as well as the accuracy on different types of distortions. As before, the proportion of coarse and fine labels during training is illustrated coloring each data point, blue indicates higher proportion of coarse samples while red higher proportion of fine samples.
4 iillustrates the results on clean test data grouping five levels budgets
Perturbation
Original
VG coarse
99.7
VG middle
99.5
middle
VG line
98.6
Table 4: Comparison of the visual label grouping (VG) and random label grouping (RG) in terms of accuracy for our model trained on MNIST.
method
Original
Distartion
Distortion
Disturtion
Distortion 4
(end-to-cnd), D3 322%
98.9/
90.7
95.5
Coarse (end-to-cnd), D3 = 20%
99.3
94.5 96.9
90.1 /96.
87.3 94,9
85.9 94.5
Coarse (nested) 11.2.3 20%
96.2/
Intermediate (end to end), D3 132%
94,3 95.8
82.1 /9099
77.2 89.5
73.0 88.2
69.6/
Intermediate (end-tn-end), D2
93.7/95.4
89.0
71.3/88.3
Intermediate (nested) Ours, D:
20%
94.2/94.7
87.3/9122
82.6 89.3
77.7 / 88.5
72.8 87.3
Fine (end-to-cad). D3 = 32%
98.3/91.5
70.8 70.8/83.5
62.6/81.3
55.07 80.5
79.6
Fine (end-to-cnd), Dg 20%
87.1 /91.1
70.1 /84.7
62.3/854
54.7 81.1
50.3/80.2
Fine (nested) Ours, D; 22, 20%
88.9
73.7 /813
67.6/777.
57.6 76.4
51.2/
Table 5: Accuracy and mean confidence (Aoc% /Con f%) Fashion-MNIST dataset. Coarse, fine, and middle indicate the accuracy at each level of the label. End-to-end, denotes the model trained with exclusively data annotated for the fine label. We compare two end-to-end models trained with different amounts of data with fine labels. We first set Dj 20%, which we increase afterwards to D3 32%. On the other hand, "nested" denotes the same architecture, trained with coarse middle and fine labeled data. In this experiment set Di D2 D3 20%.
17

---

Under review as a conference paper at ICLR 2020
Mothod
Original
Distortion 1
Distortion
Distortion
Distortion 4
Coarse (end-to-end), D3 16%
99.4/99.5
96.1 /97.8
90.9/ /9.33
82.3 /92.6
72.0/90.9
Coarse (nested) Ours, P1, 10%
99.5/99.5
96.9 /97.5
22.3/95.4
85.9/ 93.8
79.6/92.1
Intermediate (end-to-end), 12g 16%
Intermediatc (nested) Ours, D122.3 = 10%
99.2/99.2
95.5 / 96.0
87.9 99.66
77.0 89.4
66.5 /8.59
Fine(end-to-end) 23 16%
85.7
49.7
Fine (nested) Ours, D 10%
98.6 98.4
19.5593.5
82.8 17.88 67.5 83.0
50.6 80.4
Table 6: Accuracy and mean confidence (Acc% /Con for the MNIST dataset. Coarse, fine, and middle
indicate the accuracy at cach level of the label. End-to-end, denotes the model trained with exclusively data annotated for the finc label. On the other hand, "nested" denotes the same architecture, trained with coarse middle and fine labeled data. In this experiment set D DD 20%.
F.1 COMPARING WITH A COMMON MTL ARCHITECTURE
Classic Multi- Task Learning (MTL) DNN approaches extract common features that are on the top the architecture branched for the prediction of multiple tasks. To this end, common MTL ar- chitecture consists of shared convolutionnal blocks followed by task-specific classification (fully connected) layers. Figure [I illustrates an MTL architecture designed to obtain a nested classifica- tion of MNIST characters. The network enforces an information bottleneck that encodes the input in (64,) 1) feature vector. Then, is connected to the three classification branches implemented by sequence of fully-connected layers. This MTL model and nested model have approximately the same number of parameters and are trained in an identical way on MNIST dataset.
01
1 2
222 (14..))
(M43))
1 2 4
(2.143))
(81414))
(4.0023)
Figure 11: Architecture of test Multi-Task-Lermingg CNN. This architecture corresponds a sequence three blocks two (convolutional + batch normalization) layers followed by Maxpooling Layer. In order to reduce the dimension of the penultimate feature vector, we perform a global average pooling. This network and our Nested CNN have approximately the same number of parameters and are trained using the same training protocol and data.
Table shows the classification accuracy for our nested architecture presented in Section and the multi-task learning architecture presented above. On the original distribution both architectures perform roughly equivalently, but on perturbed test data the proposed nested model is considerably superior. Nested learning outperforms its MTL counterpart with a gap of 18% for the fine task and gap of more than 10% for the coarse task for distortion levels 3 and 4. These results provide additional evidence that having a hierarchical nested architecture promotes robustness and leads to more consistent models.
18

---

Under review as a conference paper at ICLR 2020
Mothod
Original
Distortion 1
Distortion 2
Distortion
Distortion 4
Coarse (MTL).D,.. 1.2.3 10%
99.5/99.7
92.4/96.
82.6 94.3
73.8 /928
66.1 /93.1
Coarse (nested) Ours, D1,2.3 = 10%
99.5/99.5
96.9 /97.5
92.3 9954
85.9 93.8
79.6/92.1
Intermediate (MTL). 2.,2.3 10%
91.1
75.5
60.2 88.8
49.9/88.2
Intermediate (nosted) Ours, ..2.5 = 10%
99.2/99.2
95.5 / 96.0
87.9 /9.66
77.0 89.4
66.5 /77.9
Fine(MTL), 21,2,3 10%
98.9/99.0
91.0/899
48.6/ 77.9
32.4/
Fine (nested) Ours, 1,,,,3 10%
98.6/ 98.4
92.8/93.5
82.8 18.88
67.5 83.0
50.6/ 80.4
Table 7: Classification accuracy and mean confidence (Ace%/ /Con//Conlff (%) for an example of MTL and nested learning on the MNIST dataset. Coarse, fine, and middle indicate the accuracy at cach level of the label. MTL denotes the standard Multi-task Learning architecture described in Section F.T On the other hand, "nested" denotes our nested model which enforces hierarchical feature embeeldings. In this experiment set Di D2 D3- 20%.
F.2 SKIPPED CONNECTIONS AND THEIR IMPACT ON THE FLOW OF INFORMATION
Obtaining an empirical measure of the mutual information (Ml) between two high dimensional random variables is very hard numerical problem (Paninski (003) However, recent progresses in deep learning made a numerical approximation (MINE) tractable by exploiting the flexibility of neural networks and properties of the mutual information. Belghazi et al. (2018a) proved that a MI estimation can be seen as an optimization problem. They rely on the following characterization of the mutual information as the Kullback-Leinn Leibler (KL-) divergence,
[(X, Z)
[Px @ Pz).
Based on this, the authors use the Donsker-Varadhan representation of the KL divergence, which introduces a dual optimization problem,
(Pll)) sup [r[T] log (e"]), T:Q-R
where the supremum is taken over all the functions T such that the two expectations are finite.
Since the parameters of a neural network can be used to encode large space of functions, the authors propose to solve Equation equation | for Ta E F, where FF denotes the space of functions encoded by pre-defined network architecture with parameters 0 E O. An approximation of the MI can be obtained solving the problem
(e 2) = sup xx (To) log (o)(""))")))
(5)
dE
Equation can be numerically solved using standard optimization tools of deep learning (see Belg- hazi et al. 2018b) for details).
In order measure the impact of skippedd-connecionss from MI perspective, measure the empirical approximation of the mutual information between different sections of the proposed nested architecture. We compared these results for the same network with and without skipped connections. Let us refer to the network with skipped connections with the subscript 1 and the network without skipped connections with the subscript 2. We define different variables of interest particular stages of the network and estimate the mutual information between them. We will call F, (X) and Fx(X) the variable corresponding the feature map created after the 2nd Maxpooling layer (see Figure I1l with or without skipped connections. Similarly, let G (X) and G₂ (X) be the feature maps obtained before the second GAP layer. We will also consider H, (X) and (x)X) the coarsest feature maps obtained before the first GAP. (Before feeding these features to MINE algorithm we performed average pooling to reduce the dimension of the input variables.)
We estimated the MI between (X), (X)) and (XF(X,, (X)), i = meaning "with skipped connections" and i = 2 "without skipped connections" (the network without skipped con- nections is re- trained to allow the model to adapt to this new configuration). Figure [2 (left side) sketches in which sections of the model we measuring the mutual information, and (right side) the evolution of the MI estimation (MINE algorithm) for 34 steps. After convergence of MINE algorithm we compared the mutual information between the random variables F., H. and G, with and without skipped connections (illustrated green in the sketch of the left side of figure 12
A = -11,,,,,1))) 0.21 A1/2.
(IF,
= 0.37,
(6)
19

---

Under review as a conference paper at ICLR 2020
Information matton surin training ATANI
F
H
i-1
3.0
Figure 12: Left: we sketch the sections of the model where the mutual information is compared. Right: evolution of the estimated MI for (X), G1 (X)), (X), G2(X),, T(F (X), (X)),, and 2(F2(X), (X)) during the optimization of the MINE estimator. The MINE network is com- posed of three fully connected hidden layers of 100 neurons each and elu activations.
As shown in Equation 6 skipped connections play an important role to allow information to flow to the feature representation of the finer classes. This provides additional numerical evidence to the discussion presented in Section 3 supporting the importance of including skipped connections and avoiding information bottleneck at the finer classification.
F.3 COMPARISON OF TRAINING PROTOCOLS
Our training protocol is organized in cascaded way. As explained in Section 3.1 the reason for this two-fold. First, this would eliminate the problem the noisy stochastic gradient computations which happens when there is not enough samples in the mini-batch for specific task/granularity. Second, it provides a good initialisation of the weights associated to the coarser levels, and we observed empirically that this leads to faster convergence and better accuracy. In order to illustrate the impact of the training protocol, we compare the results obtained by training the exact same model. with the same data, but using two different training strategies. We compared the described cascade methodology with the standard methodology (i.e., selecting batches of random samples out of the training set). As samples are annotated heterogeneously (with all or some fine to coarse annotations) we defined the unified loss each mini-batch as
M
N
Lbatch
a x Wis Co(is.
M denotes the number of samples in the batch, N the number of nested level, where wij is a masking vector which indicates if the sample j is annotated or not for the level of granularity i, a; are global weight that help to balance the loss associated to each level L.
We observed that cascaded training achieves substantially better performances than the traditional training as we can see in Table 8 The gaps are very significant also on the training distribution as we can observe in the first column. Additionally, if we studied the behaviour of the network while, e.g., see the results presented in Figure 13 As we can see, the proposed protocol is more suitable for the proposed multi-level problem leading to faster convergence and better models. After 26 epochs of coarse learning, we add the intermediate data for 30 epochs before adding the fine data for 44 more epochs. We can observe that very few epochs after the introduction of the intermediate data, the accuracy of the intermediate classifier is already superior to a standard training procedure. We observe this phenomenon again for the fine classification, see for example, the training evolution over the green area illustrated in Figure 13 In addition we can observe that the accuracy of the cas- caded training classifiers oscillates far less than the accuracy of the traditionally trained classifiers. This provides additional evidence that our method helps to deal with the noisiness of the stochastic gradient estimation.
20

---

Under review as a conference paper at ICLR 2020
standard Accuracy the coarse classdiers during training
0.98
0.96
maniing with course
framing CORISE labels
with coarse. middle fine labels
0.52
0.90
Accuracy the middle classifiers during training
standard traming cascaded training
0.98
0.96
MAM
0.52
0.90
Labels
training madile labels
madile and line labels
training
Accuracy of the fine classifiers during training
standard traiing cascaded training
0.975
0.550
tranning with coarse
Tramiing coams middle
with course, middle fine
Mmmm
M
0.500
MM
0.850
0.800
100
traniing
Figure 13: These plots compare the accuracy of the classifiers for the cascaded training and the traditional training. On the top, we represent the evolution of the coarse classifiers during training. In the middle, represent the evolition of the accuracy of the intermediate classifiers. At the bottom, we represent the evolution of the accuracy of the fine classifiers.
21

---

Under review as a conference paper at ICLR 2020
Method
Original
Distortion
Distortion
Distertion
Distortion 4
Course (Iraditional training).P1,, a.3,
82,0
04.9/75.4
72.5
73.4
Coarse (cascaded training). D1.2.3 10%
99.5 99.5
96.9 /6.9/97.5
92.3/95.4
85.9/
79.6/92.1
Intermediate (traditional training). D1.2.3 10%
96.9 81.2
91.1 74.2
81.1
66.6167.6
53,5/
Intermediate (cascaded training). D1.2,3 10%
99.2 99.2
95.5
87.9
77.0/ 89.4
66.5
Fine (traditional training). D1225 10%
94.2 84.1
86.7 /79.
734/71.9
53.2/
37.6/ 70.5
Fine (cascaded training), D: 2,5 10%
98.6 98.4
92.8/93.5
82.8/87.8
67.5/8300
/ 80.4
Table 8: Accuracy and mean confidence (Ace% Alcc%//ooo Con f%) for the MNIST dataset. Coarse, fine, and middle
indicate the accuracy each level of the label. In this table compare the results of traditional training cascaded training the same architecture. In this experiment significant improvement with our method.
DD D3 20%. We notice
22


Reviews:
[{"review_id": "SyeESNS6YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Nested learning for multi-granular tasks\n\n1. Summary\nThe paper considers a framework for classification with a hierarchy of labels [from coarse to fine]. The paper proposes a network architecture with multiple bottleneck layers, one for each label level, and skip connections. The objective function is the standard classification loss. The experiments show that coarse labels help learning and can improve label efficiency, i.e. don’t need all fine labels to get good classification performance.\n\n2. Opinion and rationales\n\nWhilst I think the execution of ideas is good and the motivation is very practical, I’m leaning towards “reject” for this paper due to the reasons below. I welcome the authors’ clarification and am willing to reconsider my view.\n\ni. The paper justifies the proposed architecture as successive information compression into label embeddings using some entropy based criteria (as in information bottleneck literature). This ensures the relationship of the entropies between the label layers. However, I’m not sure this justification is necessary (albeit being a valid one) given that the training/later sections do not come back to this justification.\n\nii. The novelty of the proposed architecture and training approach is low. The network is a nested structure of successive classifiers. The proposed calibration using rejection class and temperature scaling is not new.\n\niii. it would be better if the proposed architecture + method are validated on more real-world datasets with a more natural label grouping scheme, and compared to alternative architectures, e.g. multi-task learning with a shared network except the output layer.\n\n3. Minor details\n\nSome citations should be enclosed in brackets", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HygywE9K5B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The problem addressed is how to train a DNN to learn a hierarchical representation of the input that has heterogeneously annotated labels at various levels of granularity. After training the network’s goal is to emit sequentially finer grained labels with corresponding confidence. The authors use information theory to propose a network typology of information bottlenecks with skip connections to achieve this nested learning problem.  \n\nDecision: weak reject. \n\nReason: The proposal of learning a hierarchical representation is not new. Nor is the architecture. I think the interesting points (that are not really flesh out as much, but appear to be in the auxiliary material - section B.2) is the training regime. I would’ve liked to have seen more of what the role of the training regime is on the outcomes and how the network’s gradient’s behave in different regimes. But in general, the paper is well organized and argued, although there is a little belaboring of ideas of entropy and mutual information only to use it to buttress a point that is made and left hanging. The paper also defines too many concepts for every proposition it wants to support, making it cognitively costly to follow. \n\nFeedback:\nPleasure reading the paper. Few points of feedback:\n\n- Perhaps show the network gradients to help understand the dynamics of the learning\n- What do you think is the role of the training regime (section B.2) on the outcomes you are observing? Do you think it would be worth observing the effect of changing the learning regime (and the accompanying gradient) on the outcomes?\n- Generalization to any number of nested labels is not demonstrated\n- The empirical demonstration of contribution of skip connections is not too  \n- Corollary to above, what do you think would be the role of attention in the hierarchical representation learning?\n\nQuestions:\n1- Not clear why complementarity is a necessary condition of learning (section 3, before definition 1). Take for example the vehicle, wheels, truck example in figure 1. Learning F2 (wheels) features isn’t conditioned on correctly learning F1 (vehicle). In fact, could it be satisfactory, to first order approximation, to assume that learning finer-grained features first (roundness of wheels) and then combining lower level features (in what you refer to as Markov chain) may result in equal if not better accuracies? Is it possible to test this?\n\n2- Not clear why calibration of outputs (section “combination of nested outputs” in p 5) is an approximation of P(Y_i=q).\n\n3- Its not clear to me why the proposed rejection calibration method as a way of handling overconfidence is the right approach. Why this solution? Why not use, for example, regularization instead?\n\n4- Table 1 of results. As the distortions increases the relative improvements of the nested learner appears to increase in CIFAR-10 results. Can you demonstrate this is not an artifact of the distortion generation strategy and is indeed a stable observation?\n\n5- why is the marginal accuracy improvements so much larger for going from a budget of 1 to 2 than 2 to 5 in all cost functions? Does this not refute the claim that the nested model “gradually breaks”?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1giGrB0FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I read the author response, thank you for responding to my questions.\n\nOriginal review:\n\nThis paper presents a hierarchical learning approach that trains neural network classifiers on a known hierarchy of labels.  The experiments show that the approach makes a network more robust to distortion compared to standard end-to-end learning.  The approach in this paper and its formalization of the task are interesting, but the significance of the techniques is somewhat unclear and the experiments could be more thorough in terms of the baselines and data sets considered.\n\nFrom the related work section, the relationship between this paper and the previous work is somewhat complicated -- it is hard for a reader not deeply familiar with these previous works to understand the unique contribution made in the submission, and assess why it is significant.  For example, in the last paragraph of that section, many of the distinctions drawn between the submitted work and previous methods seem minor (including a confidence measure for a certain prediction, for example) or seemingly subjective (about whether an operation with a previous method was “natural” or could be done “transparently”).  Making crisper, less ambiguous distinctions between this work and previous work would help.\n\nLikewise, the experimental results here do not compare against any of the hierarchical learning approaches discussed in the related work section.  The results show that the paper’s approach is more robust to distortion compared to standard end-to-end learning.  However, I was unclear on why it was not appropriate to compare against the other hierarchical learning methods from previous work.  Also, if the claim is improved robustness, I feel that evaluating against adversarial training (Madry et al., ICLR 2017) or similar approaches is necessary to understand the practical relevance of these improvements.\n\nFinally, experiments that consider larger hierarchies (here, the number of target classes tends to be small, CIFAR-10 and MNIST each have ten classes, meaning the hierarchies are not very rich) would help illustrate the potential power of the techniques.\n\nMinor\nI didn’t understand the following statement, and given that it’s a fairly bold claim I would rephrase it or explain it better in the paper body rather than referring the reader to the appendix:\n“A standard DNN unknowingly uses low quality data also to train higher layers, even if there is no high level information in the data.”\n\nI don’t understand what the right arrow operator on the top of page 5 means.  From earlier statements it seems that I(f_i(X), Y_i) -> I(X, Y_i) means that the left quantity is approximately equal to the right.  But I’m not sure why to use an arrow for that rather than an \\approx symbol.  The arrow would seem to imply that the left approaches the right in the limit, but if that is what you mean you should tell us in the limit of what.  Later the arrow is used to represent links in a Markov chain, adding further confusion for me.\n\nI think it would be helpful if before Equation 1, you mentioned this holds for strictly nested Y_i’s (since earlier in the paper, Y_i referred to more general things).\n \nI assume the ECE is computed over held-out validation data (i.e., not training data)?  The paper should say this.\n\nPage 7: lineal -> linear", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model architecture and training procedure for multiple nested label sets of varying granularities and shows improvements in efficiency over simple baselines in the number of fine-grained training labels needed to reach a given level of performance.

Reviewers did not raise any serious concerns about the method that was presented, but they were also not convinced that it represented a sufficiently novel or impactful contribution to an open problem. Without any reviewer advocating for the paper, even after discussion, I have no choice but to recommend rejection.

I'm open to the possibility that there is substantial technical value here, but I think this work would be well served by more extensive comparisons and a potentially revamped motivation to try to make the case for it that value more directly.
Decision: Reject