Encoding Hierarchical Information in Neural Networks Helps in Subpopulation Shift
Authors: Amitangshu Mukherjee
Isha Garg
Kaushik Roy
Venue: ICLR 2022 Submitted (2022)
Keywords: Subpopulation shift
Hierarchical
Hierarchical Networks
Conditional Training
Domain Adaptation

Abstract: Over the past decade, deep neural networks have proven to be adept in image classification tasks, often even surpassing humans in terms of accuracy. However, standard neural networks often fail to understand the concept of hierarchical structures and dependencies among different classes for vision related tasks. Humans on the other hand, seem to learn categories conceptually, progressively growing from understanding high-level concepts down to granular levels of categories. One of the issues arising from the inability of neural networks to encode such dependencies within its learned structure is that of subpopulation shift -- where models are queried with novel unseen classes taken from a shifted population of the training set categories. Since the neural network treats each class as independent from all others, it struggles to categorize shifting populations that are dependent at higher levels of the hierarchy. In this work, we study the aforementioned problems through the lens of a novel conditional supervised training framework. We tackle subpopulation shift by a structured learning procedure that incorporates hierarchical information conditionally through labels. Furthermore, we introduce a notion of graphical distance to model the catastrophic effect of mispredictions. We show that learning in this structured hierarchical manner results in networks that are more robust against subpopulation shifts, with an improvement of around 2% in terms of accuracy and around 8.5% in terms of graphical distance over standard models on subpopulation shift benchmarks.

Under review as a conference paper at ICLR 2022
ENCODING HIERARCHICAL INFORMATION IN NEURAL NETWORKS HELPS IN SUBPOPULATION SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
Over the past decade, deep neural networks have proven be adept in image clas- sification tasks, often surpassing humans in terms of accuracy. However, standard neural networks often fail to understand the concept of hicrarchical structures and dependencies among different classes for vision related tasks. Humans on the other hand, seem learn categories conceptually, progressively growing from understanding high-level concepts down to granular levels of categories. One of the issues arising from the inability of neural networks to encode such depen- dencies within its learned structure is that of subpopulation shift where mod- els are queried with novel unseen classes taken from a shifted population of the training set categories. Since the neural network treats each class as independent from all others, it struggles to categorize shifting populations that are dependent at higher levels of the hierarchy. In this work, we study the aforementioned problems through the lens of a novel conditional supervised training framework. We tackle subpopulation shift by a structured learning procedure that incorporates hierarchi- cal information conditionally through labels. Furthermore, we introduce a notion of graphical distance to model the catastrophic effect of mispredictions. We show that learning in this structured hierarchical manner results in networks that are more robust against subpopulation shifts, with an improvement of around 2% terms of accuracy and around 8.5% in terms of graphical distance over standard models on subpopulation shift benchmarks.
INTRODUCTION
Deep learning has been tremendously successful at image classification tasks, often outperforming humans when the training and testing distributions are the same. In this work, we focus on tackling the issues that arise when the testing distribution is shifted at subpopulation level from the training distribution, problem called subpopulation shift introduced recently BREEDS (Santurkar et al.) 2021). Subpopulation shift is specific kind of shift under the broader domain adaptation umbrella. In domain adaptation, the task of a classifier remains the same over the source and target domains, but there is slight change in the distribution of the target domain (Goodfellow et al.) 2016). In the general setting, the target domain a slightly changed version of the source domain. For example, an object detector that has been trained to detect objects during day-time for a self-driving car application is used to perform the same task, but now on a shifted set of night-time images. The task remains the same i.e. identify and detect objects, but the target domain (night-time) is shifted version of the source domain (day-time), provided all other conditions (such as weather, region, etc.) remain constant.
However, in the setting of subpopulation shift, both the source and target domains remain constant. The shift here oceurs more granular level, that of subpopulations. Consider the source distribu- tion described above, that of self-driving car. Let's say the categories for classification included small-vehicles and large-vehicles. Under small-vehicles, the source set included samples of golf-car and race-car, and under large-vehicles, the source samples were from firetrucks and double-decker- buses. In the target domain for testing, the classes remain unaltered; the classifier is still learning to categorize vehicles into small or large categories. However, the testing samples are drawn from different subpopulations each class which were not present during training, such as coupe and sedan for small-vehicles and tumpster-truck and school-bus for large-ve

---

Under review as a conference paper at ICLR 2022
Given that this shift occurs naturally at the subpopulation level, which is one level of hierarchy be- low the classes to be classified, explicitly incorporating the notion of hierarchies among categories should help make the classifier more robust to this shift. Taking inspiration from humans, want our neural networks to learn in a systematic manner, learning coarse categories first and then suc- cessively refining them down more fine- fine-grained categories [Skinner] (1958), also exploited in the field of curriculum learning as introduced in innennio et al. (2009). This allows the network to encode cater-dependencies that model trained on a flat set of classes cannot incorporate. Knowledge of these inter-depente dependeeeesssssss helps under subpopulation shift, since the model has explicitly been made aware of a larger distribution at different levels covering many classes, and we hope that shifted unseen subpopulations will overlap with these larger covers at the higher levels.
Additionally, the current way of classification, in which each class is considered separate and inde- pendent of others, treats all mispredictions as having the same impact. This is contrary to intuitionn since: a Husky and a Beagle are not as equidistant impact of misclassifications becomes quite important in critical use cases. The cost of mispredicting an animate object for an inanimate object can be disastrous for a self-driving car. To address this, introduce the notion of catastrophic coefficient, which is a quantitative measure of the impact of mispredictions that follows intuitively from our hierarchical graph. It is defined as the normalized shortest path distance between the true and the predicted classes, as per the graphical structure of the underlying hierarchy. We show that incorporating hierarchical information during training reduces the catastrophic coefficient of all considered datasets, both with and without subpopulation shift.
each other as they are to a Bull-frog. The
We explicitly incorporate the hierarchical information into learning by re-engineering the dataset reflect the proposed hierarchical graph, subset of which sketched out in Figure [ We mod- ify the neural network architectures by assigning intermediate heads (one fully connected layer) corresponding to each level of hierarchy, with one-hot labels assigned to the classes at each level individually, as shown in Figure 2 We ensure that only samples correctly classified by a head are passed on for learning to the next heads (corresponding to descendants the hierarchy graph) by conditional learning mechanism. This allows the same neuron at head to represent different classes conditioned on the previous head. We show results on a custom datasct we create out of ImageNet, and a subpopulation benchmark dataset introduced by BREEDS (Santurkar et al. 2021I) We also show results on the BREEDS dataset by keeping the hierarchical structure, but changing the target subpopulations to cover more diverse range. We show that given hierarchy, our learning methodology can result in both better accuracy and lower misprediction impact under subpopulation shift.
To summarize, Our main contributions are as follows:
We mitigate the effect of subpopulation shift by incorporating hierarchical information in two ways: 1) allowing independent inference at each level of hierarchy by encoding classes at that level in a one-hot fashion and 2) enabling collaboration between these levels by introducing conditional training mechanism that trains each level only samples that are correctly classified on all previous levels.
We introduce the notion of misprediction impact, and quantify it as the shortest graphical distance between the true and predicted labels for inference.
Evaluate the performance of deep models under subpopulation shift and show that our train- ing algorithm outperforms classical training in both accuracy and misprediction impact.
Re-engineer the ImageNet dataset to create a custom dataset that reflects inter-class hier- archies Additionally, we show results on an existing benchmark dataset on subpopulation shift to show that our learning algorithm is compatible with any hierarchy.
2 RELATED WORK
Subpopulation Shift is a specific variant of domain adaptation where the models need to adapt to unseen data samples during testing but the samples arrive from the same distribution of the classes, changed only at the subpopulation levels. This setting differs from zero-shot learning since the classes remain the same. BREEDS (Santurkar et al. 2021) introduced the problem of subpopulation shift along with tailored benchmarks constructed from the ImageNet (Deng et al., 2009) dataset. WILDS (Koh et al. (202T) provides subpopulations shift benchmark but for toxicity classification
2

---

Under review as a conference paper at ICLR 2022
across demographic identities. Cai et al. (2021) tackle the problem through a label expansion algorithm similar to Li et al. (2020) but tackles subpopulation shift by using the Fix.Match (Soohn et al. (2020) method. The algorithm uses concepts such pseudo-labelling and consistency loss to learn visual concepts in a semi-supervised manner. Cai et al. 2202)) applied the same and showed how consistency based loss is suitable for tackling the subpopul lation shift problem. But these semi- supervised approaches require access to the target set, albeit unlabelled, and the algorithm makes use of these unlabelled target set to further improve upon a teacher classifier. We restrict ourselves to the supervised training framework where we have no access to the target samples. Moreover, we tackle the subpopulation shift problem by incorporating hierarchical information to the models.
Hierarchical Modeling is a tried and tested supervised learning strategy to learn superior concepts in vision datasets. Yan et al. (0155) introduce HD-CNN, which uses base classifier to distinguish between coarser categories whereas for distinguishing between confusing classes, the task is pushed further downstream to the fine category classifiers. Deng et al.) 2010) showed that the classification performance can be improved by leveraging semantic information as provided by the WordNet hier- archy. Deng et al. 2014) further introduced HEX graphs to capture semantic relations between two labels (parent and children). Although this work relabels leaf nodes to intermediate parent nodes, they train models only on the leaf node labels (single label). Our work on the other hand leverages the information at multiple levels of a hierarchy and employs a conditional training approach link multiple labels of a single image as per the provided hierarchy. Works such as McClelland et al. 2016) and Saxe et al. (2013) tried to understand the importance of hierarchical learning from a the- oretical perspective and demonstrated an implementation on a neural network based model. Jiang al. 2019) proposes a Conditional class-aware Meta Learning framework that conditionally learns better representations through modeling inter-class dependencies. Blocks (Alsallakh et al. 2018) demonstrate how learning hierarchies is an implicit method of learning for convolutional neural net- works. [ongn& & Chai (2018) shows how multiple heads of neural network can collaborate among each other in order reach a consensus on image classification tasks. Tree-CNN (Roy et al., 2020) tackles the incremental learning problem where the model expands classes. Contrary to these works, we utilize hicrarchy as a way to mitigate the effect of subpopulation shift, both under accuracy and impact of mispredictions.
tree to accommodate new
A recent work, Bertinetto et al. 2020) introduced a similar notion of catastrophic impact on mis- predictions. This work leverages the graphical distance and optimizes this directly as part of the loss function. On the other hand, we use the misprediction distance only as an evaluation metric to quantify the impact of our conditional training framework on the degree of catastrophic predictions across domain shifts during inference. We show that our models have lower catastrophic predictions than standard models under the subpopulation shift. Additional related work has been listed in A.3
HIERARCHIES TO MITIGATE THE EFFECT OF SUBPOPULATION SHIFT
3.1 SUBPOPULATION SHIFT
As described in Section I, subpopulation shift is a specific branch of the broader domain adaptation problem. In subpopulation shift we have training and testing distributions that differ at the level of subpopulations. We now introduce the required notation. Let's say we are focusing on an n- n-way classification problem, with each class denoted by i; i = {1, 2...... The data consisting of image- label pairs for the source and the target domain are denoted by (x*, W*) and (x', Y.) respectively.
Now we introduce notation for subpopulations. Each class i draws from s different subpopulations. The different subpopulations of class i for training (source domain) are denoted by S and for testing (target domain) by St. We reiterate that between source and target domains, the n classes remain the same, since the classification task is unchanged. However the data drawn for each class at the subpopulation level shifts, with no overlap between the source and target subpopulations. This reflects that the subpopulations used for testing are never observed during training, i.c. S U S Ø.
3.2 HIERARCHICAL VIEW TO TACKLE SUBPOPULATION SHIFT
In this work we tackle the subpopulation shift problem by explicitly incorporating hierarchical knowledge into learning via labels. Intuitively, if a neural network can grasp the concept of struc- tural hierarchies, it will not overfit to the observed subpopulations. Instead, the network will have a notion of multiple coarse-to-fine level distributions that the subpopulation belongs to. The coarser
3

---

Under review as a conference paper at ICLR 2022
Root
Superclass
Mammals
Amphibians
0
Class
Felidae
Canis
Frog
1
Salamander
Subclass
1
0
1
Label
000
001
010
011
400
401
410
411
Name
Persian
Tiger Cat
Husky
Beagle
Tree Frog
Bull Frog
Axoloti
Figure 1: Hicrarchical representation of custom subset of ImageNet. The classes for the classifi- cation task are at the intermediate level, denoted by 'class". The constituent subpopulations of each class particular classes from the ImageNet dataset and marked at the leaf level 'subclass' One-hot labels are provided at each level of the tree. The labels at the leaf nodes encode the overall hierarchical information. The colored arrows indicate the graphical distance from one leaf node to the other. This shows that mispredicting a Persian Cat as a Tiger Cat (two graph traversals) is far less catastrophic than predicting the same as an Axolotl (six graph traversals).
distributions would likely cover a much larger set of distributions, hopefully helping in generaliza- tion under shift. For instance, network trained with the knowledge that both fire-truck and race-car fall under vehicles, and a human and dog fall under living things, will not overfit to the particular subpopulation but have a notion of vehicles and living things. This will allow it to gener- alize to newer large-vehicle such school-bus and predict it as vehicle rather than living thing, since the network has learned much broader distribution Even if there is misprediction, it is more likely to be at the lower levels of hierarchy, confusing things that are less catastrophic to mispredict.
vehicles one level of hierarchy above.
3.3 VISION DATASETS AS HIERARCHICAL TREES
ImageNet (Deng et al. 2009) is a large-scale image database, which is collected on the basis of an underlying hierarchy called WordNet (Miller 1992). consists twelve different subtrees created by querying synsets from the WordNet hierarchy. To motivate the problem of subpopulation shift, we create two custom datasets from ImageNet, which are subpopulation shifted versions of each other. The datasets have balanced hierarchical structure of depth 3 as shown in Figure [ starting from coarse concepts such as mammals and amphibians at higher level, to fine grained specific subpopulations at the leaf nodes.
The hierarchical structure has 5 nodes at the highest level of superclasses, 10 nodes the class level (n 10), and each class draws from 3 subpopulations each (s 3), leading to total of 30 classes per dataset for source and target datasets each. Figure [ shows two subtrees from the dataset, and two out of the three subclasses at the leaf nodes per class. During testing under shift, the 10 classes the class level remain the same, but the 30 subpopulations that samples are drawn from are changed. Accuracy and catastrophic coefficients reported for the 10 classes, similar to BREEDS.
The custom trees are simple and balanced, capturing the hierarchical structure found in the dataset. We them to lay the foundations which implement conditional training framework.
4

---

Under review as a conference paper at ICLR 2022
Later, we show how method translates well complicated hierarchy such as the LIVING- 17 (Santurkar et al.) 202I) subpopulation shift benchmark. This illustrates that our algorithm is compatible with any hierarchy chosen according to the task of interest. Next, we show how to annotate the graphical hierarchical model to encode the concepts of hierarchy.
3.4 HIERARCHY ENCODING
Given a tree, we start from the root node and traverse downwards until the first level of hierarchy, which consists of superclasses such as mammals, fish, reptiles, etc. The custom dataset for instance has five superclasses and we label them as 0-4. Next we traverse downwards till the level of classes. These are the actual tasks that the network has to classify. In this level we capture slightly finer concepts conditional to the previous level, such as given an amphibian, is it a frog or a salamander; or given a bird, is it aquatic or aviatory. Each superclass in our custom dataset has only 2 classes, getting labeled or 1, making up the n = 10 classes for classification. Thus the categorical label of frog in our tree shown in Figure is 40, with the label encoding that it is amphibian and the overall 40 encoding that given it is an amphibian, it is a frog. Finally we reach the leaf nodes of the tree, where there three subclasses per class, annotated from 0 2. At the leaf nodes, we have specific classes from ImageNet, encoded in our level-wise concatenated format. Thus we have a one-hot encoded label for each level of our hierarchy, where each level captures the structural concept associated with the tree. The overall encoding conveys the information as a path arising from the superclasses and ending at the subclasses. More concretely, an encoding states that a Siberian Husky, annotated 010 is a mammal (0), is member of the Canis family (01) and it is a Siberian Husky (010). During training we split the encoded information into individual one-hot labels each level so that we can train a head per level using categorical tross-entropy.
3.5 CATASTROPHIC DISTANCE
In this section, we introduce the concept of catastrophic coefficient as a measure of the impact of misprediction. It is the shortest graphical distance between the true label and the predicted label in our hierarchy, normalized by the number of samples. It implicitly quantifies whether there is notion of semantic structure in the model's predictions. Subpopulation shift occurs at a lower level of a hierarchical tree where unseen subclasses are introduced during evaluation. So, if the hierarchically trained networks can grasp the concepts of superclasses and classes, the mispredictions during the shift will not be catastrophic. This is because they will tend to be correct the higher levels, and hence "closer" to the ground truth node in terms of graph traversal.
Neural networks trained via standard supervised learning have no explicit knowledge of inter-class dependencies. Thus, for flat models, mispredicting a specific sub-breed of a dog as sub-breed of a cat is as catastrophic as mispredicting the same as a specific species of snake. Graphical distance between the true and predicted classes intuitively captures the catastrophic impact of misprediction and accounts for the semantic correctness of the prediction. This serves as an additional metric to accuracy for evaluating the performance of models under subpopulation shifts. Additionally. it illus- trates that the improvement accuracy is truly due to incorporating better hierarchical information, rather than model architecture changes or the conditional training framework. It is pictorially illus- trated by the colored arrows in Figure [ A higher graphical distance between a misprediction and its ground truth signifies more catastrophic impact. We average the graph traversal distances of all predictions (= 0 if sample classified correctly) over the entire dataset and call the catastrophic coefficient, thus quantifying the impact of mispredictions for
pair.
Let Ok be the graph traversals needed for sample k in the shortest path between its true and predicted label. Let there be N samples for evaluation. Then, the catastrophic coefficient is defined Cal
9t De N We note that we use this distance just to evaluate, and not during training. Instead we just take the final classifier level predictions and run it via our graph to check for distances both models, irrespective of whether they have been shown the hierarchy or not.
3.6 ARCHITECTURE
In this subsection, we describe the modifications on standard ResNet (He et al. 2016) architectures to make them suitable for our conditional training framework. Our network makes classification decisions at each level of the hierarchy. Hence, we introduce a separate head to predict the one-hot encoded vectors each level. For e.g. if we want represent the hierarchical concept of dog
5

---

Under review as a conference paper at ICLR 2022
Batch Size 4
Batch Conditional Training
Superclass
Mommals
P[Marmmla)
* Loss
Conditional Loss
Class
Canis
Felidae
Validation Matrik,
Validation Matrik;
P|Canis | Mammald)
O
0
O
O
Loss 3
Conditional Loss
Subclass
Husky
Collie
P(Collie [Canis, Mammals]
(Mammals)
Instance Subtree
Loss
Head,
FC
Validation Matrix, [Mammal
Superclass
Conv Block
Conv Block
Conv Block
Conv Block
Loss
(Canis)
Conditional Loss 2
Fc2
o
Headz
Validation
Class
[Canier (Mammil)Mamala-]]
Collie (001)
Loss
(Collie)
T 0000
FC
Conditional Loss 3
Head,
Single Instance Conditional Training
Subclass
Figure 2: Figure shows our conditional training framework applied to multi-headed neural net- work architecture, the instance subtree shown the top left. The bottom of the figure shows conditional training for single instance of a class 'Canis', subclass 'Collie'. Each head predicts the hierarchical level-specific label from the instance subtree. The validation matrices dictate which samples get propagated for learning to the deeper heads. At the top right, the batch conditional train- ing details shown. Each element in validation matrix specifies whether the sample is correctly predicted at a previous level. Here, the training is shown till the subclass level.
as shown Figure 2
we want the Head, of our network to predict if it mammal (superclass), Headz to predict if is carnivore (class) and finally Heada to predict that it is dog (subclass).
Convolutional Neural Networks learn coarse fine features as we go deeper, capturing local con- cepts of images early and global concepts we deeper (Zeiler & Fergus] 2014). This lines up well with our hierarchical structure, and hence we connect the different heads at different depths of the model. The concept is pictorially depicted in Figure 2 Since work, the individual convolutional blocks here are residual blocks. The locations of these heads are determined experimentally. We got best results with Head, attached after the third residual block, Headz and Head, after the fourth residual blocks. This maps the flow of coarser to finer categories in our hierarchy the features needed for such classification. Our Hierarchy encoding enables reuse the same label at particular level of the hierarchy to mean different categories, conditioned on the prediction at previous levels. This joint prediction mechanism requires collaboration between the multiple heads, and we ensure this via a conditional training approach which we describe next.
Residual Networks in
3.7 CONDITIONAL TRAINING DETAILS
Here we describe the conditional training framework, which is independent of subpopulation shift. Let us assume we have 3 levels in our hierarchy, with the levels enumerated by = 1, 1,2,3. 3. Let the labels at each of these levels (treated one-hot) be denoted by y1 Let F be the neural network that we pass a batch of images to. Here X E Bxd where B is the batch-size and is the dimension of the input data. Further, let Ft be the neural network up head corresponding to predicting level of our hierarchical graph. Note that all Fi have overlap since the neural network is shared, rather than ensemble. This illustrated in Figure The loss head L1 is calculated as:
L Cross (F Vi-1
where Vi E RB refers to the validity matrix generated the the / th level, and Vo is initialized to identity. This validity matrix is what enforces our conditional training framework. It is a batch- sized (batch-size for training) binary matrix, where each element corresponding image was correctly classified by the / th head or not, respectively. Multiplying the previous head's validity matrix with the current head's loss before backpropagation ensures that
0, capturing whether the

---

Under review as a conference paper at ICLR 2022
learning for Fi only occurs samples that are meaningful at that head. In other words, V propagates only correctly classified samples, as per its name. This ensures that the prediction at the th head is not just p(yr), but p(y: F head's prediction was correct. This is done because the same neuron will represent multiple classes while training each head individually, for instance the same neuron represents 'Felidae" and 'Frog' in Figure I However, this collaboration between heads via conditional training separates each neuron as conditioned on previous prediction and allows network to learn progressively refining predictions.
it 11-1), the prediction probability given that the previous
4 EXPERIMENTAL RESULTS
In Section 3 we described in detail our training framework starting from encoding hierarchies into supervised labels to conditionally train multi-headed networks. In this section, we empiri- cally demonstrate how models trained with our approach perform better than models trained in a supervised learning setup, under the problem of subpopulation shift. We show results on three dif- ferent datasets (two custom) and one BREEDS LIVING-17 subpopulation shift benchmark. We see that both in terms of accuracy and catastrophic coefficient, our hierarchical models are superior to standard ones in tackling the subpopulation problem. To cover more diverse shift, we retain the hierarchy introduced in LIVING-17 but consider more sets of subpopulations. We call these LIVING-17-B and LIVING-17-C and talk more about them in subsection 4.5
4.1 SETUP
As mentioned, we consider subpopulation shift at the class level of a hicrarchy. Consider an n- way classification problem, with each class denoted by i; i 2..n...... - For our custom datasets, n = 10 and for the LIVING-17 dataset, n = 17. The total number of levels of hierarchy including the subpopulation levels, I, is and for our custom dataset and LIVING-17 17 respectively. Now, we create the shift by sampling subpopulations of each class i from different subpopulations. For the two different datasets, s 3 and 2 respectively. Thus, for the custom datasets we have 10-way classification problem with total of n X = 30 subpopulations, and 34 for LIVING-17. The classes are located at = 2 in our custom tree and at = 3 in LIVING- 17, and the subpopulations at 1 ==3 3 and / = 4 respectively. We show the subpopulation details of our custom dataset in Appendix A.1 and we add the graphical representation as supplementary files.
More concretely, the subpopulations for class i are distributed over S. (source) and S (target) domain. Let's consider i = [dogs}. Säng = [Bloodhound, Pekinese] and Singe = [Great-Pyreness, Papillon]. Thus the learning problem is that by training on just the source subpopulations Sdgge the model will be able to say that the target subpopulations of dogs belong to = {dogs}.
4.2 TRAINING DETAILS AND METRICS
We use accuracy and catastrophic coefficient to measure performance, both in the presence and absence of subpopulations shift. Both catastrophic coefficients and accuracies are reported under 4 different settings, where applicable, differentiated by the subseript. The prefix of the subseript determines the domain the model was trained on and the suffix denotes the domain it is evaluated on. There are 4 combinations. 's ss does not evaluate subpopulation shift, but shows the results of using our method as a general training methodology. is trained on training source domain data, and evaluated on the validation set in the source domain. shift: training is performed on source domain, and testing on target domain. We interchange source and training domain and get the equivalent remaining two settings,
t' evaluates results under subpopulation
t' and ''s'.
Throughout our experiments we focus mainly on two sets of training. which result in the Bascline and the Hierarchical Models. The Baseline models are trained in a flat manner, and evaluated as per (Snturrarr et al. (02T). The classification task is on the i classes, enumerated at the level of 'classes' mentioned in the hierarchy. The subpopulation shift occurs one level below, at the "subelass' level. The subclass labels are never shown to the network, and hence one level of hierarchy is automatically incorporated. We remove this implicit notion of hierarchy in 'Subclass Level' models by training all now the classification is between X subpopulations explic-
the way down to subpopulations, itly. The Hierarchical models, as the name suggests have been trained on the complete hierarchical information present in the tree, using our conditional training framework. We use ResNet- 18 and
7

---

Under review as a conference paper at ICLR 2022
Table I: Results on Custom Dataset
Source to Source (no shift)
Source to Target (Shift)
Acc.-s
Cat
Accs-:
Cat
Hierarchical-Ie
87.2
0.88
52.0
2.51
Baseline-18 18
84.79
0.89
49.53
2.70
Subclass Level-18
85.84
0.97
46.91
2.84
Target Target (no shift)
Target Source (shift)
Acc;-a
Cat
AcC;
Cal--a
Hierarchical-I5
81.33
1.18
58.88
2.18
Baseline-18 18
77.69
1.21
55.47
2.39
Subclass Level-18
79.11
1.28
55.31
2.47
ResNet-34 as our network architecture backbones for modifications as mentioned in subsection 3.6 For enumerating results, we use 'Hierarchical-18 denote a modified ResNet-18 model trained conditionally. Similarly "Baseline-34" signifies ResNet-34 architecture trained for flat classifica- tion on the n categories found at the level of 'classes' in the hierarchy. We train the models on each dataset for 20 epochs with a batch size of 32, starting with learning rate of 0.1, and a 10 fold drop every 40 epochs thereafter. We do not use data augmentation on our custom datasets. All models have been trained three random seeds each and the mean numbers are reported.
4.3 RESULTS ON CUSTOM DATASETS
In this section, we discuss the results on the Custom Datasets, shown in Table [ For the custom datasets, train our hierarchical models on the specific subpopulation classes that they have knowledge of all the levels in the hierarchy. Acc,-t and Cat.- denote the accuracy and catas- trophic coefficient of the model during the t shift. As shown Table the Hierarchical models perform better than the flat Baseline and Subclass level models in terms of accuracy and catas- trophic coefficient both in the presence and absence under shift. The models trained conditionally have 3% improvement in terms of accuracy and 7.03% and 8.7% improvement in terms of catastrophic coefficient over the flat bascline class level models under shift for the two custom sets (s f and s) respectively.
shift. The performance gap is significant
4.4 RESULTS ON BREEDS LIVING-17
In this section we discuss the results on the BREEDS LIVING-17 dataset, enumerated in Table 2 This subpopulation shift benchmark, introduced in (Santurkar al. 2021). captures finer details of hierarchy that encode richer relationships between the entities. We show that our methodology is applicable complex hierarchies and outperforms flat baseline models both Acc,-, and Cal.,t- Contrary to our custom datasets, in which we let the conditional models train down to the specific subpopulations, here we follow the BREEDS evaluation procedure and train all models only till the class level, never exposing models to subpopulation level labels. We train each architecture and model on five random seeds each and report the mean numbers. Contrary BREEDS, we report numbers without bootstrapping, but follow all their other byper-parameters. Hierarchical models achieve an improvement of around 2% and 1.5% terms of accuracy and around 8.5% and 7.36% in terms of catastrophic coefficient for ResNet- 18 and 34 respectively, as seen in Table
Additional Experiments. We perform experiments on two additional sets of models, the 'Subclass Level' models, which are the flat models trained with subpopulation level labels, and the 'SSeaartedd' models. For the separated model, we remove the collaborative conditional training and train each head separately, keeping everything else the same. It achieves Acc,-t and Cat values of 56% and 1.72 respectively compared to 59.02% and 1.60 for the hierarchical models. The performance drop justifies the use of validity matrices and conditional training for collaborative learning.
The 'Subclass Level' model has Acc,- and Cat, values of 54.63% and 1.83 respectively. Addi- tionally, as seen in Table 1 Subclass model with explicit information of hierarchy has much
8

---

Under review as a conference paper at ICLR 2022
Table 2: Results on BREEDS LIVING-17 with and without shift
Model
Acc--s
Cat--s
Acc,-t
Cat,-
Subclass-18
91.59
0.36
53.71
1.86
Baseline-Ie 18
92.3
0.33
57.02
1.75
Hicrarchical-I 18
92.1
0.33
59.04
1.60
Baseline-34
92.56
0.33
59.82
1.63
Hierarchical-35
92.32
0.32
61.32
1.51
Table 3: Results BREEDS LIVING-17 source, shifted LIVING-17-B and -C target
Model
Acc,-((B)
(B)
Acc,-e(C')
Cat,-r(C)
Baseline-18 18
53.54
1.97
53.04
1.93
Hierarchical-18
54.35
1.86
54.14
1.81
Baseline-34
55.86
1.84
55.52
1.812
Hierarchical-34
55.92
1.79
55.62
1.77
worse shift performance than the baseline (single level hierarchy) and hierarchical (multi-level hier- archy) model, confirming that knowledge of hierarchy is helpful under subpopulation shift.
4.5 RESULTS ON SHIFTED LIVING-17
To understand the adverse effects of varying subpopulations create two more shifted versions of the Target set LIVING-17, LIVING-I7-B and LIVING-I7-C by varying the S subclasses. Acc,-1(B) denote the model accuracy for the shift t from set A to B. As be seen from Table Hierarchical-I models have much better accuracy and catastrophic coefficients than the BREEDS baseline models for both shifted sets. This shows that our way of imparting hierarchical knowledge helps deep models to adapt to various adverse degrees of the subpopulation shift problem.
CONCLUSION
In this paper, we target the problem of subpopulation shift, which specific kind of shift under the broader umbrella of domain adaptation. Here, the subpopulations that make up the categories for the classification task change between training and testing. For instance, the testing distribution may contain new breeds of dogs not seen during training, but all samples will be labeled "dog". We note an implicit notion of hierarchy in the framing of the problem itself; in the knowledge of all constituent subpopulations sharing the common immediate ancestry. In line with this, extend the notion of hierarchy and make explicit to better tackle the issue of subpopulation shift. We consider the underlying hierarchical structure of vision datasets, in the form of both and a benchmark dataset for subpopulation shift. We incorporate this information explicitly into training via labeling each level with individual one-hot label. We then encourage collaboration between multiple heads of a model via our conditional training framework, wherein each head is only trained on samples that were correctly classified all levels before the present one. We further introduce a metric to capture the notion of semantic correctness of predictions. It uses the shortest graphical distance between the misprediction and the true label as per the hierarchy quantify the catastrophic impact of mispredictions. We show that our hierarchy- aware conditional training setup outperforms flat baselines by around ~ (2 3)% in terms accuracy and (7 8.5)% in terms of catastrophic coefficient over standard models custom and subpopulation shift benchmarks.
custom subset
REFERENCES
Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, and Mario Marchand. Domain- adversarial neural networks. ArXiv, abs/1442.4444. 2014.

---

Under review as a conference paper at ICLR 2022
Bilal Alsallakh, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. Do convolutional neural networks lear class hierarchy? IEEE Transactions on Visualization and Computer Graphics, 2018.
Martín Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. ArXiv, 0et///907.029,, 2019.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando C Pereira. Analysis of representations for domain adaptation. In NeurIPS, 2006.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML,, 2009.
Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A. Lord. Making better mistakes: Leveraging class hierarchies with deep networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Tianle Cai, Ruiqi Gao, J. Lee, and Qi Lei. A theory of label propagation for subpopulation shift, In ICML, 2021.
Tao Chen, Shijian Lu, and Jiayuan Fan. Ss-henn: Semi-supervised hierarchical convolutional neural network for image classification. IEEE Transactions on Image Processing, 2019.
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. NeurIPS, 2020.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamoniy. Optimal transport for do- main adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Jia Deng, Alexander C. Berg, Li, and Li Fei-Fei. What does classifying more than 10, 000 image categories tell us? In ECCV, 2010.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs In ECCV, 2014.
Ankit Dhall. Learning representations for images with hierarchical labels. ArXiv, abs/2004.00909, 2020.
Ankit Dhall, Anastasia Makarova, Octavian-Eugee Ganea, Dario Pavllo, Michael Greeff, and Andreas Krause. Hierarchical image classification using entailment cone embeddings. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darell. Decaf: A deep convolutional activation feature for generic visual recognition. In JCML, 2014.
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. ArXiv, abs//409, 10W14.99799 2015.
Yaroslav Ganin, F. Ustinova, Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversenn training of neural networks. The Journal of Machine Learning Research, 2016.
Muhammad Ghifary, W. Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. IEEE International Conference on Computer Vision (ICCV), 2015.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sentiment classification: deep learning approach. In ICML. 2011.
10

---

Under review as a conference paper at ICLR 2022
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kemnel for unsupervised domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Mingming Gong, Kun Zhang. Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. JMLR workshop and conference proceedings, 2016.
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT Press, 2016.
Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object recognition:
An unsupervised approach. International Conference on Computer Vision. 2011.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Xiang Jiang, Mohammad Havaci, Farshid Varno, Gabriel Chartrand, Nicolas Chapados, and Stan Matwin. Learning to learn with conditional class dependencies. In ICLR, 2019.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund. Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Wei hua Hu, Michihiro Yasunaga, Richard Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. ICML, 2021.
Brendan Kolisnik, Isaac Hogan, and Farhana H. Zulkernine. Condition-cnn: hierarchical multi- label fashion image classification model. Expert Syst. Appl., 2021.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In ICML, 2020.
Bo Li, Yezhen Wang, Tong Che, Shanghang Zhang, Sicheng Zhao, Pengfei Xu, Wei Zhou, Yoshua Bengio, and Kurt Keutzer. Rethinking distributional matching based domain adaptation. ArXiv, abs/2006. 13352, 2020.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization. ArXiv, abs/1710.03465; 2018.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. ArXiv, abs/1502.02791 2015.
James L. McClelland, Zahra Sadeghi, and Andrew M. Saxe. A critique of pure hierarchy: Uncover- ing cross-cutting structure in a natural dataset. 2016.
George A. Miller. Wordnet: lexical database for english. Commun. ACM, 1992.
Hieu Pham, Tung T. Le, Dat Thanh Ngo, Dat Q. Tran, and Ha Q. Nguyen. Interpreting chest x-rays via cnns that exploit hierarchical disease dependencies and uncertainty labels, 2021.
Yanyun Qu, Li Lin, Fumin Shen, Chang Lu, Yang Wu, Yuan Xie, and Dacheng Tao. Joint hierarchi- cal category structure learning and large-scale image classification. IEEE Transactions on Image Processing, 2017.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off- the-shelf: An astounding baseline for recognition. IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014.
Deboleena Roy, Priyadarshini Panda, and Kaushik Roy. Tree-cnn: A hierarchical deep convolutional neural network for incremental learning. Neural networks: : the official journal of the International Neural Network Society, 2020.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, 2010.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. (BREEDS): Benchmarks for subpop- ulation shift. In International Conference Learning Representations, 2021.
11

---

Under review as a conference paper at ICLR 2022
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Learning hierarchical categories in deep neural networks. Cognitive Science, 2013.
B. F. Skinner. Reinforcement today. In American Psychologist, 1958.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. ArXiv, abs/2001 100605, 2020.
Guocong Song and Wei Chai. Collaborative learning for deep neural networks. In NeurlPS, 2018.
Salma Taoufiq, Balázs Nagy, and Csaba Benedek. Hierarchynet: Hierarchical enn-based urban building classification. Remote Sensing, 2020.
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cnn: Hierarchical deep convolutional neural networks for large scale visual recognition. IEEE International Conference on Computer Vision, 2015.
Hao-Tong Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards theoretical framework of out-of--istribuuee generalization. ArXiv, abs/2106.04496, 04446,, 2021.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Yu Zheng, Jianping Fan, Ji Zhang, and Xinbo Gao. Hierarchical learning of multi-task sparse metrics for large-scale image classification. Pattern Recognition, 2017.
Xinqi Zhu and Michael Bain. B-enn: Branch convolutional neural network for hierarchical classifi- cation. ArXiv, 09990,, 2017.
A APPENDIX
A.1
CUSTOM DATASET
In this section, we provide detailed description the subpopulations of our custom datasets. We also provide the entire graph structure separate supplementary files.
Custom Dataset
Class
Source
Target
Felidae
Persian Cat, Tiger, Leopard
Egyptian Cat, Siamese Cat, Chee-
tah
Canis
White Wolf, Siberian Husky, beagle
Eskimo dog. Border Collie. Pomeranian
Small Fish
lionfish, puffer, goldfish
garfish, anemone fish, tench
Big Fish
Hammerhead, Tiger Shark, Great
Grey Whale, Killer Whale, Stingray
White Shark
Salamander
common newt, European fire sala- mander, spotted salamander
axolotl, eft, spotted salamander
Frog
bullfrog, tree frog, tailed frog
bullfrog, tree frog, tailed frog hognose snake, rock python, green mamba
Snakes
Indian cobra, boa constrictor, green snake
Non Snake
African crpcodile, mud turtle, Ko- modo Dragon
American alligator, leatherback tle. Gila monster
Aviatory Bird
vulture, bald eagle, hummingbird
great gret owl, kite, redshank
Aquatic Bird
flamingo, black swan, king penguin
albatross, goose, pelican
As can be seen there 10 classes in the set and under each class, there are subpopulations. Since these classes are obtained from the ImageNet (Deng et al.] (009) dataset, the total train dataset size 1500.
1300 30 39000 and the validation size (s or
12

---

Under review as a conference paper at ICLR 2022
As seen from Table A.1 there is partial overlap between the S Sslamanner and St Satamender and complete overlap between frog and frog This is due to the fact that the ImageNet dataset does have anymore Amphibian subpopulation classes. Thus, for fairness, while performing the (t (t-8) 8) or (s f) shift experiments remove the amphibian subtree from the notion which consists of the classes, {frogs, salamanders). Thus, the shift experiments a 8-way classification problem rather than 10 way classification problem during (s or (t t).
A.2 HIBRARCHICAL LABELS FOR BREEDS LIVING-17
We present the hierarchical labels for LIVING-17 17 dataset here and we have added the graphical structure supplementary file.
LIVING-17 Labels
Class
Source
Target
Labels
salamander
eft, axoloti
common newt, spotted salamander
100
turtle
box turtle, leatherback turtle
loggerhead, mud turtle
210
lizard
whiptail, alligator lizard
African chameleon, banded gecko
220
snake
night snake, garter snake tarantula, black and gold garden spider
sea snake, boa constrictor
200
spider
garden spider, wolf spider
300
grouse
ptarmigan, prairic chicken
ruffed grouse, black grouse
010
parrot
macaw, lorikeet
African grey, sulphur-crested cock-
000
atoo
crab
Dungeness crab, fiddler crab
rock crab, king crab
310
dog
bloodhound, Pekinese
Great Pyrenees, papillon
410
wolf
coyote, red wolf
white wolf, timber wolf
411
Fox
grey fox, Arctic fox
red fox, kit fox
412
domestic cat
tiger cat, Egyptian cat
Persian cat, Siamese cat
413
bear
sloth bear, American black bear
ice bear, brown bear
414
beetle
dung beetle, rhinoceros beetle
ground beetle, long-horned beetle
320
butterfly
sulphur butterfly, admiral
cabbage butterfly, ringlet
321
ape
gibbon, orangutan
gorilla, chimpanzee
400
monkey
marmoset, titi
spider monkey, howler monkey
401
A.3 ADDITIONAL RELATED WORK
Domain Adaptation and its variants are a well studied set problems deep learning. One direction of works (Ben- David et al. (2006) Saenko et al. (2010), Ganin & Lempitsky 2015), Courty ct al. (2017). Gong et al. (2016), Donahue et al. (2014), Razavian et al. ))014)) is aimed at tackling the problem of adapting target domains by learning domain itself. Another line of work aims to match the source and target distributions the feature space (Glorot al. (001I). Ajakan al. (2014). Long al. (2015). Ganin et al.) (2016)). The main motive behind these set of works is to tackle out-of-support domain adaptation tasks by sharing common representation between the two. To adapt to newer environments, deep models are trained gradually to make them more suitable for transition to these newer environments (Gopalan et al. (201T) Gong et (2012). Glorot et al. (20IT), Kumar al.) (2020), Chen et (2020)). Domain generalization enables the use of multiple different environments during training, but requires having a prior knowledge on the target distribution (Ghifary et al. (2015), [i et al. (2018), [Arjossyy et al. more specific problem of distribution
selective set of samples from the target
(011),, Ye et al. (2021)). We the other hand, focus shift, wherein the shift occurs at a subpopulation level in the target domain.
Hierarchical Modeling Dhall et al. (2020) and Dhall (220)) show how an image classifier aug- mented with hierarchical information based on entailment cone embeddings outperforms flat clas- sifiers Entomological Dataset. B-CNN (Zhu & Bain 2017) learns multi-level concepts via a branch training strategy on low dimensional datasets. Condition CNN (Kolisnik et al.] 202T) improves upon the training time of B-CNN and shows improved classification results Kaggle
13

---

Under review as a conference paper at ICLR 2022
Fashion Product Images dataset. Pham et al. (2021) takes advantage of the relationship between diseases in chest X-rays to learn conditional probabilities through image classifiers. [Tauffiq et al. (2020) adapts a similar approach learn urban structural relationships. The following works that tackle the problem of learning hierarchical concepts on large scale image datasets ( (Zheng et al. (2017) and Qu et al. (2017)). SS-HCNN (Chen al., (019) applies semi-supervised approach to learn cluster level concepts at higher level of a hierarchy and categorical features at leaf node levels. However, these works do not focus on the problem of subpopulation shift.
A.4 LEARNING MODELS
We introduce the different learning strategies we train the neural networks on. We show results sets of models, described below.
These are the primary models for evaluating subpopulation shift. The classification task on the classes, enumerated at level of 'classes' mentioned in the hier- archy. The subpopulation shift occurs one level below, at the 'subclass" level. The subclass labels are never shown to the network, and hence one level of hierarchy is automatically incorporated. The results shown in (Santurkar et al. 202T) follow this methodology.
Baseline Models
Hierarchical Models These are models resulting from training with our conditional frame- work described in this work. As the name suggests, the models have been trained on the complete hierarchical information present in the tree. Thus these models have the concepts of all levels of the hierarchy i. superclasses, classes and subclasses. This is a collaborative way of knowledge representation where you do not need separate models each trained on separate levels of the hierarchy.
Subclass Level Models - These models are trained on the individual subpopulation classes, for example on given Sings Bloodhound, Pekinese]. These models lack even the one level of hierarchy available to class level models. Strictly speaking, the shift in distribution between source and target is no longer subpopulation shift, since we are already exposing the subpopulation labels to the network, making them the classification classes. Here, the shift is more general evaluation of what happens when the models are queried with domain shifted examples of each class.
Separated Models These models have the concepts of hierarchy the hierarchical mod- els but they are not trained under the conditional training framework. The individual heads mentioned in the subsection a.e all trained separately with no conditional loss or valid- ity matrix. In subsection 4.4 we show that such models perform poorly despite of having knowledge of the underlying hierarchy. Hence, these models corroborate the need for con- ditional training under our hierarchical setup, irrespective of whether subpopulation shift occurs or not.
A.5 RE-WEIGHING SOFTMAX PROBABILITIES TO TARGET IMBALANCE OF LIVING-17
Although the BREEDS LIVING-17 17 dataset catches the rich hierarchical relationships among differ- ent entities in the tree, it suffers from the issue of dataset imbalance at various levels of the tree. As can be seen from the graph for LIVING-17 attached as supplementary data, there is only one sala- mander class under the superclass amphibian, whereas under the mammal superclass there are seven classes. This kind imbalance noticed at other parts of the tree as well. To mitigate this problem, re-weigh the soft-max weights of each of our heads. Since specific head is responsible for the prediction of particular level of our tree, re-weighing each softmax probability at a particular head does not interfere with the re-weighing of the same at different head. Through the re- weighing, we want to give more weightage to the nodes that are responsible to output classes that occur less in the dataset and less weightage to the nodes which predict the more frequently occurring classes.
We do this by using the following formula, W, ΣX_1 7kk Here the numerator sums the total number of samples in the dataset, making the normalization factor for inverse proportionality. In the denominator, the term n, refers to the number of samples for each class. And c refers to the total 44200
cene
number of classes for a particular head. For example, Wo 5e 160000 refers to the soft-max node responsible to predict the label 0 at the third level of the BREEDS LIVING-17 17 hierarchical tree. The numerator represents the dataset size. In the denominator, 5 represents the number of classes (c) at
14

---

Under review as a conference paper at ICLR 2022
the third level of the tree. The number 11 2600 represents the number of samples with label O. The label 0 appears 11 times at the third level as per our hierarchy encoding. 2600 the number of samples per class such salamander, grouse, etc each of which has label O.
A.6 LIVING-17-B AND C
It is very difficult to incorporate all subpopulation classes in particular dataset. To understand the adverse effects of varying subpopulations we create two more shifted versions of the Target set of LIVING-17, IVING-17-B and LIVING-I7-C by varying the Sf subclasses. We do this either by adding disjoint subclasses of the ImageNet (Deng et al. 2009) or by creating different combinations of the existing S with new disjoint subclasses. We reuse some of the S subclasses due to the unavailability of the same the ImageNet database. All the subpopulations of {wolf} from the ImageNet database have already been covered in the Swotf and Siott set, just reuse the Swott in the sets B and C.
We present the hierarchical labels for extended LIVING-17 dataset here.
LIVING-17 Labels
Class
Source
Target
Labels
salamander
European fire salamander, spotted salamander
European fire salamander, common
100
newt
turtle
terrapin, loggerhead
terrapin, mud turtle
210
lizard
Komodo dragon,
American
frilled lizard, common iguana
220
chameleon
snake
green snake, Indian cobra
sidewinder, horned viper
200
spider
barn spider, black widow
barn spider, black widow
300
grouse
partridge, quail
partridge, quail
010
parrot
African grey, sulphur-crested cock-
African grey, sulphur-crested cock-
000
atoo
atoo
crab
hermit crab, king crab
hermit crab, king crab
310
dog
dalmatian, otterhound
Australian terrier, German shep-
410
herd
wolf
white wolf, timber wolf
white wolf, timber wolf
411
fox
red fox, kit fox
red fox, kit fox
412
domestic cat
tabby, cougar
cheetah, tiger
413
bear
koala, brown bear
ice bear, koala
414
beetle
tiger beetle, leaf beetle
tiger beetle, leaf beetle
320
butterfly
monarch, lycaenid
monarch, lycaenid
321
ape
siamang, baboon
siamang. baboon
400
monkey
proboscis monkey, squirrel monkey
colobus, macaque
401
15


Reviews:
[{"review_id": "krt_H82Z2O_", "reviewer": "Reviewer_TcMm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n - The problem is of great interest to the vision community.\n - Paper is well motivated, and method is technical sound.\n\n**Weaknesses**\n - The proposal hierarchical classification method is not completely novel. For instance, [1] [2] proposed similar approaches / architectures for the hierarchical classification problem. \n* I have a particular concern of the per-level parameter sharing design. This design requires the class hierarchy be perfectly balanced (i.e. each parent class at the same level has the same number of children classes). The authors may want to comment on how the proposed approach apply to these non-perfect but practically-common situations.\n - Experimental results are insufficient.  It would be more convincing if the authors include comparison with other hierarchical classification approaches in the literature. Also, limiting the method to subpopulation shift problems seem unnecessary.\n\n\n[1] HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition\n\n[2] Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks", "summary_of_the_paper": "This paper presents a new hierarchical classification method. Instead of using a flat classifier layer for all classes, the proposed approach trains a neural network with a hierarchy of classifier layers (attached to different blocks of the DNN), where each classifier layer corresponds to a specific level in the class hierarchy, and models the conditional classification probability given classes at parent level. The authors use the proposed method to tackle the subpopulation shift problem. Empirical results on a custom ImageNet dataset and the BREEDS dataset show improvements over baseline approaches.", "summary_of_the_review": "Overall, contributions of the paper seem small. The proposal approach is not extremely novel and is limiting in practical settings (non-balanced class hierarchy). Experimental results lack comparison with literature hence not convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7pUzG9goSRF", "reviewer": "Reviewer_GFoK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper is well written. The architecture and evaluation methods are clearly explained and the research problem is well motivated.\n- The proposed architecture is simple and computationally inexpensive which means it could be easy applied to any object recognition problem.\n- On the datasets tested the authors demonstrated a performance benefit from using their training method.\n\nWeaknesses:\n- The proposed mulitheaded architecture is only evaluated on datasets containing images and a label hierarchy of living things. It may be the case that this limited range of object types have properties such as similar degrees of intraclass variability and the presence of distinct textures compared to other object types. Further experiments are required to prove that there are performance benefits to using this architecture.\n- I would like to see more discussion and experiments evaluating the impact of the particular class hierarchy chosen for your custom dataset on your new models performance. As a baseline, what happens if subclasses are assigned to class groups randomly, what if the hierarchy is determined by visual features instead of semantic meaning (ex. feathers, scales, fur)? More experiments in this direction would provide a more convincing argument that your performance improvements are not only a result of the particular set of classes and hierarchy structure that you chose to use.\n- In table 1, why is the source to source and target to target accuracy lower for Baseline-18 than it is for Subclass Level-18? Shouldn't we expect the opposite trend?\n- Is there past work you could cite that supports the statement: \"Humans on the other hand, seem to learn categories conceptually, progressively growing from understanding high-level concepts down to granular levels of categories.\" Couldn't humans also learn fine-grained categories first and then build them up into more abstract groupings?", "summary_of_the_paper": "1. Introduces a new multi headed architecture for explicitly incorporating class hierarchy into training.\n2. Created a new metric to measure the impact of a miss-prediction according to class hierarchy tree distance\n3. Created a new label hierarchy for a subset of ImageNet\n3. Evaluated the performance of their new architecture under subpopulation shift", "summary_of_the_review": "Overall, this paper demonstrates some promise that there could be an advantage to explicitly specifying a label hierarchy during training. More experiments are necessary to demonstrate this method really works more generally on a wider variety of object types and label hierarchies.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7oShBkxrmZf", "reviewer": "Reviewer_1XvT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the ideas presented are clear and straightforward to understand, and the evaluation results convincing.\nThe results are unsurprising given how the custom setup better addresses the goal of generalizing to subpopulation shift.\nIt is still informative to have a formal quantification of such generalization potential, although, such contribution renders the work as a limited experiment to address a custom problem or evaluate a specific question rather than a major contribution.\n\nTo this extent, my main concern is about the novelty of the presented work. In fact, several aspects claimed to be novel have been addressed several time in the literature over the past decade.\nIn particular, the idea of quantifying \"misprediction impact\" / \"catastrophic coefficient\" is very similar to the “hierarchical cost” by Deng et al. (ECCV 2010), the followup Hierarchy and Exclusion graphs (ECCV 2014). \nAlso, the proposed architecture is similar in spirit to HD-CNN (Yan et al. CVPR’15).\nI expected the authors to reference the above pieces of work and to compare their solution and results against them.\nThe authors did reference other related pieces of work, again, without comparing against them (dismissing the need by mentioning that \"Contrary to these works, we utilize hierarchy as a way to mitigate the effect of subpopulation shift\"). Nevertheless, these pieces share several aspects of the proposed solution even if the exact tasks they address are slight different.\n\nWith that, the current contribution is rather incremental. I would have deemed it otherwise had the authors provided theoretical insights about the problem of hierarchy, (e.g. as in the work by McClelland, Saxe and others).\n\nReferences:\n\nDeng, J., Berg, A. C., Li, K., & Fei-Fei, L. “What does classifying more than 10,000 image categories tell us?.\" European conference on computer vision. Springer, Berlin, Heidelberg, 2010.\n\nDeng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H. and Adam, H. “Large-scale object classification using label relation graphs” In European conference on computer vision (pp. 48-64). Springer, 2014.\n\nMcClelland, J. L., Z. Sadeghi, and A. M. Saxe. \"A Critique of Pure Hierarchy: Uncovering Cross-Cutting Structure in a Natural Dataset.\" Neurocomputational Models of Cognitive Development and Processing: Proceedings of the 14th Neural Computation and Psychology Workshop. 2017.\n\nSaxe, Andrew M., James L. McClellans, and Surya Ganguli. \"Learning hierarchical categories in deep neural networks.\" Proceedings of the Annual Meeting of the Cognitive Science Society. Vol. 35. No. 35. 2013.\n\nYan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., & Yu, Y. “HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition”. In Proceedings of the IEEE international conference on computer vision (pp. 2740-2748), 2015.\n\nLanguage issues:\n- at a the subpopulation\n- One hot labels => One-hot labels\n- if a neural networks\n- are lesser catastrophic => less\n- Each heads predicts\n- Our networks makes \n- a batchsized binary matrix => unclear\n- are superior than standard => superior to\n- at level of => at the level of\n- the the notion\n- subclasses from the ImageNet => of ImageNet\n- We now introduce some notation. => avoid vague statements involving “some”.\n\nFormatting issues:\n- Then the catastrophic coefficient, is defined => Then, …\n- individually , => no space\n- End all sentences with a full stop (e.g. the bullet points on page 2).\n- for Toxicity classification => why capitalized?\n- (Koh et al. (2021)) => avoid nested parentheses.", "summary_of_the_paper": "The authors address the problem of generalization under subpopulation shift where the labels remain identical in the new domain, however, their instances belong to different subcategories than in the source domain. The authors propose infusing knowledge both about these subcategories and about super-categories of the labels as additional supervision signals. For this purpose, they define a conditional training framework along with an adapted architecture and a hierarchical loss. The evaluation results demonstrate how their framework significantly improves robustness to subpopulation shift. ", "summary_of_the_review": "The contribution is incremental, and many of the claimed novelty overlaps with the existing work (e.g. the references listed in the main review).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies subpopulation shift in object recognition when classes obey a hierarchy. It proposes an architecture, a relevant metric and a dataset (subset of imagenet).  The problem of classification in hierarchical label spaces is important and of great interest, and the effect on domain shift is interesting. Naturally, this problem was studied quite intensively over the years. 

Reviewers were concerned that the current proposal was not placed well enough in context of previous literature, both in terms of the method and in terms of experimental results.  Also, the paper would be strengthen if it provides more theoretical analysis about how the hierarchy helps with the domain shift. The authors addressed some of these issues in the rebuttal, adding references and highlighting the differences from previous methods, but the paper would need more time to make the proper experimental comparisons with previous work and subsequent analysis. As a result, the paper is still not ready for acceptance to ICLR in its current form.
Decision: Reject