Encoding Hierarchical Information in Neural Networks Helps in Subpopulation Shift

1 INTRODUCTION . Deep learning has been tremendously successful at image classification tasks , often outperforming humans when the training and testing distributions are the same . In this work , we focus on tackling the issues that arise when the testing distribution is shifted at a subpopulation level from the training distribution , a problem called subpopulation shift introduced recently in BREEDS ( Santurkar et al. , 2021 ) . Subpopulation shift is a specific kind of shift under the broader domain adaptation umbrella . In domain adaptation , the task of a classifier remains the same over the source and target domains , but there is a slight change in the distribution of the target domain ( Goodfellow et al. , 2016 ) . In the general setting , the target domain is a slightly changed version of the source domain . For example , an object detector that has been trained to detect objects during day-time for a self-driving car application is used to perform the same task , but now on a shifted set of night-time images . The task remains the same i.e . to identify and detect objects , but the target domain ( night-time ) is a shifted version of the source domain ( day-time ) , provided all other conditions ( such as weather , region , etc . ) remain constant . However , in the setting of subpopulation shift , both the source and target domains remain constant . The shift here occurs at a more granular level , that of subpopulations . Consider the source distribution described above , that of a self-driving car . Let ’ s say the categories for classification included small-vehicles and large-vehicles . Under small-vehicles , the source set included samples of golf-car and race-car , and under large-vehicles , the source samples were from firetrucks and double-deckerbuses . In the target domain for testing , the classes remain unaltered ; the classifier is still learning to categorize vehicles into small or large categories . However , the testing samples are now drawn from different subpopulations of each class which were not present during training , such as coupe and sedan for small-vehicles and dumpster-truck and school-bus for large-vehicles . Given that this shift occurs naturally at the subpopulation level , which is one level of hierarchy below the classes to be classified , explicitly incorporating the notion of hierarchies among categories should help make the classifier more robust to this shift . Taking inspiration from humans , we want our neural networks to learn in a systematic manner , learning coarse categories first and then successively refining them down to more fine-grained categories Skinner ( 1958 ) , also exploited in the field of curriculum learning as introduced in Bengio et al . ( 2009 ) . This allows the network to encode inter-dependencies that a model trained on a flat set of classes can not incorporate . Knowledge of these inter-dependencies helps under subpopulation shift , since the model has explicitly been made aware of a larger distribution at different levels covering many classes , and we hope that shifted unseen subpopulations will overlap with these larger covers at the higher levels . Additionally , the current way of classification , in which each class is considered separate and independent of others , treats all mispredictions as having the same impact . This is contrary to intuition , since a Husky and a Beagle are not as equidistant to each other as they are to a Bull-frog . The impact of misclassifications becomes quite important in critical use cases . The cost of mispredicting an animate object for an inanimate object can be disastrous for a self-driving car . To address this , we introduce the notion of catastrophic coefficient , which is a quantitative measure of the impact of mispredictions that follows intuitively from our hierarchical graph . It is defined as the normalized shortest path distance between the true and the predicted classes , as per the graphical structure of the underlying hierarchy . We show that incorporating hierarchical information during training reduces the catastrophic coefficient of all considered datasets , both with and without subpopulation shift . We explicitly incorporate the hierarchical information into learning by re-engineering the dataset to reflect the proposed hierarchical graph , a subset of which is sketched out in Figure 1 . We modify the neural network architectures by assigning intermediate heads ( one fully connected layer ) corresponding to each level of hierarchy , with one-hot labels assigned to the classes at each level individually , as shown in Figure 2 . We ensure that only samples correctly classified by a head are passed on for learning to the next heads ( corresponding to descendants in the hierarchy graph ) by a conditional learning mechanism . This allows the same neuron at a head to represent different classes conditioned on the previous head . We show results on a custom dataset we create out of ImageNet , and a subpopulation benchmark dataset introduced by BREEDS ( Santurkar et al. , 2021 ) . We also show results on the BREEDS dataset by keeping the hierarchical structure , but changing the target subpopulations to cover a more diverse range . We show that given a hierarchy , our learning methodology can result in both better accuracy and lower misprediction impact under subpopulation shift . To summarize , Our main contributions are as follows : • We mitigate the effect of subpopulation shift by incorporating hierarchical information in two ways : 1 ) allowing independent inference at each level of hierarchy by encoding classes at that level in a one-hot fashion and 2 ) enabling collaboration between these levels by introducing a conditional training mechanism that trains each level only on samples that are correctly classified on all previous levels . • We introduce the notion of misprediction impact , and quantify it as the shortest graphical distance between the true and predicted labels for inference . • Evaluate the performance of deep models under subpopulation shift and show that our training algorithm outperforms classical training in both accuracy and misprediction impact . • Re-engineer the ImageNet dataset to create a custom dataset that reflects inter-class hierarchies . Additionally , we show results on an existing benchmark dataset on subpopulation shift to show that our learning algorithm is compatible with any hierarchy . 2 RELATED WORK . Subpopulation Shift is a specific variant of domain adaptation where the models need to adapt to unseen data samples during testing but the samples arrive from the same distribution of the classes , changed only at the subpopulation levels . This setting differs from zero-shot learning since the classes remain the same . BREEDS ( Santurkar et al. , 2021 ) introduced the problem of subpopulation shift along with tailored benchmarks constructed from the ImageNet ( Deng et al. , 2009 ) dataset . WILDS ( Koh et al . ( 2021 ) ) provides a subpopulations shift benchmark but for toxicity classification across demographic identities . Cai et al . ( 2021 ) tackle the problem through a label expansion algorithm similar to Li et al . ( 2020 ) but tackles subpopulation shift by using the FixMatch ( Sohn et al . ( 2020 ) ) method . The algorithm uses concepts such as pseudo-labelling and consistency loss to learn visual concepts in a semi-supervised manner . Cai et al . ( 2021 ) applied the same and showed how consistency based loss is suitable for tackling the subpopulation shift problem . But these semisupervised approaches require access to the target set , albeit unlabelled , and the algorithm makes use of these unlabelled target set to further improve upon a teacher classifier . We restrict ourselves to the supervised training framework where we have no access to the target samples . Moreover , we tackle the subpopulation shift problem by incorporating hierarchical information to the models . Hierarchical Modeling is a tried and tested supervised learning strategy to learn superior concepts in vision datasets . Yan et al . ( 2015 ) introduce HD-CNN , which uses a base classifier to distinguish between coarser categories whereas for distinguishing between confusing classes , the task is pushed further downstream to the fine category classifiers . Deng et al . ( 2010 ) showed that the classification performance can be improved by leveraging semantic information as provided by the WordNet hierarchy . Deng et al . ( 2014 ) further introduced HEX graphs to capture semantic relations between two labels ( parent and children ) . Although this work relabels leaf nodes to intermediate parent nodes , they train models only on the leaf node labels ( single label ) . Our work on the other hand leverages the information at multiple levels of a hierarchy and employs a conditional training approach to link multiple labels of a single image as per the provided hierarchy . Works such as McClelland et al . ( 2016 ) and Saxe et al . ( 2013 ) tried to understand the importance of hierarchical learning from a theoretical perspective and demonstrated an implementation on a neural network based model . Jiang et al . ( 2019 ) proposes a Conditional class-aware Meta Learning framework that conditionally learns better representations through modeling inter-class dependencies . Blocks ( Alsallakh et al. , 2018 ) demonstrate how learning hierarchies is an implicit method of learning for convolutional neural networks . Song & Chai ( 2018 ) shows how multiple heads of a neural network can collaborate among each other in order reach a consensus on image classification tasks . Tree-CNN ( Roy et al. , 2020 ) tackles the incremental learning problem where the model expands as a tree to accommodate new classes . Contrary to these works , we utilize hierarchy as a way to mitigate the effect of subpopulation shift , both under accuracy and impact of mispredictions . A recent work , Bertinetto et al . ( 2020 ) introduced a similar notion of catastrophic impact on mispredictions . This work leverages the graphical distance and optimizes this directly as a part of the loss function . On the other hand , we use the misprediction distance only as an evaluation metric to quantify the impact of our conditional training framework on the degree of catastrophic predictions across domain shifts during inference . We show that our models have lower catastrophic predictions than standard models under the subpopulation shift . Additional related work has been listed in A.3 . 3 HIERARCHIES TO MITIGATE THE EFFECT OF SUBPOPULATION SHIFT . 3.1 SUBPOPULATION SHIFT . As described in Section 1 , subpopulation shift is a specific branch of the broader domain adaptation problem . In subpopulation shift we have training and testing distributions that differ at the level of subpopulations . We now introduce the required notation . Let ’ s say we are focusing on an n-way classification problem , with each class denoted by i ; i = { 1 , 2 ... n } . The data consisting of imagelabel pairs for the source and the target domain are denoted by { Xs , Ys } and { Xt , Yt } , respectively . Now we introduce notation for subpopulations . Each class i draws from s different subpopulations . The different subpopulations of class i for training ( source domain ) are denoted by Ssi and for testing ( target domain ) by Sti . We reiterate that between source and target domains , the n classes remain the same , since the classification task is unchanged . However the data drawn for each class at the subpopulation level shifts , with no overlap between the source and target subpopulations . This reflects that the subpopulations used for testing are never observed during training , i.e . Ssi ∪ Sti = ∅ .

Summary: 1. Introduces a new multi headed architecture for explicitly incorporating class hierarchy into training. 2. Created a new metric to measure the impact of a miss-prediction according to class hierarchy tree distance 3. Created a new label hierarchy for a subset of ImageNet 3. Evaluated the performance of their new architecture under subpopulation shift