Encoding Hierarchical Information in Neural Networks Helps in Subpopulation Shift

1 INTRODUCTION . Deep learning has been tremendously successful at image classification tasks , often outperforming humans when the training and testing distributions are the same . In this work , we focus on tackling the issues that arise when the testing distribution is shifted at a subpopulation level from the training distribution , a problem called subpopulation shift introduced recently in BREEDS ( Santurkar et al. , 2021 ) . Subpopulation shift is a specific kind of shift under the broader domain adaptation umbrella . In domain adaptation , the task of a classifier remains the same over the source and target domains , but there is a slight change in the distribution of the target domain ( Goodfellow et al. , 2016 ) . In the general setting , the target domain is a slightly changed version of the source domain . For example , an object detector that has been trained to detect objects during day-time for a self-driving car application is used to perform the same task , but now on a shifted set of night-time images . The task remains the same i.e . to identify and detect objects , but the target domain ( night-time ) is a shifted version of the source domain ( day-time ) , provided all other conditions ( such as weather , region , etc . ) remain constant . However , in the setting of subpopulation shift , both the source and target domains remain constant . The shift here occurs at a more granular level , that of subpopulations . Consider the source distribution described above , that of a self-driving car . Let ’ s say the categories for classification included small-vehicles and large-vehicles . Under small-vehicles , the source set included samples of golf-car and race-car , and under large-vehicles , the source samples were from firetrucks and double-deckerbuses . In the target domain for testing , the classes remain unaltered ; the classifier is still learning to categorize vehicles into small or large categories . However , the testing samples are now drawn from different subpopulations of each class which were not present during training , such as coupe and sedan for small-vehicles and dumpster-truck and school-bus for large-vehicles . Given that this shift occurs naturally at the subpopulation level , which is one level of hierarchy below the classes to be classified , explicitly incorporating the notion of hierarchies among categories should help make the classifier more robust to this shift . Taking inspiration from humans , we want our neural networks to learn in a systematic manner , learning coarse categories first and then successively refining them down to more fine-grained categories Skinner ( 1958 ) , also exploited in the field of curriculum learning as introduced in Bengio et al . ( 2009 ) . This allows the network to encode inter-dependencies that a model trained on a flat set of classes can not incorporate . Knowledge of these inter-dependencies helps under subpopulation shift , since the model has explicitly been made aware of a larger distribution at different levels covering many classes , and we hope that shifted unseen subpopulations will overlap with these larger covers at the higher levels . Additionally , the current way of classification , in which each class is considered separate and independent of others , treats all mispredictions as having the same impact . This is contrary to intuition , since a Husky and a Beagle are not as equidistant to each other as they are to a Bull-frog . The impact of misclassifications becomes quite important in critical use cases . The cost of mispredicting an animate object for an inanimate object can be disastrous for a self-driving car . To address this , we introduce the notion of catastrophic coefficient , which is a quantitative measure of the impact of mispredictions that follows intuitively from our hierarchical graph . It is defined as the normalized shortest path distance between the true and the predicted classes , as per the graphical structure of the underlying hierarchy . We show that incorporating hierarchical information during training reduces the catastrophic coefficient of all considered datasets , both with and without subpopulation shift . We explicitly incorporate the hierarchical information into learning by re-engineering the dataset to reflect the proposed hierarchical graph , a subset of which is sketched out in Figure 1 . We modify the neural network architectures by assigning intermediate heads ( one fully connected layer ) corresponding to each level of hierarchy , with one-hot labels assigned to the classes at each level individually , as shown in Figure 2 . We ensure that only samples correctly classified by a head are passed on for learning to the next heads ( corresponding to descendants in the hierarchy graph ) by a conditional learning mechanism . This allows the same neuron at a head to represent different classes conditioned on the previous head . We show results on a custom dataset we create out of ImageNet , and a subpopulation benchmark dataset introduced by BREEDS ( Santurkar et al. , 2021 ) . We also show results on the BREEDS dataset by keeping the hierarchical structure , but changing the target subpopulations to cover a more diverse range . We show that given a hierarchy , our learning methodology can result in both better accuracy and lower misprediction impact under subpopulation shift . To summarize , Our main contributions are as follows : • We mitigate the effect of subpopulation shift by incorporating hierarchical information in two ways : 1 ) allowing independent inference at each level of hierarchy by encoding classes at that level in a one-hot fashion and 2 ) enabling collaboration between these levels by introducing a conditional training mechanism that trains each level only on samples that are correctly classified on all previous levels . • We introduce the notion of misprediction impact , and quantify it as the shortest graphical distance between the true and predicted labels for inference . • Evaluate the performance of deep models under subpopulation shift and show that our training algorithm outperforms classical training in both accuracy and misprediction impact . • Re-engineer the ImageNet dataset to create a custom dataset that reflects inter-class hierarchies . Additionally , we show results on an existing benchmark dataset on subpopulation shift to show that our learning algorithm is compatible with any hierarchy . 2 RELATED WORK . Subpopulation Shift is a specific variant of domain adaptation where the models need to adapt to unseen data samples during testing but the samples arrive from the same distribution of the classes , changed only at the subpopulation levels . This setting differs from zero-shot learning since the classes remain the same . BREEDS ( Santurkar et al. , 2021 ) introduced the problem of subpopulation shift along with tailored benchmarks constructed from the ImageNet ( Deng et al. , 2009 ) dataset . WILDS ( Koh et al . ( 2021 ) ) provides a subpopulations shift benchmark but for toxicity classification across demographic identities . Cai et al . ( 2021 ) tackle the problem through a label expansion algorithm similar to Li et al . ( 2020 ) but tackles subpopulation shift by using the FixMatch ( Sohn et al . ( 2020 ) ) method . The algorithm uses concepts such as pseudo-labelling and consistency loss to learn visual concepts in a semi-supervised manner . Cai et al . ( 2021 ) applied the same and showed how consistency based loss is suitable for tackling the subpopulation shift problem . But these semisupervised approaches require access to the target set , albeit unlabelled , and the algorithm makes use of these unlabelled target set to further improve upon a teacher classifier . We restrict ourselves to the supervised training framework where we have no access to the target samples . Moreover , we tackle the subpopulation shift problem by incorporating hierarchical information to the models . Hierarchical Modeling is a tried and tested supervised learning strategy to learn superior concepts in vision datasets . Yan et al . ( 2015 ) introduce HD-CNN , which uses a base classifier to distinguish between coarser categories whereas for distinguishing between confusing classes , the task is pushed further downstream to the fine category classifiers . Deng et al . ( 2010 ) showed that the classification performance can be improved by leveraging semantic information as provided by the WordNet hierarchy . Deng et al . ( 2014 ) further introduced HEX graphs to capture semantic relations between two labels ( parent and children ) . Although this work relabels leaf nodes to intermediate parent nodes , they train models only on the leaf node labels ( single label ) . Our work on the other hand leverages the information at multiple levels of a hierarchy and employs a conditional training approach to link multiple labels of a single image as per the provided hierarchy . Works such as McClelland et al . ( 2016 ) and Saxe et al . ( 2013 ) tried to understand the importance of hierarchical learning from a theoretical perspective and demonstrated an implementation on a neural network based model . Jiang et al . ( 2019 ) proposes a Conditional class-aware Meta Learning framework that conditionally learns better representations through modeling inter-class dependencies . Blocks ( Alsallakh et al. , 2018 ) demonstrate how learning hierarchies is an implicit method of learning for convolutional neural networks . Song & Chai ( 2018 ) shows how multiple heads of a neural network can collaborate among each other in order reach a consensus on image classification tasks . Tree-CNN ( Roy et al. , 2020 ) tackles the incremental learning problem where the model expands as a tree to accommodate new classes . Contrary to these works , we utilize hierarchy as a way to mitigate the effect of subpopulation shift , both under accuracy and impact of mispredictions . A recent work , Bertinetto et al . ( 2020 ) introduced a similar notion of catastrophic impact on mispredictions . This work leverages the graphical distance and optimizes this directly as a part of the loss function . On the other hand , we use the misprediction distance only as an evaluation metric to quantify the impact of our conditional training framework on the degree of catastrophic predictions across domain shifts during inference . We show that our models have lower catastrophic predictions than standard models under the subpopulation shift . Additional related work has been listed in A.3 . 3 HIERARCHIES TO MITIGATE THE EFFECT OF SUBPOPULATION SHIFT . 3.1 SUBPOPULATION SHIFT . As described in Section 1 , subpopulation shift is a specific branch of the broader domain adaptation problem . In subpopulation shift we have training and testing distributions that differ at the level of subpopulations . We now introduce the required notation . Let ’ s say we are focusing on an n-way classification problem , with each class denoted by i ; i = { 1 , 2 ... n } . The data consisting of imagelabel pairs for the source and the target domain are denoted by { Xs , Ys } and { Xt , Yt } , respectively . Now we introduce notation for subpopulations . Each class i draws from s different subpopulations . The different subpopulations of class i for training ( source domain ) are denoted by Ssi and for testing ( target domain ) by Sti . We reiterate that between source and target domains , the n classes remain the same , since the classification task is unchanged . However the data drawn for each class at the subpopulation level shifts , with no overlap between the source and target subpopulations . This reflects that the subpopulations used for testing are never observed during training , i.e . Ssi ∪ Sti = ∅ .

Summary: The authors address the problem of generalization under subpopulation shift where the labels remain identical in the new domain, however, their instances belong to different subcategories than in the source domain. The authors propose infusing knowledge both about these subcategories and about super-categories of the labels as additional supervision signals. For this purpose, they define a conditional training framework along with an adapted architecture and a hierarchical loss. The evaluation results demonstrate how their framework significantly improves robustness to subpopulation shift.