Year: 2021

Abstract:
Despite the impressive performance of deep networks in vision, language, and healthcare, unpredictable behaviors on samples from the distribution different than the training distribution cause severe problems in deployment. For better reliability of neural-network-based classifiers, we define a new task, natural attributebased shift (NAtS) detection, to detect the samples shifted from the training distribution by some natural attribute such as age of subjects or brightness of images. Using the natural attributes present in existing datasets, we introduce benchmark datasets in vision, language, and medical domain for NAtS detection. Further, we conduct an extensive evaluation of prior representative out-of-distribution (OOD) detection methods on NAtS datasets and observe an inconsistency in their performance. To understand this, we provide an analysis on the relationship between the location of NAtS samples in the feature space and the performance of distanceand confidence-based OOD detection methods. Based on the analysis, we split NAtS samples into three categories and further suggest a simple modification to the training objective to obtain an improved OOD detection method that is capable of detecting samples from all NAtS categories.

heading: 1 INTRODUCTION
text: Deep learning has significantly improved the performance in various domains such as computer vision, natural language processing, and healthcare (Bojarski et al., 2016; Mikolov et al., 2013; Esteva et al., 2016). However, it has been reported that the deep classifiers make unreliable predictions on samples drawn from a different distribution than the training distribution (Amodei et al., 2016; Hendrycks & Gimpel, 2017; Nguyen et al., 2015). Detection of unreliable predictions is important to build a robust model but it is relatively hard when the test distribution is gradually shifting due to a natural attribute since it is difficult to determine whether the classifier fails when the shift is not significant. These shifts occur in the real-world as a result of a change in specific attribute. For example, a clinical text-based diagnosis classifier trained in 2021 will gradually encounter increasingly shifted samples as time flows, since writing styles change and new terms are introduced in time. Detection of such samples is a vital task especially in safety-critical systems, such as autonomous vehicle control or medical diagnosis, where wrong predictions can lead to dire consequences. To this end, we take a step forward by proposing a new task of detecting samples shifted by a natural attribute (e.g., age, time) that can easily be observed in the real-world setting. We refer to such shifts as Natural Attribute-based Shifts (NAtS), and the task of detecting them as NAtS detection.
Detection of NAtS is both different from, and also more challenging than out-of-distribution (OOD) detection (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018; Chandramouli & Sageev, 2020), which typically evaluates the detection methods with a clearly distinguished in-distribution (ID) samples and OOD samples (e.g., CIFAR10 as ID and SVHN as OOD, which have disjoint labels). In contrast, we aim to detect samples from a natural attribute-based shift within the same label space. Since NAtS samples share more features with the ID than the typical OOD samples do, identifying the former is expected to be more challenging than the latter. Although OOD detection has some relevance to NAtS detection, comprehensive evaluation of the existing OOD detection methods on the natural attribute-based shift is an unexplored territory. Therefore, in this paper, we perform an extensive evaluation of representative OOD methods on NAtS samples.
Depending on the task environment, NAtS detection can be pursued in parallel to domain generalization (Seo et al., 2020; Gulrajani & Lopez-Paz, 2020; Carlucci et al., 2019), which aims to overcome domain shifts (e.g., image classifier adapting to sketches, photos, art paintings, etc.). For example, an X-ray-based diagnosis model should detect images of unusual brightness so that the X-ray ma-
chine can be properly configured, and the diagnosis model can perform in the optimal setting. In other cases, domain generalization can be preferred, such as when we expect the classifier to be deployed in a less controlled environment (e.g., online image classifier) for non-safety critical tasks.
In this paper, we formalize NAtS detection to enhance the reliability of real-world decision systems. Since there exists no standard dataset for this task, we create a new benchmark dataset in the vision, text, and medical domain by adjusting the natural attributes (e.g., age, time, and brightness) of the ID dataset. Then we conduct an extensive evaluation of representative confidence- and distance-based OOD methods on our datasets and observe that none of the methods perform consistently across all NAtS datasets. After a careful analysis on where NAtS samples reside in the feature space and its impact on the distance- and confidence-based OOD detection performance, we identify the root cause of the inconsistent performance. Following this observation, we define three general NAtS categories based on two criteria: the distance between NAtS samples and the decision boundary, the distance between NAtS samples, and the ID data. Finally, we conduct an additional experiment to demonstrate that a simple modification to the negative log-likelihood training objective can dramatically help the Mahalanobis detector (Lee et al., 2018), a distance-based OOD detection method, generalize to all NAtS categories. We also compare our results with various baselines and show that our proposed modification outperforms the baselines and is effective across the three NAtS datasets.
In summary, the contributions of this paper are as follows: • We define a new task, Natural Attribute-based Shift detection (NAtS detection), which aims to
detect the samples from a distribution shifted by some natural attribute. We create a new benchmark dataset and provide them to encourage further research on evaluating NAtS detection.
• To the best of our knowledge, this is the first work to conduct a comprehensive evaluation of the OOD detection methods on shifts based on natural attributes, and discover that none of the OOD methods perform consistently across all NAtS scenarios.
• We provide novel analysis based on the location of shifted samples in the feature space and the performance of existing OOD detection methods. Based on the analysis, we split NAtS samples into three categories.
• We demonstrate that a simple yet effective modification to the training objective for deep classifiers enables consistent OOD detection performances for all NAtS categories.

heading: 2 NATURAL ATTRIBUTE-BASED SHIFT DETECTION
text: We now formalize a new task, NAtS detection, which aims to enhance the reliability of real-world decision systems by detecting samples from NAtS. We address this task in the classification problems. Let DI = {X ,Y} denote the in-distribution data, which is composed of N training samples with inputs X = {x1, ..., xN} and labels Y = {y1, ..., yN}. Specifically, xi ∈ Rd represents a d-dimensional input vector, and yi ∈ K represents its corresponding label where K = {1, ...,K} is a set of class labels. The discriminative model fθ : X → Y learns with ID dataset DI to assign label yi for each xi. In the NAtS detection setting, we assume that an in-distribution sample consists of attributes, and some of the attributes can be shifted in the test time due to natural causes such as time, age, or brightness.
When a particular attribute A (e.g., age), which has a value of a (e.g., 16), is shifted by the degree of δ, the shifted distribution can be denoted as DA=a+δS = {X ′,Y ′}. X ′ = {x′1, ..., x′M} and Y ′ = {y′1, ..., y′M} represents the M shifted samples and labels respectively. Importantly, in the NAtS setting, although the test distribution is changed from the ID, the label space is preserved as K, which is the set of class labels in DI. In the test time, the model fθ might encounter the sample x′ from a shifted data DA=a+δS , and it should be able to identify that the attribute-shifted sample is not from the ID.

heading: 3 NATS DATASET DESCRIPTION
text: In this section, we describe three benchmark datasets which have a controllable attribute for simulating realistic distribution shifts. Since there exists no standard dataset for NAtS detection, we create new benchmark datasets using existing datasets by adjusting natural attributes in order to reflect real-world scenarios. We carefully select datasets from vision, language, and medical domains containing natural attributes (e.g., year, age, and brightness), which allows us to naturally split the
samples. By grouping samples based on these attributes, we can induce natural attribute-based distribution shifts as described below.
Image. We use the UTKFace dataset (Zhang et al., 2017) which consists of over 20, 000 face images with annotations of age, gender, and ethnicity. As shown in Figure 1, we can visually observe that the facial images vary with age. Therefore, we set the 1, 282 facial images of 26 years old age as DI. For creating the NAtS dataset, we vary the age of UTKFace dataset. To obtain an equal number of samples in the NAtS dataset, the age groups that has less than 200 images are merged into one group until it has 200 samples. Finally, 15 groups DageS are produced for the NAtS datasets, varying the ages from 25 to 1 (i.e., Dage=25S ,D age=24 S , . . . ,D age=1 S ).
Text. We use the Amazon Review dataset (He & McAuley, 2016; McAuley et al., 2015) which contains product reviews from Amazon. We consider the product category ”book” and group its reviews based on the year to reflect the distributional shift across time. We obtain 9 groups with each group containing reviews from the year between 2005 and 2014. Then, the group with 24, 000 reviews posted in 2005 is set as DI, and the groups with reviews after 2005 as DyearS (i.e., Dyear=2006S ,D year=2007 S , . . . ,D year=2014 S ). Each D year S group contains 1500 positive reviews and 1500 negative reviews. We observed that as we move ahead in time, the average length of a review gets shorter and it uses more adjectives than previous years. Due to the space constraint, we provide a detailed analysis of the dataset in the Section B of the Appendix.
Medical. We use the RSNA Bone Age dataset (Halabi et al., 2019), a real-world dataset that contains left-hand X-ray images of the patient, along with their gender and age (0 to 20 years). We consider patients in the age group of 10 to 12 years for our dataset. To reflect diverse X-ray imaging set-ups in the hospital, we varied the brightness factor between 0 and 4.5 and form 16 different dataset DbrightnessS (i.e., D brightness=0.0 S ,D brightness=0.2 S , . . . ,D brightness=4.5 S ), and each group contains Xray images of 200 males and 200 females. Figure 1 presents X-ray images with different levels of brightness with realistic and continuous distribution shifts. In-distribution data DI is composed of 3, 000 images of brightness factor 1.0 (unmodified images).

heading: 4 CAN OOD DETECTION METHODS ALSO DETECT NATS?
text: In this section, we briefly discuss about OOD detection methods and conduct an extensive evaluation of OOD detection methods on our proposed benchmark datasets.

heading: 4.1 OOD DETECTION METHODS
text: In this work, we use three widely-used post-hoc and modality-agnostic OOD detection methods. We use maximum of softmax probability (MSP) (Hendrycks & Gimpel, 2017) and ODIN (Liang et al., 2018) as confidence-based OOD detection baselines, and Mahalanobis detector (Lee et al., 2018) as distance-based OOD detection baseline. Note that ODIN and Mahalanobis detector assume the availability of OOD validation dataset to tune their hyperparameters. However, for all our experiments, we use variants of the above methods that do not access the OOD validation dataset as Hsu et al. (2020). The exact equations and details of how each OOD detection method assigns an OOD score to a given sample is provided in Section A of the Appendix.

heading: 4.2 EXPERIMENTS AND RESULTS
text: We now systematically evaluate the performance of the three OOD detection methods under NAtS. We report the AUROC of all OOD detection methods averaged across five random seeds, evaluated for all NAtS datasets. The details of the implementation are described in Section C of the Appendix.
Experimental Settings. In image domain, we train a gender classification model on our UTKFace NAtS dataset using ResNet18 model and the cross-entropy loss. We use our Amazon Book Review
NAtS dataset in text domain and train a 4-layer Transformer with the cross-entropy loss for the sentiment classification task. Lastly, in medical domain, we use our RSNA Bone Age NAtS dataset and train a ResNet18 with cross-entropy loss to predict the gender given the hand X-ray image of the patient. We then evaluate the trained model on the corresponding test set in image, text, medical domains, respectively. Further, we evaluate the NAtS detection performance of representative OOD detection methods in image, text, and medical domains on their corresponding NAtS datasets, which gradually shift with age, year, and brightness, respectively.
Results. We present the classification accuracy of the trained models on the ID test set in Table 1. We observe that the models trained using the cross-entropy loss obtain high accuracy and perform well on their corresponding tasks. We further demonstrate the effectiveness of the existing representative OOD detection methods on our benchmark datasets in Figure 2. The higher AUROC indicates that more NAtS samples are distinguished from the ID samples using OOD detection methods. To provide the reference point, we calculate the AUROC values where there is no shift in the attribute. We observe that in the UTKFace NAtS dataset, samples are detected by ODIN and MSP, which are confidence-based methods, but not by Mahalanobis detector (Figure 2a). In the Amazon Review dataset, NAtS samples are detected only by the Mahalanobis detector, while MSP and ODIN fail (Figure 2b). Moreover, the scores of confidence-based methods are lower than 50 in AUROC. Lastly, Figure 2c shows that inputs from NAtS in the RSNA Bone Age dataset are detected well by all three methods.

heading: 5 ANALYZING INCONSISTENCY OF OOD DETECTION METHODS
text: In this section, we first study the behavior of NAtS samples in three datasets using PCA visualization. Then we analyze the inconsistent performance of the OOD detection methods by considering them in two categories, namely confidence-based and distance-based methods. Lastly, based on the analysis, we conclude this section by defining three NAtS categories.

heading: 5.1 ANALYSIS OF THE LOCATION OF NATS SAMPLES
text: As illustrated in Figure 3, we apply principal component analysis (PCA) on the feature representations obtained from the penultimate layer of the models to visualize the movement of NAtS samples as we monotonically increase the degree of attribute shift (i.e., age, year, and brightness). Further, Figure 4 presents the model’s prediction confidence across varying degrees of the attribute shift.
Image. By gradually changing the age, NAtS samples move toward the space between the two clusters of ID samples (i.e., the decision boundary) as can be seen in Figure 3 [Top]. Further, Figure 4a demonstrates that confidence decrease as we increase the degree of attribute shift, which indicates that NAtS samples move close to the decision boundary. Note that the majority of the NAtS samples still overlap with ID sample clusters as we change the age.
Text. As shown in Figure 3 [Middle], NAtS samples gradually move away from the ID samples (and away from the decision boundary) as the year changes. In contrast to the UTKFace dataset, the confidence gradually increases, as shown in Figure 4b, since the NAtS samples are getting far away from the decision boundary.
Medical. Figure 3 [Bottom] demonstrates that when we increase the brightness, NAtS samples move to the middle of the two classes and also move towards the outer edge of the ID sample clusters. Furthermore, as shown in Figure 4c, the relatively decreased confidence indicates that NAtS samples are placed near the decision boundary as we increase the brightness of the images.

heading: 5.2 COMPARISON BETWEEN CONFIDENCE-BASED AND DISTANCE-BASED OOD DETECTION
text: Confidence-based methods. Figure 2 and Figure 3 illustrate that confidence-based methods achieve high AUROC when the NAtS samples are near the decision boundary due to their low confidence. In the image and medical domain, we observed the NAtS samples moving towards the decision boundary with increasing attribute shift, thus they are detected by the confidence-based methods. In contrast, in the text domain, the high prediction confidence of the shifted NAtS samples causes the degradation in the AUROC of the confidence-based methods. Therefore, we conclude that to effectively utilize the confidence-based methods in all three NAtS datasets, it is necessary to reduce the confidence of samples outside the ID, namely, enforce NAtS samples to move near the decision boundary, which is not always possible (e.g., Amazon Review dataset).
Distance-based methods. From Figure 2 and Figure 3, we observe that the distance-based OOD detection method (i.e., Mahalanobis Detector) achieves high AUROC when NAtS samples are sufficiently away from the ID samples. In the text and medical domains, the Mahalanobis detector worked well since NAtS samples moved sufficiently away from the ID samples as the shift increased. However, in the image domain, the method fails to detect NAtS samples because instead of deviating from the ID, they move intermediately between the classes.
Prior works (Luo et al., 2020; Liu et al., 2017) report that the cross-entropy loss cannot guarantee a sufficient inter-class distance. In other words, representations do not need to be far from the decision boundary to lower the cross-entropy loss. In this regard, we assume that the performance degradation of the Mahalanobis detector is caused by the cross-entropy loss learning latent features that are not separable enough to detect the NAtS samples located between the classes (e.g., Figure 3 [Top]). Specifically, if some classes are located nearby in the feature space, samples moving between classes (i.e., the case of the UTKFace dataset) will not be far from the ID. Even though NAtS samples move away from one of the ID class cluster, they will gradually get closer to another ID class cluster.

heading: 5.3 NATS CATEGORIZATION
text: Considering the performance of confidence-based and distance-based detection methods, we now divided NAtS into three categories based on two criteria: 1) whether the dataset is near the decision boundary or not; 2) whether they are close or far from the in-distribution dataset. Since a dataset (i.e., samples) far from the decision boundary and overlapping with the in-distribution dataset is not a distributionally shifted dataset, our work focuses on the remaining three cases that cover all possible scenarios of NAtS. Without loss of generality, we use a classification task with three classes as a motivating example depicted by Figure 5.
• NAtS category 1: This category comprises of NAtS samples that are located near the decision boundary, and between ID samples of different classes. Purple Stars in Figure 5 represents samples from this category. Such samples are easily detected by confidence-based methods, MSP and ODIN, but harder to be detected by Mahalanobis Detector, which is a distance-based method.
• NAtS category 2: This category consists of NAtS samples that are placed away from the decision boundary and the ID data. For example, Gray Diamonds in Figure 5. Mahalanobis detector regards such samples as NAtS, whereas confidence-based methods fail to detect them since NAtS samples have a higher prediction confidence (i.e., higher ID score) than ID samples.
• NAtS category 3: This category mainly comprises of NAtS samples located anywhere near the decision boundary but far away from ID data. For example, Orange Triangles in Figure 5. Such samples are easily detected by both distance-based and confidence-based OOD detection methods.

heading: 6 METHOD FOR CONSISTENT NATS DETECTION PERFORMANCE
text: In this section, we suggest a modification in the training objective for deep classifiers to encourage consistent NAtS detection performance on all NAtS categories. Then we provide experiment results where the proposed method was compared against diverse OOD detection methods on the three NAtS datasets (UTKFace, Amazon Review, RSNA Bone Age).

heading: 6.1 METHOD
text: For a generally applicable OOD detection method to all NAtS categories, we suggest a new training objective for deep classifiers comprised of classification loss (LCE), distance loss (Ldist) and entropy loss (Lentropy). The proposed objective improves the performance of the Mahalanobis detector on NAtS samples from category 1 without sacrificing performance on NAtS samples from other categories. We focus on improving the distance-based OOD detection method rather than the confidence-based method since it is not always possible to enforce NAtS samples to be embedded near the decision boundary, as discussed in Section 5.2.
The proposed training loss is defined as:
Ltotal = LCE + Ldist + Lentropy. (1)
Note that for classification loss, we use the standard cross-entropy loss, but other losses such as focal loss can also be used. The distance loss is used to increase the distance between distinct class distributions of ID samples so that NAtS samples have larger space to move around without overlapping with ID clusters, especially in NAtS category 1:
Ldist = λ1 1( K 2 ) ∑ l ∑ k ̸=l ∥µl − µk∥2√ D , (2)
where λ1 < 0 is a hyperparameter, K is the number of target classes, D is the dimension of feature representation in the latent space (typically the penultimate layer), and µl is the mean vector of the features of samples with label l. Since the value of the vector norm increases as the dimension of the feature space increases, we normalize the distance by the square root of the feature dimension.
As will be discussed in Section D of Appendix, we discovered after some initial experiments that the distance loss often made the model use a very limited number of latent dimensions to increase the distance between class mean vectors, which degraded the NAtS detection performance. In other words, adding only Ldist to LCE caused the latent feature space to collapse into a very small number of dimensions (i.e. rank-deficient), which caused all NAtS samples to be embedded near the ID samples. Therefore, we add entropy loss to increase the number of features used to represent samples.
The entropy loss is defined as:
Lentropy = λ2 D∑
i Var(zi) + λ3 1( D 2 ) ∑ i ∑ j ̸=i C2ij , (3)
where λ2 > 0 and λ3 > 0 are hyperparameters, z ∈ RD is the feature representation in the latent feature space, Var(·) is variance, and Cij is the correlation coefficient between ith and jth dimension of the feature space. Specifically, Cij is described as:
Cij = Cov (zi, zj)
σ(zi)σ(zj) , (4)
where Cov(·) and σ(·) are covariance and the standard deviation, respectively. Intuitively, the first term in equation 3 encourages each latent dimension to have diverse values, therefore preventing the latent feature space from collapsing into a confined space. With the first term alone, however, all latent dimensions might learn correlated information, thus making the latent space rank-deficient. Therefore, we use the second term in equation 3 to minimize the correlation between different latent dimensions. Note that minimizing the feature correlation was also used in previous works under different contexts such as self-supervised learning (Zbontar et al., 2021).

heading: 6.2 RESULTS AND DISCUSSION
text: To demonstrate the effectiveness of the suggested method, we train all classifiers using the standard cross-entropy loss and our modified loss and compare post-hoc OOD detection methods across three NAtS datasets. Specifically, we present the results of the confidence-based (i.e., MSP and ODIN) and the distance-based methods (i.e., Mahalanobis distance). We also include a recently proposed OOD detection method (Chandramouli & Sageev, 2020) which computes channel-wise correlations in CNN with the gram matrix and estimates the deviation of the test samples from the training samples to detect the OOD samples. 1 Although this method uses the distance in the channel correlation space, we expect it to behave more similarly to the Mahalanobis detector than confidence-based methods, namely MSP and ODIN. We also compare with another recent baseline which exploits the energy score to detect OOD samples (Liu et al., 2020). As the method leverage the logit layer to calculate the energy score, and the softmax score is based on the logit values, we conjecture that the energy score demonstrates detection ability similar to the confidence-based methods.
1However, in the text domain, the notion of gram matrix is vague, and hence we do not compare against this baseline.
For a fair comparison, we use only the penultimate layer for evaluating the Mahalanobis detector and Gram matrix since our method is developed based on the analysis of the NAtS samples in the penultimate layer feature space. We also provide results of the ensemble version that utilize all layers in Section E of the Appendix. We describe the details of experiment setups and selecting the values for λ1, λ2 and λ3 in the Appendix Section C, where the three values can be reasonably chosen without any explicit NAtS validation datasets. We also present an ablation study to investigate the effect of different terms in the proposed loss in Section D of the Appendix. We also compare with other recent baselines that suggest other training objective for OOD detection in Section E of the Appendix.
As shown in Table 1, the in-distribution classification accuracy of the model trained with our suggested loss is comparable to that of the model trained with the cross-entropy loss. Further, we present the NAtS detection performance of the baselines and our method in Table 3. As we expected above, the gram matrix achieves performance similar to the Mahalanobis detector, demonstrating the low AUROC in the UTKFace dataset. Interestingly, even though the energy score is calculated based on
the logit values, which are more tempered than the softmax score, it shows NAtS detection performance similar to that of MSP, which is a confidence-based method. These results show that similar to MSP, ODIN, and Mahalanobis, Gram matrix and Energy-based methods also have performance inconsistency on NAtS datasets. We report the additional NAtS detection performance on four other metrics, which are often used in the OOD detection community in Section F of Appendix.
Also, it is readily visible that our proposed training objective makes the Mahalanobis detector a robust NAtS detection method for all NAtS categories. In UTKFace, we can see the dramatic NAtS detection performance increase for Mahalanobis detector. And for the other two datasets, the proposed loss does not decrease the NAtS detection performance of Mahalanobis detector. Note that ODIN is more sensitive to OOD samples than Mahalanobis detector for some brightness levels in the RSNA Bone Age dataset, but it shows inconsistent performance across all three datasets.

heading: 7 RELATED WORK
text: Extensive evaluation on distribution shifts. Recent works have leveraged the distribution shifts for OOD detection and uncertainty estimation. Rabanser et al. (2019) primarily focus on detecting the entire shifted distribution, not on detecting a single shifted sample. Yaniv et al. (2019) quantify the predictive uncertainty and investigate the quality of the calibration under the dataset shift. Engkarat & Takayuki (2019) aims to predict the classification accuracy of a shifted distribution by utilizing the average OOD score of the distribution. Hsu et al. (2020) also consider the distribution shift with preserved label space. However, Hsu et al. (2020) focus on the shifted distribution from different domain (e.g., real images vs. sketches ). To the best of our knowledge, none of these works conduct an in-depth analysis on the performance of existing OOD detection methods on samples shifted based on natural attributes.
OOD detection methods. Post-hoc OOD detection methods (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018; Chandramouli & Sageev, 2020) that utilize the classification models to obtain the OOD scores have achieved remarkable performance. All of these works aim to detect shifted sample which might affect decision making system in terms of confidence and hidden representation. Other OOD detection approach is to train the generative model (Ren et al., 2019; Choi et al., 2018; Mahmood et al., 2020) on the training distribution and estimate the density of OOD samples in test time. While these approaches are viable, it is not directly related with the downstream task but aims to detect features which is different from training distribution. In this paper, we mainly focus on methods that utilize the classification models to detect OOD samples since we are mainly interested in samples that affect decision-making systems.
Model uncertainty. A number of previous works measure model uncertainty using various methods such as Bayesian neural networks (Blundell et al., 2015), Monte Carlo dropout (Gal & Ghahramani, 2016), and deep ensembles (Lakshminarayanan et al., 2017). Note that technically, model uncertainty can be used to detect NAtS samples, especially for NAtS categories 1 and 3, since sampling model weights from the function space can be seen as redrawing the decision boundary, and NAtS samples in categories 1 and 3 will be affected heavily by this process. Model uncertainty, however, aims to capture the uncertainty in the model weights rather than detecting OOD samples, making the two rather independent research directions.

heading: 8 CONCLUSIONS
text: To enhance the reliability of decision-making systems, we define a new task, Natural Attributebased Shift (NAtS) detection, that aims to detect the samples shifted by a natural attribute. We introduce NAtS detection benchmark datasets by adjusting the natural attributes present in the existing datasets. Through extensive evaluation of existing OOD detection methods on NAtS datasets, we observe inconsistent performance depending on the nature of NAtS samples. Then, we analyze the inconsistency by probing the relationship between the location of NAtS samples and the performance of existing OOD detection methods. Based on this observation, we suggest a simple remedy to help Mahalanobis OOD detection method to have consistent performance across all NAtS categories. We hope our dataset and task inspire fellow researchers to investigate practical methods for identifying NAtS, which is crucial for deploying the prediction models in real-world systems.
ETHICS STATEMENT
We believe this is not applicable to us.
REPRODUCIBILITY STATEMENT
We describe the implementation and training details along with the details on computation resources in Section C of the Appendix. We submit our source codes along with the instructions to reproduce our results in the Appendix.

heading: A Details of OOD detection methods 13
text: A.1 Maximum of Softmax Probability (MSP) . . . . . . . . . . . . . . . . . . . . . 13 A.2 ODIN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 A.3 Mahalanobis Detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

heading: B Analysis on the Text Dataset 14
text: C Implementation Details 15 C.1 Training Details and Computing Infrastructure . . . . . . . . . . . . . . . . . . 16 C.2 Hyperparameter Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 C.3 Links for Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

heading: D Ablation Studies on Distance and Entropy Loss 17

heading: E Comparison with Additional Baselines 19

heading: F Quantitative Results with Other Metrics 20

heading: G Additional PCA Analysis 25

heading: H Quantitative Results 26

heading: I Experiments on a Multi-class Synthetic Dataset 27

heading: A DETAILS OF OOD DETECTION METHODS
text: In this section, we describe the three post-hoc and task-agnostic OOD detection methods in detail, focusing mainly on their formulation and how each method assigns an OOD score to an input sample.
A.1 MAXIMUM OF SOFTMAX PROBABILITY (MSP)
In this method, the maximum of softmax probability is considered as confidence score (Hendrycks & Gimpel, 2017). Formally, we calculate the maximum softmax probability as follows:
SMSP(x) = max c
exp ( z(c) )∑C j=1 exp ( z(j)
) , where C is the number of target classes, c is the index of a class, and z(j) denotes jth attribute of the feature in the logit layer.
A.2 ODIN
ODIN (Liang et al., 2018) utilized two well established techniques, namely temperature scaling and input preprocessing to increase the difference between softmax scores of in-distribution and OOD samples. Temperature scaling was originally proposed in Hinton et al. (2015) to distill the knowledge in neural networks and was later adopted widely in classification tasks to calibrate confidence of prediction(Guo et al., 2017). In addition to temperature scaling, the input is preprocessed in order to increase the softmax score of given input by adding small perturbations which are obtained by back-propagating the gradient of the loss with respect to the input. More specifically, ODIN is computed as follows:
S(x;T ) = max c
exp ( z(c)/T )∑C j=1 exp ( z(j)/T
) , where T ∈ R+ is the temperature scaling parameter, C is the number of target classes, c is the index of class, and z(j) denotes jth attribute of the logit layer features of input x. During training, T is set to 1.
For OOD detection, the input is first pre-processed as follows:
x̃ = x − ε sign (−∇x logS(x;T )) ,
where ε represents the magnitude of perturbation.
Next, the network calculates the calibrated softmax score of the preprocessed input as follows:
SODIN(x;T ) = max c
exp ( z̃(c)/T )
∑C j=1 exp ( z̃(j)/T ) , where z̃(j) denotes jth attribute of the logit layer features of the preprocessed input x̃ .
Lastly, the modified softmax score is compared to a threshold value δ. If the score is greater than the threshold, then the input is classified as ID sample and otherwise OOD. Originally, T , ε, and δ are hyperparameters and are selected such that the false positive rate (FPR) at true positive rate (TPR) 95% is minimized on the validation OOD dataset. However, the performance saturates when T is greater than 1000 and therefore, in general, a large value of T is preferred. Following this, in this paper, we fix T = 1000 for our experiments.
A.3 MAHALANOBIS DETECTOR
To obtain Mahalanobis distance-based OOD score (Lee et al., 2018) of a sample, we calculate the mahalanobis distance from the clusters of classes to the sample. Then, the distance from the closest class is chosen as the confidence score.
Specifically, the Mahalanobis score of an input x is defined as Smahala(x) = ∑
l αl max c − ( f lθ (x)− µc,l )⊤ Σ−1l ( f lθ (x)− µc,l ) ,
where c and l are the class and layer index, respectively, f lθ is the l th layer’s feature representation of an input x, µc,l and Σl are their class mean vector and tied covariance of the training data, correspondingly.
Note that ODIN and Mahalanobis Detector assume the availability of OOD validation dataset. However, some recent works (Shafaei et al., 2019; Techapanurak et al., 2020) report that this assumption limits the OOD detection generalizability since a model is biased towards an OOD validation set. In response, this paper validate the performance of OOD methods in the version that does not require to tune with OOD validation dataset. We perform the performance of ODIN as Hsu et al. (2020). We do not perform Mahalanobis Detector ensembling over on all layers with the optimal linear combination which requires explicit OOD data. Instead, we perform two version that use only the penultimate layer of hidden representation and sum uniformly over all layers. Therefore, for all our experiments, we use modified OOD detection methods that do not require the OOD validation dataset.

heading: B ANALYSIS ON THE TEXT DATASET
text: In this section, we provide a detailed analysis of the text dataset. We use the Amazon Review (He & McAuley, 2016; McAuley et al., 2015) dataset. We consider the product category ”book” and conducted an analysis to see the impact of time on product reviews. We then performed an analysis to see the impact of time on the length of the reviews. Figure 6 presents a comparison between density plot of review length for each year from 2006 to 2014. We observe that as we move ahead in time,
the length of the reviews gradually reduces. Further, Figure 7 presents the distribution of the average ratio of important words in a sequence by year. First, we train a model to classify sentiment polarity using Catboost (Prokhorenkova et al., 2017) using document term frequency vector. We then extract top-100 important words using feature importance. Lastly, we manually select important words over 100 words. We find that the distribution of ratio of important words in a sequence gradually increase over time. Based on this analysis, we figure out that the feature related with downtream task is shifted by time.
C IMPLEMENTATION DETAILS
In this section, we describe the training details. Followed by it, we describe the algorithm we used to select the hyperparameters in our suggested modification to the loss function. Then, we provide the links to download the datasets.
C.1 TRAINING DETAILS AND COMPUTING INFRASTRUCTURE
Image. We used ResNet18 (He et al., 2016) that is pretrained with ImageNet (Deng et al., 2009) to train a gender classifier using a UTKFace NAtS dataset. After the average pooling layer, we trained a fully-connected network composed of 2 hidden layers which have 128 and 2 units respectively with a relu activation. The network is trained for 100 epochs with a batch size of 64. We used stochastic gradient descent with Adam optimizer (Kingma & Ba, 2014) and set learning rate as 3e − 5. For data augmentation, the technique of SimCLR (Chen et al., 2020) is used for all the experiments.
Text. We perform experiment with Transformer network with 4 layers. The network is trained for 10 epochs with batch size of 128. We used Adam as an optimizer and set the learning rate as 3e− 6. Medical. We use a ResNet18 (He et al., 2016) model, pretrained on ImageNet (Deng et al., 2009), and add two fully-connected layers containing 128 and 2 hidden units with a relu activation. We train the network to classify gender given the x-ray image. Each model is trained for 30 epochs using SGD optimizer with a learning rate of 0.01 and momentum of 0.9, using a batch size of 64.
All the experiments are conducted on a single GeForce RTX 3090 GPU with 24GB memory. The results are measured by computing mean and standard deviation across 5 trials upon randomly chosen seeds.
C.2 HYPERPARAMETER TUNING
Our suggested modification to the training loss mainly comprises of distance loss (Ldist) and a entropy loss (Lentropy). More formally, the training loss is given by:
Ltotal = LCE + Ldist + Lentropy. (5)
The entropy loss comprises of two terms and is defined as:
Lentropy = λ2 D∑
i Var(zi) + λ3 1( D 2 ) ∑ i ∑ j ̸=i C2ij , (6)
where λ2 > 0 and λ3 > 0 are hyperparameters, z ∈ RD is the feature representation in the latent feature space, and Cij is the correlation coefficient between ith and jth dimension of the feature space. For more details, please refer to Section 6.1.
For simplicity, in this section, we will refer the first term of entropy loss as variance loss, and the second term as correlation loss. To obtain the hyperparameters of different terms in our loss function, we explore the value of λ2 in [0.01, 0.1, 1.0, 10.] and λ3 in [0.0001, 0.001, 0.01, 0.1, 1.0]. The hyperparameter corresponding the distance loss, λ1, is set as 0.1, and then the hyperparameters of the variance loss and the correlation loss are chosen by a simple algorithm.
Algorithm: We now describe the algorithm we used to find the hyperparameters of variance and correlation loss. First, we calculate the harmonic mean of the variance loss and correlation loss using our training dataset. Next, we select the hyperparameters with the lowest value of harmonic mean. Then, to ensure that entropy loss prevents the feature space collapsing problem, we apply the singular value decomposition (SVD) in the penultimate features and test if the sum of singular values except the two largest values is improved by the entropy loss or not. Concretely, to prove the enhancement, we compare the values with those of the model trained with the LCE + λ1Ldist. If the selected hyperparameters do not improve the values, we reject them and investigate the hyperparameters that have the next lowest harmonic mean. The hyperparameters satisfying the above steps are selected as the hyperparameters of our loss function.
Note that we reject the hyperparameters if they significantly degrade the classification accuracy. In image domain, we applied the algorithm and obtained 0.1, 0.1, 0.0001 as hyperparameters of λ1, λ2, and λ3, respectively. In the medical domain, we obtain the hyperparameter corresponding to λ3 as 1.0, and the other hyperparameters are the same as those of the image domain. In the text domain, 10. and 1.0 are used for the λ2 and λ3. The distance loss is set by 0.1.
C.3 LINKS FOR DATASETS
In our work, we use the openly available datasets, namely, UTKFace dataset2, Amazon Review dataset3, and RSNA Bone Age dataset4.

heading: D ABLATION STUDIES ON DISTANCE AND ENTROPY LOSS
text: In this section, we conduct an ablation study to investigate the effect of each proposed term in the loss function in equation 5. Table 4 presents an ablation study to find the effect of the entropy loss and distance loss in the proposed modification to training loss. In UTKFace, when training with solely attaching Ldist or Lentropy to LCE, the NAtS detection performance using mahalanobis OOD detection method improves compared to training with only LCE, but the performance does not monotonically increase in both cases along with the variation of Age. However, when both Ldist and Lentropy are used, there is a large improvement in performance with the results being monotonically increasing as we move from ID towards NAtS, which implies that these two losses are mutually beneficial. In the text and medical domains, although there is a little degradation of AUROC when only Ldist is added, the performance is improved by using Lentropy in the training objective. Further, to analyze the impact of each loss term to the feature-level representations, we perform a singular value decomposition (SVD) in the penultimate layer of ResNet18 trained on the UTKFace dataset, similar to Verma et al. (2019). When trained with LCE only, the sum of the two largest singular values was 617.47, while the sum of the remaining singular values was 785.54. When trained with LCE + Ldist, the sum of the two largest singular values increased (4481.88), while the sum of the remaining singular values decreased (371.86).
As discussed in Section 6.1 in the main paper, this indicates that the addition of the distance loss is likely to collapse the latent feature space (i.e. rank-deficient), thus possibly decreasing the model’s sensitivity to NAtS samples. For example, if a model is trained to classify apples and banaNAtS based only on the color, it will not be able to tell that a firetruck is a NAtS sample. When we train the classifier with Lentropy added, the problem is effectively alleviated. The sum of the two largest singular value is reduced from 4481.88 to 2960.63, and the sum of the remaining singular values increased from 371.86 to 513.13.
2https://susanqq.github.io/UTKFace/ 3https://jmcauley.ucsd.edu/data/amazon/ 4https://www.rsna.org/education/ai-resources-and-training/
ai-image-challenge/rsna-pediatric-bone-age-challenge-2017

heading: E COMPARISON WITH ADDITIONAL BASELINES
text: In this section, we investigate NAtS detection performance using additional baselines, including recent work, and compare them with those of the model trained with our modified loss on three NAtS datasets. Firstly, as we mentioned in the main paper, we evaluate the model trained with crossentropy loss with ensembled version of Mahalanobis detector (Lee et al., 2018) and the method that utilizes the gram matrix in OOD detection (Chandramouli & Sageev, 2020).
We also consider the other recent baselines for the comparison. Recently, Sehwag et al. (2021) propose a unified framework to leverage self-supervised learning for OOD detection. We set the
method as our baseline and denote this baseline as SSD+. The experiment on SSD+ requires a data augmentation strategy. In the Image domain, we augment the training dataset as Chen et al. (2020). We construct an augmented training dataset using a back-translation model for text experiments as (Fang et al., 2020; Gunel et al., 2020). We train SSD+ for 100 epochs for UTKFace and RSNA Bone age dataset and 50 epochs for Amazon Review dataset.
Some recent methods (Hsu et al., 2020; Techapanurak et al., 2020) exploit the cosine similarity to detect OOD samples. Hsu et al. (2020) suggest to decompose confidence scores and to modify input preprocessing method in ODIN (Liang et al., 2018). For OOD detection, they calculate the cosine similarity between the features from the penultimate layer and the class specific weights and use the maximum value as an OOD score. Techapanurak et al. (2020) also propose to compute cosine similarity between the class weight vector and the penultimate layer feature of each input to obtain OOD scores. Specifically, they train the model with cross entropy loss as standard classification model, but logit value is set by the scaled cosine similarity value. After training, the cosine similarity values between the penultimate layer feature of the input and the weight vector for each class are calculated. The maximum cosine similarity value is used as the OOD score for the samples. We name the methods as GODIN and Scaled Cosine in Table 6, respectively.
NAtS detection performance of our method and other baselines are provided in Table 6. In UTKFace dataset, GODIN, Scaled Cosine, and ensembled version of Mahalanobis detector have relatively low detection performance than the other baselines. Although Gram matrix shows reasonable detection performance in UTKFace and RSNA Bone Age NAtS groups, this method is hard to applied to the Amazon Book Review dataset because the notion of gram matrix is vague in the text domain. In three NAtS categories, SSD+ has the most robust detection performance among the five baselines, however, our simple remedy in training loss help Mahalanobis detector to demonstrate improved performance particularly under the UTKFace NAtS dataset. In conclusion, when compared with the five more recent baselines, our method shows robust and high detection performance in all the three NAtS datasets, regardless of the domains. Scaled Cosine, which compute logit based on cosine similarity, have similar tendency with confidence-based methods.

heading: F QUANTITATIVE RESULTS WITH OTHER METRICS
text: We report the NAtS detection performance of baselines and our method based on four other metrics: Detection Accuracy, AUPR-In, AUPR-Out, and TNR@95%TPR, which are often used in the OOD detection community. We present the detection performance on NAtS datasets in three domain measured by Detection Accuracy, AUPR-In, AUPR-Out, and TNR@95%TPR in Tables 7,8,9, and 10 respectively. The results indicate that our simple modification in the training objective improves the performance of the Mahalanobis detector, thus making it a robust NAtS detection method for the three NAtS categories presented in the paper. Specifically, in UTKFace, we observe that using our suggested training loss results in a significant increase in NAtS detection performance using the Mahalanobis detector. At the same time, the suggested method effectively detects the NAtS samples in the other two datasets. In summary, based on the results obtained using five different metrics (AUROC, Detection Accuracy, AUPR-In, AUPR-Out, and TNR@95%TPR), our suggested modification makes the Mahalanobis detector a general NAtS detection method for all three NAtS categories.

heading: G ADDITIONAL PCA ANALYSIS
text: In this section, we provide the additional PCA visualization results. Figure 8 presents PCA plots applied on the penultimate features obtained from the model trained with our loss. We observe that the shifted samples in the image domain move in between the two ID classes, and the majority of the NAtS samples do not overlap with the ID clusters. This helps in effectively detecting the shifted samples as NAtS.
Furthermore, we present the PCA visualization on a model trained with a cross-entropy loss in Figure 9. We categorize NAtS samples based on their grouth-truth labels as opposed to Figure 3 in which we do not segregate NAtS samples based on their ground truth labels. In the main paper, we intentionally do not categorize (and colorize) the NAtS samples because the ground truth class labels are not available at the inference time. In an ideal situation, the class labels should be predicted with the trained model only if the samples are not classified as NAtS, otherwise, they should be rejected for human assessment. Note that the PCA visualization cannot be used to infer the real increase in the distance between two classes since PCA compresses the original feature space into 2 dimensions.

heading: H QUANTITATIVE RESULTS

heading: I EXPERIMENTS ON A MULTI-CLASS SYNTHETIC DATASET
text: We conduct additional experiments on a multi-class classification dataset to show that the three NAtS categories can exist in different tasks and datasets. In order to determine the NAtS category to which a shifted sample belongs, we need to train the model for a task on a dataset, evaluate the performance of existing OOD detection methods, and analyze the location of samples in feature space. However, we would have to conduct a series of experiments on numerous tasks and datasets until we encounter samples from each of the three NAtS categories. (There is no way to know the NAtS category of a given dataset until we run experiments on it.) To this end, we synthesized 2D multi-class dataset with 8 classes and conducted experiments on it.
We synthesized the 2-dimensional vectors for training dataset with 8 classes, and those samples are visualized as red points in the 2D plane in Figure 12. In particular, we create samples in each class by randomly sampling from a gaussian distribution with a
random mean and variance. Deep neural networks are primarily useful in complex datasets with high dimensional inputs. Therefore, for training, we increase the input dimensions by adding 8- dimensional vector which are filled with the Gaussian noise which have mean of zero and variation of 0.1. The network would focus on the 2 important input features which corresponds x and y coordinates to classify the classes and ignore others.
We train the feed-forward classifier using the standard cross-entropy loss and our modified loss. The classifier comprises 9 linear layers, and each hidden layer consists of 10 units, and the output layer consists of 8 units (one for each class). We then evaluate the performance of MSP, ODIN, Mahalanobis under NAtS for model trained using CE loss in Figure 12 (a), (b), and (c) respectively. Purple Stars, Gray Diamonds and Orange Triangles indicate samples from NAtS categories 1, 2 and 3 respectively. We observe that MSP and ODIN detects samples from NAtS category 1 and 3 but fails to detect from NAtS category 3. Mahalanobis detects samples from NAtS category 2 (Gray Diamonds) and 3 (Orange Triangles), but fails to detect samples from NAtS category 1 (Purple Stars). We then evaluate the performance of Mahalanobis under NAtS for model trained using our proposed loss in Figure 12 (d). We observe that Mahalanobis (on model trained with our loss) effectively detects samples from all NAtS categories. This demonstrates that our proposed training objective makes the Mahalanobis detector a robust NAtS detection method for all NAtS categories.

Review summaries:
The paper identifies the problem of a Natural Attributed-based Shift (NAS) in the data distribution affecting Deep Learning (DL) systems designed for the task of interest. An example may be a DL system for X-ray based diagnosis being able to handle a different brightness that it was trained for. The paper argues that it is similar to Out-of-Distribution (OOD) data but instead of the difference being in the label space (CIFAR vs SVHN) the label space is similar but the test data distribution is shifted due to a systemic shift in an important data attribute (brightness of image in the X-ray example above). The paper claims the following contributions: - A new task of NAS detection and datasets to study it. - A comprehensive evaluation of OOD methods on the NAS dataset and demonstration that none of them perform consistently across all NAS scenarios. - A novel analysis based on the location of shifted samples in the feature space and performance of existing OOD methods. - Demonstrate that a simple modification to training objective for deep classifiers enables consistent OOD performance on all NAS scenarios. 

The authors propose a new task named NAS (natural attribute-based shift), which is a sub-task of OOD (out of distribution). Three datasets are designed from existing datasets for the NAS experiment. By visualizing the distribution of category features produced by the model under different datasets using PCA, the authors find three categories for NAS distribution. Total three OOD methods are tested on NAS datasets and show an inconsistency in their performance. Finally, the authors propose to use distance loss and entropy loss to retrain the models to improve the distance-based OOD methods. 

The paper defines a task called Natural Attribute-based Shift detection (NAS detection) to identify shifted samples on some natural attributes, e.g. age, time and lighting.  The authors create three benchmark dataset from existing datasets in three domains, i.e. vision, text and medical. Their contributions are in three aspects: evaluation and analysis of out-of-distribution detection (OOD) methods on those NAS benchmark datasets, classify NAS samples as three different categories and present a modification to training objective of deep classifiers to improve an OOD detection method.