Out-of-Distribution Image Detection Using the Normalized Compression Distance

1 INTRODUCTION . The distributional uncertainty refers to an uncertainty originated from the inconsistency between training and test distributions ( Malinin & Gales , 2018 ) . Recently , measuring the distributional uncertainty of deep neural networks has gained much attention , in order to detect the out-of-distribution sample which comes from outside the training distribution . In contrast to the existing belief that the softmax output of deep neural networks is not appropriate measure for uncertainty , Hendrycks & Gimpel ( 2017 ) found out that the softmax score is able to distinguish between in-distribution and out-of-distribution samples to a degree . Since then , several methods have shown promising results for out-of-distribution detection ( Liang et al. , 2018 ; Lee et al. , 2018b ; Lakshminarayanan et al. , 2017 ; Lee et al. , 2018a ) , but the most of them still have limitations from a practical perspective in that they need to find the optimal values of hyperparameters using the out-of-distribution samples or retrain the model using a new objective function specifically designed for detection . For example , Liang et al . ( 2018 ) improved the ability of distinguishing the out-of-distribution data by using temperature scaling and input perturbation , which involves in several hyperparameters ; this kind of calibration techniques require out-of-distribution samples for validation to adjust its hyperparameters to manipulate the softmax outputs . On the other hand , Lee et al . ( 2018a ) defined an another loss function by adding the KL divergence , assuming that the softmax output of the out-of-distribution is uniform ; it has to retrain the model because of the newly designed loss function . For these reasons , they have been difficult to be deployed for real-world applications . In this sense , our motivation is how far out-of-distribution detection can be done without using the out-of-distribution set for validation while employing the existing softmax classifier without retraining it . The state-of-the-art method ( Lee et al. , 2018b ) tries to detect out-of-distribution samples based on the Mahalanobis distance from class means instead of using the softmax output , by utilizing the feature maps obtained by convolutional neural networks . In particular , they estimate the class-conditional probabilities for each class , which are more appropriate and effective to detect out-of-distibution samples than the softmax output which provides a mere relative values between classes . However , they obtain the feature vectors of input images by averaging each feature map , also known as global average pooling . We notice that the spatial information about input images disappears in the process of averaging feature vectors , and this eventually degrades the detection performance . To tackle this challenge , we offer to use the normalized compression distance ( NCD ) , which can measure the distance between two images without the loss of such spatial information of the feature maps . NCD is a general method of measuring the intrinsic distance between two arbitrary objects using the off-the-shelf compression algorithm ( e.g . 7-zip , bzip , or lzw ) . With its solid theoretical background and general applicability , NCD is widely used in various fields , including brain diagnosis or spam filtering ( Berek et al. , 2014 ; Spracklin & Saxton , 2007 ) . The high-level idea of NCD is to evaluate how many shared patterns are eliminated when two binary sequences are compressed together . By using the NCD , we aim to accurately measure how similar the feature maps of a test sample is with those of training examples , and use it to determine whether the test sample is from out-of-distribution or not . Our experiments demonstrate that the proposed method , termed as MALCOM , achieves better or comparable performances for detecting out-of-distribution images to the existing methods . 2 BACKGROUNDS . 2.1 CONSTRAINED OUT-OF-DISTRIBUTION DETECTION . Before proceeding to describe the notions closely related to our method , let us state what we want to tackle exactly : out-of-distribution detection without retraining network and without sniffing outof-distribution samples . Starting with the definition , let Dtrain = { ( xi , yi ) } Ni=1 be the training dataset and f ( · ; θ ) be the model trained on Dtrain . Suppose that each data xi from Dtrain is an observation of random variableXtrain . Thereby , the out-of-distribution detection task is to obtain the out-of-distribution score function κ ( · ; Dtrain , θ , φ ) for a discrimination between in-distribution and out-of-distribution . In other words , given a test instance x∗ , the score κ ( x∗ ) should be high if x∗ is not an observation ofXtrain , otherwise should be low . There are many ways to get score function κ , but we work on the task under the following constraints : • The score function parameter φ should be independent with the distribution of test data x∗ , i.e. , φ is determined by only in-distribution . This is intuitive in that x∗ does not assume any test distribution . This requirement implies that hyperparameter tuning by using out-ofdistribution samples is not allowed . • The model parameter θ should not be changed , i.e. , the trained model primitively for the purpose of classification should not be retrained . Then , the score function can be utilized by the neural networks which are already trained and deployed in real-world applications . The existing methods that satisfy these conditions are the softmax output detector ( Hendrycks & Gimpel , 2017 ) and the simplified Mahalanobis detector ( Lee et al. , 2018b ) . 2.2 MAHALANOBIS DETECTOR . The Mahalanobis detector ( MAHALANOBIS ) , one of the best performing methods , introduces an effective method to remove the correlation between feature maps of the last hidden layer ( Lee et al. , 2018b ) , but combining layers leads to the degradation of the detection performance . 2.2.1 MAHALANOBIS-VANILLA . We factor out a simplified version of MAHALANOBIS ( MAHALANOBIS-vanilla ) , because the whole MAHALANOBIS detector needs out-of-distribution samples to tune the hyperparameters of input preprocessing and layer weights , which fails to satisfy our requirements ( Sec 2.1 ) . For the whole method of MAHALANOBIS which utilizes out-of-distribution samples to boost the performance , we refer the reader to see ( Lee et al. , 2018b ) . MAHALANOBIS-vanilla uses the feature maps of the last hidden layer of convolutional neural networks . To be exact , we denote the output of the l-th hidden layer of the network f by fl for l = 1 , . . . , L , where L is the number of hidden layers . For the l-th hidden layer , let f il be the i-th feature map of its output for i = 1 , . . . , Ml , where Ml is the number of feature maps of the l-th hidden layer . Therefore , fl ( x ) is a 3D tensor of the size Ml ×Hl ×Wl and f il ( x ) is a 2D feature map of the size Hl ×Wl for any i and l , where Hl andWl are the height and width of each feature map in l-th layer , respectively . For a notational convenience , we define ml as the global average pooling of the l-th layer , i.e. , ml ( x ) : =‖Mli=1 mean ( f i l ( x ) ) , where mean is the average function and ‖ is the concatenation operation . MAHALANOBIS-vanilla utilizes only the last hidden layer ( i.e. , the L-th layer ) , so it can be obtained by calculating class means and tied-covariance : µ̂k : = 1 Nk N∑ n=1 yn=k mL ( xn ) , Σ̂ : = 1 N K∑ k=1 N∑ n=1 yn=k ( mL ( xn ) − µ̂k ) ( mL ( xn ) − µ̂k ) T , where K is the number of classes and Nk is the number of the data of class k. Then , the outof-distribution score function of MAHALANOBIS-vanilla detector κmv ( also known as Mahalanobis distance ) is defined by κmv ( x ∗ ) : = min k=1 , ... , K ( mL ( x ∗ ) − µ̂k ) T Σ̂−1 ( mL ( x∗ ) − µ̂k ) . In this equation , MAHALANOBIS-vanilla computes the centered data for each class , then performs linear projection for the centered data by using the inverse of Σ̂ . Practically , the inverse of empirical covariance matrix is approximated by ( Moore-Penrose ) pseudo-inverse , which performs the singular value decomposition ( SVD ) ( Barata & Hussein , 2012 ) . For this reason , computing the Mahalanobis distance becomes equivalent to computing the norm of the sample ( i.e. , the Euclidean distance from the origin ) after the principal component analysis ( PCA ) assuming that the origin is the class mean , which removes the feature correlations . 2.2.2 DEGENERACY OF MAHALANOBIS-ASSEMBLE . From the observation that MAHALANOBIS-vanilla has the power of removing the correlation over feature maps , one may extend this concept to all the other hidden layers . To define the MAHALANOBIS-assemble1 detector , we concatenate the average of feature maps ( i.e. , global average pooling ) of all layers and similarly define the class means and tied-covariance : m ( x ) : =‖Ll=1 ml ( x ) , µ̂k : = 1 Nk N∑ n=1 yn=k m ( xn ) , Σ̂ : = 1 N K∑ k=1 N∑ n=1 yn=k ( m ( xn ) − µ̂k ) ( m ( xn ) − µ̂k ) T , κma ( x ∗ ) : = min k=1 , ... , K ( m ( x∗ ) − µ̂k ) T Σ̂−1 ( m ( x∗ ) − µ̂k ) 1We use the term “ assemble ” to distinguish it from the “ feature ensemble ” , which is the method using logistic regression in the original paper . From the experiments , we find an interesting observation that the simply extended detector κma significantly improve the performance in many cases ; however , in some cases , its performance , especially AUPR ( In ) , is poorer than κmv ( see the first two columns of Table 1 for more details ) . 2.3 NORMALIZED COMPRESSION DISTANCE . To address the degeneracy problem in MAHALANOBIS-assemble , we focus on hindering the loss of spatial information of the feature maps caused by the average pooling . To this end , we introduce a compression-based method . Before defining our method precisely , we need to state some notions briefly because its fundamental properties are originated from the concept of information theory . 2.3.1 KOLMOGOROV COMPLEXITY . The Kolmogorov complexity ( or algorithmic complexity ) is the complexity of an arbitrary object which can be represented in string format . Formally , given an universal Turing machine U , the Kolmogorov complexity K of string x is defined as the minimum length of the programs that are decoded into x by U . That is , K ( x ) : = min ( { length ( p ) |U ( p ) = x where p is a valid program for U } ) , where length ( p ) is the length of the program p and U ( p ) is the result of decoded string with the program p through U . The conditional Kolmogorov complexity is the minimum program length for a given universal Turing machine which already has the string : K ( x|y ) : = min ( { length ( p ) |U ( p|y ) = x } ) . If x is highly correlated with y , its conditional complexity would be low .

Summary: This paper proposes a new framework for out-of-distribution detection, based on global average pooling and spatial pattern of the feature maps to accurately identify out-of-distribution samples. MAHALANOBIS distance based methods were discussed, and the shortcomings of using mahalanobis distance were given (i.e. assumption that features are independent, etc). They propose to use compression based distance measures off the shelf from standard compression techniques to detect spatial feature patterns in feature space and demonstrate its effectiveness on several datasets and comparison with baselines is reported and well discussed.