Decomposing Texture and Semantics for Out-of-distribution Detection

1 INTRODUCTION . The out-of-distribution ( OOD ) detection is the task that recognizes whether the given data comes from the distribution of training samples ( also known as in-distribution ) or not . Any machine learning-based system could receive input samples that have a completely disparate distribution from the training environments ( e.g. , dataset ) . Since the distribution shift can severely degrade the model performance ( Amodei et al. , 2016 ) , it is a potential threat for a reliable real-world AI system . However , an ambiguous definition of the “ in-distribution ” limits the feasibility of the OOD detection method in real-world applications , considering the various OOD scenarios . For example , subtle corruption is a clear signal of the OOD in the machine vision field while a change in semantic information might not be . On the other hand , an autonomous driving system may assume the in-distribution from the semantic-oriented perspective ; e.g. , an unseen traffic sign is the OOD . Interestingly , the interpretations of the in-distribution described in the above scenarios are not correlated ; rather , they are contradicted . Unfortunately , most of the conventional OOD detection methods ( Zhang et al. , 2021 ; Tack et al. , 2020 ; Ren et al. , 2019 ) assume the in-distribution as a single-mode ; thus are not able to handle other aspects of the OOD ( Figure 1a ) . To tackle this issue , we revisit the definition of the in-distribution by decomposing it into two different factors : texture and semantics ( Figure 1b ) . For the texture OOD case , we define the OOD as the textural difference between the in- and out-of-distribution datasets . On the contrary , the semantic OOD focuses on the class labels that do not exist in the in-distribution environment . Note that the two aspects have a trade-off relationship , thus detecting both OOD problems with a single model is challenging with the ( conventional ) entangled OOD point of view . Similar to ours , Geirhos et al . ( 2018 ) investigated the texture-shape cue conflict in the network , and a series of following studies ( Hermann et al. , 2019 ; Li et al. , 2020 ; Ahmed & Courville , 2020 ) explored the way to find a balance between these perspectives . However , aforementioned works utilize texture-shape to analyze the bias inherited in deep networks . In this work , instead , we focus on analyzing the texture and semantic nature underlying the in-distribution to build a more practically applicable OOD detection method . However , to the best of our knowledge , none of the studies on the OOD detection benchmark have thoroughly investigated the definition of the in-distribution . This can be problematic when the OOD detection method judges the image corrupted by minor distortion as OOD , even when the environment is tolerant to the small changes in texture . Because of such a complicated scenario , it is important to evaluate the OOD detection method in a comprehensive way that goes beyond the simple benchmarks . In this work , we propose a new approach to measuring the performance of the method according to the decomposed definition of the in-distribution . One notable observation in our benchmark is that most of the previous OOD detection methods are highly biased to the texture information and ignore the semantic clues in many cases . To mitigate this issue , our method tackles the texture and semantic information separately and aggregates these at the final module ( Figure 2 ) . To effectively extract the texture information , we use the 2D Fourier transform motivated by the recent frequency domain-driven deep method ( Xu et al. , 2020 ) . For the semantic feature , we design an extraction module upon the Deep-SVDD ( Ruff et al. , 2018 ) with our novel angular distance-based initialization strategy . We then combine two features using the normalizing flow-based method ( Dinh et al. , 2016 ) , followed by our factor control mechanism . This control module provides the flexibility to handle various OOD scenarios by choosing which decomposed feature is more important in the given surrounding OOD circumstance . The main contributions of this work are as follows : • We decompose the “ unclear ” definition of the in-distribution into texture & semantics . To the best of our knowledge , this is the first attempt to clarify the OOD itself in this field . • Motivated by real-world problems , we create new OOD detection benchmark scenarios to evaluate the models based on the decomposed in-distribution factors . • We propose a novel OOD detection method that is effective on both texture & semantics as well as the conventional benchmark setups . In addition , our method does not require any auxiliary datasets or class labels unlike the previous models . 2 RELATED WORK . In this section , we briefly overview the notable studies on the OOD detection field . We categorize the deep learning-based OOD detection methods into three groups based on the characteristics of the information that they used . Class labels of the in-distribution . Early studies on deep OOD methods rely on class supervision . ODIN , Generailized ODIN ( Liang et al. , 2017 ; Hsu et al. , 2020 ) use the uncertainty measure derived by the Softmax output . It determines the given sample as the OOD when the output probability of all classes is less than the predefined threshold . Sastry & Oore ( 2020 ) ; Lee et al . ( 2018 ) utilize the extracted feature map ( e.g. , gram matrix ) from the pre-trained networks to calculate the OOD score . Auxiliary distribution . Outlier exposure ( OE ) ( Hendrycks et al. , 2018 ) exploits additional datasets that are disjointed from the test dataset to guide the network to better representations for OOD detection . Papadopoulos et al . ( 2021 ) further improves the performance of OE by regularizing the network with the total variation distance of the Softmax output . Data augmentation . Recently , contrastive learning-based methods have shown remarkable success on the tasks related to visual representation ( He et al. , 2020 ; Chen et al. , 2020 ) . Motivated by this , several studies employ data augmentation methods such as image transformation or additional noise on the OOD detection task ( Hendrycks et al. , 2019 ; Tack et al. , 2020 ) . Unlike the prior studies that exploit additional information other than the in-distribution , we only utilize the given ( in-distribution ) training dataset . In addition , we separate and clarify the assumption of the OOD as texture and semantics to improve the practicability in the real world . 3 METHOD . In this section , we present an overview of our proposed method ( Section 3.1 ) , and the feature extraction modules of the model ( Section 3.3 and 3.2 ) . Finally , we introduce the normalizing flow-based conditional probabilistic modeling component ( Section 3.4 ) . Conventional OOD detection has an assumption that in-distribution data are sampled from the distribution of the training dataset , x ∼ pdata . We decompose the image with two factors and calculate the anomaly score based on each factor ’ s likelihood . The texture information T ( x ) extracted rigorous Fourier analysis process from input x . The semantic information S ( x ) extracts the content label features such as shape . Our framework calculates the likelihood of these two factors and then combines these likelihoods . Since we use the Normalizing flow model that trains the exact likelihood , the extracted information is adjusted using a lambda that the user can control . 3.1 MODEL OVERVIEW . We aim to train our method with the decomposed in-distribution likelihood , p ( T ( x ) |x ) and p ( S ( x ) |x ) ( Figure 2 ) . With the given input image x , we extract the features for each variable with different approaches . In detail , we distil the information from the texture and semantic information with T ( x ) and S ( x ) , respectively . The extracted features are combined by the controllable normalizing flow method . Since our normalizing flow-based model explicitly calculates the negative log-likelihood , we model each extracted information as log pθ ( T ( x ) |x ) and log pφ ( S ( x ) |x ) , where θ and φ are trainable parameters of the networks . In addition , we introduce the control parameter λ ∈ [ 0 , 1 ] to model the final probability as λ · log p ( x|T ( x ) ) + ( 1− λ ) · log p ( x|S ( x ) ) . With this control mechanism , a user can determine the appropriate model “ mode ” by referring to the prior knowledge . For example , in the case where the texture information overwhelms the semantic one for detecting OOD , we can overweight λ for better performance . By default , we use the λ value of 0.5 ( no prior knowledge ) . 3.2 EXTRACTING THE SEMANTIC INFORMATION . Multi-SVDD . Beyond the one-class anomaly detection that considers the normal data as a single class ( e.g. , DeepSVDD ( Ruff et al. , 2018 ) ) , recent studies have viewed the normal data as the union of the multiple hidden semantic information ( Ghafoori & Leckie , 2020 ; Park et al. , 2021 ) . Inspired by this idea , we use the multi-SVDD method to extract the semantic information in an unsupervised manner for the OOD detection task . Multi-SVDD embeds the samples to the multiple center vectors as close as possible . Suppose the set of center vectors C = { c1 , ... , cK } is initialized via K-means and the radius of each center is r = [ r1 , ... , rK ] . In multi-SVDD , the objective function is defined as follows . min W , r K∑ k=1 r2k + 1 νn n∑ i=1 max { 0 , ‖φ ( xi ; W ) − cj‖2 − r2j } + η 2 ∑ ‖W‖2 . ( 1 ) Here , φ ( xi ; W ) is the deep network with a set of weight parameters W and cj is assigned to φ ( xi ; W ) . As the set r is decreased , the samples are condensed into the center vectors . By using the distance between the center vectors and the samples , we get an anomaly score . Angular distance initialization . The SVDD method is originally introduced for the anomaly detection task . Because of the disparity between the OOD and the anomaly detection scenarios , direct application of the SVDD-based model to OOD detection causes unexpected performance degradation . In anomaly detection , even though the abnormal samples lie in the in-distribution manifold , it is possible to detect them as abnormal unless they are close to the center vectors cj . For example , as shown in Figure 3a , the OOD samples ( red ) that are located inside of the in-distribution manifold ( light blue shade ) can be detected as abnormal since they are outside of the tight cluster boundary ( dark blue shade ) . Because of such characteristics , a mixture of Gaussian probability density is a reasonable density space for the anomaly detection model . Unlike the anomaly detection task , the definition of the OOD detection task is to find the samples that are not the “ in-distribution ” . In Figure 3a , all the OOD samples placed in the in-distribution manifold ( light dark shade ) are recognized as the in-distribution . To tackle this issue , we propose an angular distance-based center vector initialization strategy : ck = γ v ‖v‖ , v ∈ Rh ∼ N ( 0 , 1 ) ( 2 ) where h is the dimension of the embedding space and γ is the hyper-parameter for the radius of the sphere . After φ ( xi ; W ) is trained based on angular initialization , semantic features are extracted through this model ; S ( xi ) = φ ( xi ; W ) . By setting the γ value as large enough , we ensure that all sample data are within a radius of the sphere as illustrated in Figure 3b . While Equation 1 drives the training samples to be embedded around the center vectors on the sphere , the OOD samples remain near the origin . This embedding space may be weak to recognize the semantic label of a given sample , but it is sufficient to identify whether the sample is OOD or not .

Summary: The submission proposes to evaluate OOD detection problems with regards to two aspects — detect distributional shift in texture, or that in object-identity. A Fourier transform is used to identify changes in texture, and a modification of SVDD is used to identify changes in object identity, by building density models on top of a PCA-reduction of the extracted features in both cases. Experiments are conducted which showcase that the two components can detect textural vs. object-identity shift while not mis-identifying one type of shift for the other.