Decomposing Texture and Semantics for Out-of-distribution Detection
Authors: Jeong-Hyeon Moon
Namhyuk Ahn
Kyung-Ah Sohn
Venue: Submitted to ICLR 2023 (2023)
Keywords: Out-of-distribution detection
Fourier analysis
Normailzing flow model

Abstract: Out-of-distribution (OOD) detection has made significant progress in recent years because the distribution mismatch between the training and testing can severely deteriorate the reliability of a machine learning system.Nevertheless, the lack of precise interpretation of the in-distribution limits the application of OOD detection methods to real-world system pipielines. To tackle this issue, we decompose the definition of the in-distribution into texture and semantics, motivated by real-world scenarios. In addition, we design new benchmarks to measure the robustness that OOD detection methods should have. To achieve a good balance between the OOD detection performance and robustness, our method takes a divide-and-conquer approach. That is, the model first tackles each component of the texture and semantics separately, and then combines them later. Such design philosophy is empirically proven by a series of benchmarks including not only ours but also the conventional counterpart.

Under review as a conference paper at ICLR 2023
DECOMPOSING TEXTURE AND SEMANTIC FOR OUT- OF-DISTRIBUTION DETECTION
An Anonymous authors
Paper under double-blind review
ABSTRACT
out-of-distribution (OOD) detection tasks have made significant progress recently since the distribution mismatch between training and testing can severely deterio- rate the reliability of Al systems. Nevertheless, the lack of precise interpretation for the in-distribution (ID) limits the application of the OOD detection methods to real-world systems. To tackle this, we decompose the definition of the II) into texture and semantics, motivated by the demands of real-world scenarios. We also design new benchmarks to measure the robustness that OOD detection methods should have. To achieve good balance between the OOD detection performance and robustness, proposed model first handles each component of the texture and semantics sepa- rately and then fuses these later. This philosophy is empirically proven by a series of benchmarks including both the proposed and the conventional counterpart.
method takes
and-conquer approach. Specifically, the
INTRODUCTION
Out-of-distribttion (OOD) detection the task that recognizes whether the given data comes from the distribution of training samples, also known as in-distribution (ID), or not. Any machine learning-based system could receive input samples that have a completely disparate distribution from the training environments (e.g., dataset). Since the distribution shift can severely degrade the model performance (Amodei et al. 2016), is a potential threat to reliable real-world AI systems.
However, the ambiguous definition of the ID limits the feasibility of the OOD detection method in real-world applications, considering the various OOD scenarios. For example, subtle corruption clear signal of OOD in the machine vision field while change in semantic information might not be. On the other hand, an autonomous driving system may assume the ID from the semantic- oriented perspective; e.g., unseen traffic sign is OOD. Unfortunately, most of the conventional OOD detection methods and benchmarks (Zhang et al., 202 Tack et al. 2020) Ren et al.) 2019 Chan al. 202 assume the ID as single-mode thus they cannot handle other aspects of OOD properly (Figure Th).
To tackle this, we revisit the definition of the ID by decomposing it into factors: fexture and semantics (Figure [[p). For the texture factor, we define OOD as the textural difference between the ID and OOD datasets. On the contrary, the semantic OOD focuses on the class labels that do not exist in the ID environment. Note that the two aspects have a trade-off relationship, thus detecting both problems with single model challenging with the (conventional) entangled OOD perspective.
Geirhos et al. 2018) investigated the texture-shape cue conflict in the deep network and a series of
subsequent studies (Hermann al. 2019 Li et al.) 2020 Ahmed & Courville 2020) explored how
to achieve a balance between these perspectives. However, they only analyzed the texture-shape bias inherited in deep networks. Instead, we focus on analyzing the texture and semantic characteristics underlying the ID to build more practically applicable OOD detection method.
Unfortunately, to the best of our knowledge, none of the studies on the OOD detection benchmark have thoroughly analyzed the definition of the ID. It can be problematic when the method judges the image corrupted by negligible distortion as OOD, even when the environment can tolerate the small changes in texture. Because of such a complicated scenario, it is crucial to evaluate the OOD detec- tion method in a comprehensive way that goes beyond the simple benchmark. Thus, in this study, we propose approach to measuring the performance of the method according the decomposed

---

Under review as a conference paper at ICLR 2023
In-distribution manifold
In-distribution manifolds
Training dataset
Semantic driven in-distribution
Texture driven in-distribution
discrepancy
Semantic discrepancy
(a) Traditional view the ID
(b) Decomposed view the ID (ours)
Figure I: How to define the ID? (a) Traditional OOD detection studies manage the ID in an entan- gled view. However, this could be naive considering the complex nature of the real environments. (b) We decompose the definition of the ID into texture and semantic; this provides the flexibility to handle complicated scenarios by determining which view of the ID is suitable for a given scenario.
definition of the ID. One notable observation our benchmark that most of the previous OOD detection methods are highly biased to the texture information and ignore the semantic clues.
To mitigate aforementioned issue, our proposed method tackles the texture and semantic infor- mation separately and aggregates these at the final module (Figure 2 To effectively extract tex- ture information, we use a 2D Fourier transform motivated by the recent frequency domain-driven deep method (Xu et al., 2020). For the semantic feature, design extraction module upon the Deep support vector data description (Deep-SVDD) (Ruff et al. 2018) with a novel angular distance-based initialization strategy. We then combine two features using the normalizing flow- based method (Dinh et al.) 2016), followed by the factor control mechanism. The control system provides the flexibility to handle different OOD scenarios by choosing which decomposed feature is more important in the given surrounding OOD circumstances. The main contributions of this work are follows:
We decompose the "unclear" definition of the ID into texture and semantics. To the best of our knowledge, this is the first attempt clarify OOD itself in this field.
Motivated by real-world problems, we create new OOD detection benchmark scenarios.
We propose a novel OOD detection method that is effective on both texture and semantics well as the conventional benchmark. Furthermore, our method does not require any auxiliary datasets or labels unlike the previous models.
RELATED WORK
Class labels of the ID. Early studies on deep OOD methods rely class supervision. ODIN and Generalized ODIN (Liang et al. 2017 Hsu et al. 2020) use the uncertainty measure derived by the Softmax output. It determines a given sample
OOD when the output probability of all classes is less than a predefined threshold. Sastryy & Oore 2020 Lee et al.) 2018) utilize the extracted feature map (e.g., gram matrix) from the pre-trained networks to calculate the OOD score. Also, Zhang et al. (2020) employ flow-based model that is comparable to ours, but they require class information during the training and solely pay attention to semantics.
Auxiliary distribution. Outlier exposure (OE) (Hendryeks et al. 2018) exploits additional datasets that are disjoint from the test dataset to guide the network to better representations for OOD de- tection. (Papadopoulos et al.) 2021) further improves the performance of OE by regularizing the network with the total variation distance of the Softmax output.
Data augmentation. Recently, contrastive learning- leareing-based methods have shown remarkable success on the tasks related to visual representation (He et al. 2020 Chen et al. 2020 Motivated by this, several studies employ data augmentation methods such image transformation or additional noise
on the OOD detection task or model (Hendrycks et al. 2019 Tack et al. 2020
Kirichenko et al.
2020
2

---

Multi-SVDD embeds the samples to the multiple center vectors as closely as possible. Suppose a set of center vectors C {C1y..., CR) is initialized via K-means and the radius of each center is Under review as a conference paper at ICLR 2023
E
ReaINVP
T(x)
[x)
2D DFT
A
RealNVP
logp (S(x) [x)
X
ResNet
Multi-SvDD
S(x)
Es
Figure 2: Model overview. Our framework extracts the texture and semantic information with the corresponding modules, then combines them via the normalizing flow-based method. feature T(x) is distilled by the Fourier spectrum-based module. (b) We use multi-SVDD with novel angular initialization to extract the semantic information S(z). (c) Output features are merged by the explicit probability inference method, RealNVP. Here, we introduce the user control parameter À E {0.0, 0.5, 1.0} to determine which feature is more suitable for a given OOD scenario.
(a) Texture
Unlike the prior studies that exploit additional information other than the ID dataset, we only utilize the given (ID) training dataset. In addition, separate and clarify the assumptions of OOD in terms of texture and semantics to improve the real-world practicability.
3 METHOD
We present an overview of the proposed method (Section 3.1). and each feature extraction module (Section 3.2 and 3.3 Finally, we introduce the normalizing flow-based conditional probabilistic modeling component (Section 3.4). Conventional OOD detection has assumption that ID data are sampled from the distribution of the training dataset, x ~ Pdata- We decompose the image with two factors and calculate the anomaly score based on each factor's likelihood. The texture module T(x) uses Fourier analysis to extract frequency-based features from input x. The semantic branch S(x) extracts the content label features such shape. Our framework calculates the likelihood of these two factors and then combines them. Since we use the normalizing flow model that trains the exact likelihood, the extracted information is adjusted using user-controllable hyperparameter X.
3.1 MODEL OVERVIEW
We aim to train our method with the decomposed ID likelihoods, p=]()) and p(S(x) (r) as shown 2 With the given input image x, we extract features for each factor with different modules: texture- and semantic-aided sub components.
The extracted features are combined by the controllable normalizing flow method. Since the normal- izing flow-based model explicitly calculates the negative log-likelihood, information as log po (T{x)<>) and log togp&(( (S(x)) where 0 and are trainable parameters of the networks. In addition, adjusted log probaability as À log ((S())
model each extracted
introduce the control parameter X E [0.0. 0.5, 1.0) to model the final (1- A) log (<T(x) With this control mecha- nism, users switch to the appropriate model "mode" by referring their prior knowledge. For example, when the texture information overwhelms the semantic one for detecting OOD, we can overweight for better performance. By default, we use the À value of 0.5 (no prior knowledge).
3.2 EXTRACTING THE SEMANTIC INFORMATION
Multi-SVDD. Beyond the one-class anomaly detection methods that consider the normal data single class (e.g., DeepSVDD (Ruff et al., 2018)), recent studies have viewed the normal data as the union of the multiple hidden semantic information (Ghafoori & Leckie 2020 Park al.) (021). Inspired by this idea, we use the multi-SVDD method to extract the semantic information in an unsupervised manner for the OOD detection task.
3

---

Under review as a conference paper at ICLR 2023
Center vector
In-distribution
In-distribution
OOD sample
sample
Angular intislization
K-means based initialization
(b) Angular distance initialization
Figure 3: Comparison of the initialization strategy in multi-SVDD. (a) The model with K-means initialization is effective the anomaly detection but for the OOD detection scenario. This is because the definition of the anomaly is when the samples do not belong to the cluster regions (dark shade), while the definition of OOD is when the samples do not lie the ID manifold (light shade). (b) Our proposed angular distance- based initialization guides the initial center vectors to be positioned following a (virtual) circular line. As result, it prevents the OOD samples from not belonging to the ID by creating tight cluster layouts without hole.
[[r,,-. rk]. In multi-SVDD, the objective function is defined follows.
K
1 vn
n max {0, [,, W) - c,113 13) Σ |i|l
min W.r
(1)
Here, 600;; W) is the deep network with set of weight parameters W and Cj is assigned to 602,; W). As the set is decreased, the samples are condensed into the center vectors. By us- ing the distance between the center vectors and the samples, we get an anomaly score.
Angular distance initialization. SVDD is proposed for the anomaly detection task. Because of the disparity between OOD and anomaly detection, the direct application of the SVDD-based model to OOD detection causes unexpected performance degradation. In anomaly detection, although the abnormal samples lie in the ID manifold, it is possible to detect them as abnormal unless they are close the center vectors Cj. For example, as Figure 3a OOD samples (red) that are located inside of the ID manifold (light blue shade) can be detected as abnormal since they are outside of the tight cluster boundary (dark blue shade). Because of these, a mixture of Gaussian probability density is reasonable density space for the anomaly detection model.
Unlike the anomaly detection task, the OOD detection is to find the samples that are not ID. In Figure 3a all the OOD samples placed in the ID) manifold (light dark shade) are recognized as the ID. We propose an angular distance-based center vector initialization strategy to tackle this issue. Angular-bases based embedding has a clear geometric interpretation (Deng et al.) (019)), and they can handle the complex and subtle differences in feature space. Our proposed method as follows:
C5 ? |v]l
v VER (0,,
where h is the dimension of the embedding space and 7 is the hyper-parameter for the radius of the sphere. After o(z;; W) is trained based on angular initialization, semantic features are extracted through this model; S(x)) o(0;; W). By setting the y value sufficiently large, we ensure that all sample data are within a radius of the sphere as illustrated in Figure 3b While Equation ! drives the training samples be embedded around the center vectors on the sphere, the OOD samples remain near the origin. This embedding space may be weak to recognize the semantic label of given sample, but sufficient to identify whether a sample OOD or not.
3.3 EXTRACTING THE TEXTURE INFORMATION
To effectively extract the texture property of the ID, interpret the image in the frequency space. With a given input image E p3xhx we first convert it into the frequency domain using Discrete Fourier Transform (DFT) J as shown below.
P- 1-11
F(fo, fu)
ΣΣ 1(p.g) 112(/.p/h + fy9/w)
(3)
hww
p=0 9=0
4

---

Under review as a conference paper at ICLR 2023
Here, I [pp.9) 9) denotes the pixel value of the image at the (p, -ocordinnt and F(fer fy) is the output of the DFT at the Cartesian coordinate (fa. fu) in the frequency space. In order to construct scale and rotation invariant frequency information in 2D image.
modify the coordinate system from Cartesian (Sa, fy) to polar (Sr,0) 0). following (Dzanic et al. 2019).
8 + f3
= F(aan
+ V (m22 +22)
0 = atan2 (Sus Sz).
(4)
Since directly computing the polar coordinate is computationally expensive and tricky, we iteratively calculate the rotation invariant frequency feature. To do that, we only utilize the first channel of the
image r E R' xxxw and assume that the image is square (i.e., h = w). Let (xx) Rw/2 be the texture feature vector of the image 2. Then, the i-th element of (2(x) is calculated by Equation 5
10
(x)x) R2 RG-1
Rw
F(J,,
Ro F(0,0)
(5)
w fy=-w
Note that fr represents the direction of the frequency of an image. Since (x)) gathers the boundary region of the rectangular-shap feature map, R2 (as Equation 5S. the same frequency power of an image collapses into (z) Consequently, T(x) loses the information of the object's 's shape and only contains the frequency- based texture information.
Discussion. We compare the power spectrum density (PSD) of CIFAR-10 (C10), CIFAR-100 (C100), and distorted C10 (Fig- ure 4 The PSD discrepancy between the corrupted C10 and the vanilla one indicates that the image feature acquired from the frequency domain is adequate to represent the texture cue. In contrast, C10 and C100 are not distinguishable quency domain since they have a very similar image texture due the small resolution. These observations support our assump- tion that the OOD detection model should decouple the texture and semantic information to handle real-world cases properly.
comprion
the fre-
C10
corruption
C100
Figure 4: PSD analysis.
3.4 FEATURE COMPOSITION VIA NORMALIZING FLOW
Since design our framework to directly sample the probability, it does not require any ad-hoc scoring functions. Instead, we use a normalizing flow-based method (Dinh et al. 2014; Rezende & Mohamed, 2013 Dinh et al. 2016) that uses the probability of given samples loss function. In the following, we will describe how to get the probability of samples from the prior probability (Normal distribution) using the normalizing flow.
Given sample x. a normal prior probability distribution pz on a latent variable 2 E Z, and a bijection f: X Z (with g -, the change of variables defines a model distribution X by
(.)()) (
px(x) = pz(((())) det
(6)
8f(x) is the Jacobian of f at x. The function f can be decomposed into / where d=
To compose the features, we use a flow-based RealNVP with coupling layers (Dinh et al.) 2016)
Here, T(x:) and S(x;; are input of RealNVP. Given the ID training dataset D. the objective of the RealNVP model with the trainable parameters and & is to minimize the following loss function.
N
N
(D) N
Σ
llg po (T():)l=;). (D)
logg (S(x)))))))
N
Although we extract image features from two different aspects, it is not guaranteed that the features are completely disentangled. To mitigate the unexpected entangling, we propose disentanglement loss using the independence property of log probabilities. T(x;) and S(x;) are independent given s; then log p( (x+)) + log p(S(x)) = log ((g((( S(x;)). ). Note that log ((T())), S(x:)) can be computed by concatenating features S(x) and T(x). The disentanglement loss calculated as:
1 Ldisentangle
(log)((((x)) + logp(S(z;))) log ((T(),
(8)
N
5

---

Under review as a conference paper at ICLR 2023
Using Labels Maha
Other Dist.
Self-supervised SSL
Ours
ID OOD
Gram
OE
OEC
Rot
CSI
0.0
0.5
1.0
C10
99.3
97.3
99.3
99.8
99.8
93.9
99.9
100.
ASS NHHS
C100
99.0
99.9
99.8
91.1
99.8
99.9
TinyImgNet
99.3
97.3
99.5
100.
100.
LSUN
99.9
99.8
99.9
99.9
99.9
99.9
100.
100.
SVHN
99.1
99.5
98.2
99.2
97.8
99.2
99.8
86.1
99.9
99.9
C10
C100
88.2
79.0
92.9
93.8
82.3
93.8
89.2
55.7
93.6
93.5
TinyImgNer
99.5
99.7
65.4
99.9
99.9
LSUN*
99.3
99.9
96.4
98.9
92.8
98.9
97.5
99.9
99.8
81.9
SVHN
98.4
97.3
82.8
95.8
95.8
80.0
99.9
100.
C100
C10
77.5
67.9
77.5
77.7
77.7
49.4
83.1
84.2
TinyImgNet
97.4
99.0
91.7
100.
100.
LSUN"
98.2
99.3
79.5
88.8
88.8
99.9
100.
100.
Table I: Conventional OOD detection benchmark. We evaluate the detection performance by AUC (in- vs. out-distribution detection based on confidence/score) in percentage (higher better). indicates the high-resolution dataset.
The final loss is the sum of all the aforementioned losses; - L; + + C. Edisentangle- One nice
side-effect of this probabilistic modeling for texture-semanticn decomposition is that adjust the contribution of these components by referring to the given OOD environment. Since we designed the features be separate, we combine them with linear interpolation as:
log (x))(a)) (1 >) log Po
where À E 0.5,, 1.0] is the control parameter (default 0.5). In our experiments, we denote 0.0 "texture mode" and À 1.0 "semantic mode".
EXPERIMENTS
We demonstrate the effectiveness of our method on various benchmark setups. In Section 4.1 report the performance on the conventional OOD detection task and discuss the limitations of the previous OOD detection studies. We use the under the curve (AUC) for the receiver operating characteristic (ROC) curve to evaluate the performance. Then, we will discuss the factor-aware OOD detection that is proposed fill the gap between the current benchmark and the real-world environment (Section
We conclude this part with model analysis (Section
4.1 CONVENTIONAL OOD DETECTION
Setups. Here, evaluate OOD detection methods on the widely used OOD detection benchmark. We select SVHN (Netzer et al., 201T), CIFAR-10 (Krizhevsky et al. 2009 (C10), and CIFAR- 100 (Krzhevsky et al.) 2010) (000) as the ID dataset. To simulate the OOD samples, LSUN (Yu
et al. 2015) and ImageNet (Torralba et al. 2008) datasets are additionally used.
Baselines. We compare our approach the methods belonging to three different OOD detection groups. 1) Methods that use class labels of the training samples; feature-based methods such as Maha (Lee et al.) 2018) and Gram (Sastry & Oore (020) fall into this category. 2) Methods that utilize additional distribution (dataset) such as OE (Hendrycks et al. (2018) and OFC (Papadopou- los et al. 202I). 3) Self-supervised based methods: rotation- based (Rot) (Hendrycks et al. 2019), SSL (Mohseni et al. 2020), and CSI (Tack al. (020) are in this group.
Results. As shown in Table I our proposed method surpasses the competitors on the conventional OOD detection tasks without using any extra information such class labels, other datasets, or mage-traansfrrmaiio techniques that the other methods required. In detail, ours with = 1.0 (semantic mode) achieves the best performance in all the cases with the exception of two scenarios.
Discussion. As we have discussed, it is often risky to assume that the input samples of real-world system have single and entangled characteristic. We argue that it is more natural consider in

---

Under review as a conference paper at ICLR 2023
terms of multiple factors such as texture and semantics when detecting OOD. For example, when given environment requires detection of the textural discrepancy as OOD, then the detection method should concentrate on the textural side alone, not the semantic counterpart.
Unfortunately, conventional OOD detection benchmarks are not developed to measure the disen- tangled view of the ID. Here, dissect the traditional benchmark to quantify the effect of each factor we decompose. We first categorized the datasets into the high- and low-resolution groups by their image resolution: C1O, C100, SVHN, and TinyImageNee
belong to the low-resolution (LR) group while LSUN is high-resolution (HR). Table T shows that previous studies achieve high per- formance on the LR + HR scenario, and especially the label-based methods produce outstanding results. However, we will show the superior performance of these methods derives from their abuse of texture information (e.g., detect as OOD by highly referring to image resolution) in Section 4.2
On the other hand, we argue that the semantic property is the key component to identity the OOD of LR + LR scenarios. For example, C10 ++ C100 solely requires semantic information to detect OOD since these datasets use very similar images (in terms of the texture and image resolution). This is the reason why ours with À = 0.0 (texture mode) cannot detect OOD at all (55.7 and 49.4 AUC). Note that other competitors also show inferior performance, especially in the C1O ++ C100 case, which demonstrates that these methods have weakness at handling semantic information.
4.2 FACTER-AWARE OOD DETECTION
Motivation. For a more in-depth evaluation of OOD detection performance, we propose a novel OOD detection benchmark in which multiple factors (or environments) are involved. Conventional setups simply define the ID and OOD as different datasets. Unfortunately, this assumption is often invalid in real-world use cases. Imagine a distorted image scenario; would we have to label this cor- rupted image as OOD? Most previous studies would obviously determine it as OOD, but this heavily depends on the surrounding circumstances; for example, an OOD detection system is granted a role to send an alert when a new class object appears, then it has to disregard mild corruption.
Therefore, the goal of this benchmark is to evaluate from the OOD detection performance (Figure 5) and b) and the robustness, whether the model can judge the given sample as ID when it differs from the training set while still coming from inside the "interest boundary" (Figure 5pp and d). One consecutive question may arise: what kind of factors should be considered? Certainly, in real-world applications, various factors can be derived from given samples or datasets. However, we narrow the scope of the factors to texture (Figure 5] and c) and semantics (Figure [5p and d), since these are the most important aspects considering the decision- decision-making making process in the real world.
Setups. To evaluate OOD detection performance, AUC is used; 100.0% AUC is the best. To mea- sure robustness, we define 50% AUC as the best since this indicates that the detection method cannot distinguish test samples OOD (determined as ID). In this benchmark, we use corrupted C1O, SVIRO (Cruz et al. 2020), BTAD (Mishra et al. (02T), MNIST variants, and rotated MNIST datasets shown in Figure A. Detailed descriptions of each dataset are described in the experience parts.
Detecting texture discrepancy of OOD, The goal of this scenario is detect the texture discrep- ancy. In real-world application, the factory process requires quality control to find cracks in indus- trial products. To meet the demand, we use BTAD dataset (Mishra et al. 202T. which composed of 2,830 cedustry--oiented images of normal and defective products. We set the normal samples as the ID and the cracked ones as OOD, Furthermore, as discussed earlier, we simulate the severe dis- tortion scenario by applying severe levels of frost, shot noise, haze, and motion blur to C10. In this case, the model is required to detect the distortions as OOD, simulating the cireumstances where the system should find the corrupted sensing module.
As shown in Table [ our method with texture mode (A = 0.0) outperforms all the others in BTAD case and achieves comparable performance for severe distortion scenario. On the other hand, the semantic mode (A- = 1.0) shows substantially lower AUC. This is unsurprising because the current benchmark scenario does not require semantic information; distortions and cracks (BTAD) are only related to the texture clue. Gram and Maha also achieve high performance; however, we find that these methods do not have sufficient robustness as will be discussed in the Robustness experiment.
7

---

Under review as a conference paper at ICLR 2023
(a) Detect. Texture
(b) Deteci. Semantics
AUC (Orack)
Goal - Factor
Experiments
BTAD MMsshra et al
2021
(12
Detect Texture
distortion
(OOO)
Dounsaay
throcet (O00)
100.
Rotated MNIST
Detect Seman.
(c) Robust.
Texture
(d) Robust. Semantics
MNIST++F-MNIST
Image resolution
Robust Texture
Mild distortion
50.0
SVIRO Cruz 2020
Robust Seman.
SANST+-K-MNSSS
dstanet (I))
datanel (E))
Figure 5: Taxonomy of our proposed benchmark. Left: (a, b) OOD detection scenario on two factors: texture, and semantics. In this case, the training set is ID, while the test set is OOD, (c, d) Evaluating the robustness on factor discrepancy of ID by providing ID) samples at test time. Here, oracle produces 50.0 AUC (cannot distinguish it OOI) all) since the given samples are from ID. Right: A list of the corresponding experiments in each scenario.
Severe distortion
BTAD
Rotated MNIST
MNISTs
Method
CTO->
Normal Anomaly
Method
0
MNIST
Frost
Haze
Motion
Shot
1.5
30
45
60
75
FMNIST
ODIN
79.9
69.5
80.9
84.2
52.2
ODIN
68.1
84.4
91.9
96.3
97.7
74.4
Maha
97.2
99.8
97.4
99.8
79.1
Maha
80.0
86.1
88.1
87.9
89.9
96.8
Gram
99.9
99.9
99.9
99.8
81.1
Gram
88.8
91.0
96.6
98.8
99.1
99.9
CSI
84.3
87.6
97.1
61.1
CS1
59.2
60.0
60.8
61.8
61.9
56.1
0.0
83.4
94.4
91.0
96.1
92.1
0.0
51.1
51.1
52.3
54.6
59.1
91.8
0.5
80.4
73.5
90.8
82.5
90.0
X
0.5
65.5
84.1
90.1
95.5.
91.1
100.
1.0
67.5
67.1
63.1
74.1
61.0
1.0
67.1
89.6
96.8
98.5
99.8
99.9.
Table 2: Comparison of the texture discrepancy detection OOD.
Table 3: Comparison of the semantic discrepancy detection OOD.
Detecting semantic discrepancy of OOD, With this benchmark, we evaluate the method's sen- sitivity to the semantic difference between the training and test samples. Catching semantic dis- crepancies is important in many applications such as autonomous systems. For example, this should judge whether given traffic symbol is geometrically distorted for stable recognition system (some recognition models are vulnerable to rotation). Here, we use two testbeds: MNIST- -FISST+F-MNIST and rotated MNIST scenarios. For the rotated MNIST, we train the model on MNIST and evaluate the rotated version. In most cases, our method with semantic mode surpasses the others (Table
Robustness Texture Diserepancy of ID. We conduct the benchmark distortion and varying image resolution. Both cases assume that the definition of OOD is in the disparity of semantic information. Particularly, subtle change of image should not be considered as signal of OOD, but rather recognized as ID. For example, the system that is requested to detect abnormal events should robust on the mild image distortions. To implement the distortion case, we corrupt C10 with mild level. For the varying resolution, we center-crop and resize back to the original resolution. We carefully adjust the crop operation so as not to harm the image information.
scenarios: mild
Table the AUC score of this benchmark. Since this experiment focuses on detecting the test samples as the ID, 50.0% AUC the best. We observe that the deep feature-based methods (e.g., Maha and Gram) have little robustness against the texture discrepancies. They not potentially appropriate for applications that do not treat a mild corruption as OOD. Furthermore, all the methods except with semantic mode are extremely sensitive image resolution change, although no other information is modified. In contrast, our method with = 1.0 (semantic mode) achieves the best performance because this focuses on the semantic information alone.
Robustness on Semantic Discrepancy ID. Unlike the above, we focus on the robustness against the semantic discrepancy presented in ID.. To evaluate this, KMISSS and
8

---

Under review as a conference paper at ICLR 2023
Mild distortion
Image resolution
SVIRO
MNISTs
Method
C10-
32
+
Method
A-class-2
MNIST
Frost
Haze
Motion
Shot
36
40
442
48
Tesla
Escape
Tiguan
i3
KMNIST
ODIN
72.6
75.2
80.9
84.2
55.7
62.7
67.5
91.2
ODIN
67.3
53.4
56.2
58.9
94,4
Maha
88.3
99.3
97.4
99.8
69.0
71.1
79.9
80.0
Maha
90.0
76.0
88.1
77.2
91.6
Gram
99.8
99.6
99.9
99.8
66.5
74.8
82.4
92.9
Gram
91.1
91.9
81.1
81.2
97.1
CSI
73.1
60.1
87.6
97.1
81.5
86.1
89.9
93.5
CSI
65.5
54.4
53.2
57.2
72.2
0.0
68.8
59.3
79.0
91.1
59.9
67.7
75.8
82.8
0.0 60.0
59.4
55.8
55.3
50.6
À =
0.5
64.3
58.0
71.8
82.5
50.2
55.8
71.2
72.8
X
0.5
90.0
79.2
79.0
76.6
100.
1.0
56.6 55.0
54.1
74.1
50.1
55.0
60.9 66.0
1.0
96.5
84.1
87.6
90.9
100.
Table Comparison of the robustness on the texture
Table 5:
Comparison of the robustness on
discrepancy of ID.
the semantic discrepancy of ID.
C10-
C100->
SVHN
Method
C10
C100
SVHN
Method
C100 SVHN
C10 SVHN
C10 C100
ODIN
56.2
53.3
52.2
Multi-SVDD
53.7
99.7
67.5
84.2
87.5 84.2
Gram
79.1
69.1
71.1
+ Lakeenttngle
69.5
99.8
70.9
91.0
99.9 98.9
CSI
61.9
52.2
51.1
Angular init.
93.5
99.9
84.2
100.
100. 99.9
Ours
50.1
49.8
50.0
Table 6: Ablation study on the semantic module. Multi-SVDD without our proposed components can- not properly handle the semantics information.
Table ID=ID scenario. All the models are trained on the "training set" of the ID dataset and evaluated on its "test set".
SVIRO (Cruz et al. 2020) scenarios. SVIRO the collection images from diverse vehicle interior rear seats. Among them, we set Mercedes A-class as ID and other car brands as the OOD datasets. With this, we simulate the following scenario: the machine learning system utilizing the camera installed inside of the vehicle should be robust to the vehicle type (semantics discrepancy).
For MNIST - K-MNIST (Table 5), our method with termines that K-MNIST is the IDD (AUC is 50.6). In contrast, our model with other À values (0.5 and 1.0) cannot, since these mostly depend on the semantic clue which is meaningless in this case. Similarly, our method with X = 0.0 (texture mode) achieves good performance SIVRO.
0.0 (texture mode) successfully de-
4.3 MODEL ANALYSIS
Semantic module. We validate the proposed component of the semantic module (Table 66) The disentangle loss plays an important role in the model performance and the angular initialization also improves the conventional initialization method in huge margin, implying the superiority of our method when extracting the semantic information. To further present the effectiveness of our strategy, we plot the distribution of the embedding results Figure B.2
ID + ID scenario. If we test an OOD detection model using the test set of the ID dataset, then ideally the prediction for should be ID. On the other hand, when the OOD detection model overfits the training set, then the AUC score for the test set would go to 100.0%. As shown in Table 7 our method determines the ID almost perfectly, and CSI and ODIN also show reasonable performance. However, the Gram-based method is rather sensitive to this and we argue that might be due to abuse of features of the seen (training) dataset.
5 CONCLUSION
In this work, we introduce novel viewpoint of the ID for the practically applicable OOD detection. We separate the definition of the "single-mode" ID into "texture" and "semantics" factors by fol- lowing the requirements of the real-world applications. To effectively handle both aspects, we take a sovide-and-congee and then combines these with the normalizing bowebased model. By doing so, method outper- forms previous models on both our newly propased benchmark scenarios and the conventional OOD detection cases. We hope that our work provide useful guidance for future OOD detection work.
strategy that extracts the features using the appropriate method in each factor

---

Under review as a conference paper at ICLR 2023
REFERENCES
Faruk Ahmed and Aaron Courville. Detecting semantic anomalies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3154-3162, 2020.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: (6l.:6660606665,, 2016.
Robin Chan, Krzysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, Mathieu Salzmann, Pascal Fua, and Matthias Rottmann. Segmentmeifyoucan: A benchmark for anomaly segmentation. arXiv preprint arXiv:2104.1811, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning. pp. 1597-1607. PMLR, 2020.
Steve Dias Da Cruz, Oliver Wasenmuller, Hans-Peter Beise, Thomas Stifter, and Didier Stricker. Sviro: Synthetic vehicle interior rear seat occupancy dataset and benchmark. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 973-982, 2020.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 4690-4699, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605 000.00000000000.. 2016.
Tarik Dzanic, Karan Shah, and Freddie Witherden. Fourier spectrum discrepancies in deep network generated images. arXiv preprint arXiv: 1ttl::/99110064444 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-traiie trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint 1 1. 2018
Zahra Ghafoori and Christopher Leckie. Deep multi-sphere support vector data description. In Proceedings of the 2020 SIAM International Conference Data Mining, pp. 109-117. SIAM, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. 122611 2019.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1812::880000000 2018.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using supf-supe learning can improve model robustness and uncertainty. arXiv preprint arXiv: 12340, 2019.
Katherine I. Hermann, Ting Chen, and Simon Korbblith. The origins and prevalence of texture bias in convolutional neural networks. arXiv preprint /rXiv: 1.0907,, 2019.
Yen- Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from duitt--ditti data. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pp- 10951-10960, 2020.
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised leam- ing with normalizing flows. International Conference on Machine Learning, pp. 4615-4630.
PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Polina Kirichenko, Pavel Izmailov, and Andrew Wilson. Why normalizing flows fail to detect out-of-distribution data. Advances in neural information processing systems, 33:20578 33:20578-20589, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). URL http:///ww. cs. toronto. edu/kriz/cifar html, 5:4, 2010.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distiibution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Yingwei Li, Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang, Wei Shen, Alan Yuille, and Cihang Xie. Shape-texture debiased neural network training. arXiv preprint arXiv:2010.05981, 2020.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 2017.
Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distribution detection. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15313- 15323, 2021.
Pankaj Mishra, Riccardo Verk, Daniele Fornasier, Claudio Piciarelli, and Gian Luca Foresti. Vt-adl: A vision transformer network for image anomaly detection and localization. In 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE), pp. 01-06. IEEE, 2021.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable ditoo--distribution detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 5216-5223, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Aristotelis-Angelle Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for diitt--distributtin detection. Neurocomputing, 441: 138-150, 2021.
JuneKyu Park, Jeong-Hyean Moon, Namhyuk Ahn, and Kyung-Ah Sohn. What is wrong with one-class anomaly detection? arXiv preprint MrXiv:2104.0979) 2021.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in Neural Information Processing Systems, 32:14707-14718, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Interna- fional conference machine learning, pp. 1530-1538. PMLR, 2015.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 4393 4402. PMLR, 2018
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram
matrices. In International Conference on Machine Learning, pp. 8491 2491-8501. PMLR, 2020.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. arXiv preprint 2xXiv:2007.08876, 2020.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 1978- 2008
11

---

Under review as a conference paper at ICLR 2023
Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang. Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1740-1749) 2020.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint wrXiv::506.03360 2015.
Alireza Zaeemzadeh, Niccolò Bisagno, Zeno Sambugaro, Nicola Conei, Nazanin Rahnavard, and Mubarak Shah. Out-of-distribution detection using union of I-dimensional subspaces. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 9452- 9461, 2021.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open recognition. In European Conference on Computer Vision, pp. 102-117. Springer, 2020.
Mingtian Zhang, Andi Zhang, and Steven McDonagh. On the out-of outt---distribution generalization of probabilistic image modelling. arXiv preprint arXiv:2109.02639 2021.
Ev Zisselman and Aviv Tamar. Deep residual flow for out of distribution detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13994-14003, 2020.
12

---

Under review as a conference paper at ICLR 2023
A EXPERIMENT SETUPS
A.1 MODEL. TRAINING
We use Adam (Kingma Ba 2014) with a weight decay of 5c-5 and batch size of 256 for 100 epochs to train all the modules in our framework. Extracted features by the ÆT and Es have the same 112 dimensions. We train and test with single NVIDIA 1080TI GPU.
A.2 MODEL ARCHITECTURE
Multi-SVDD. We use ResNet-18 18 as the encoder network and modify the dimension of the last layer to 512. All the hidden dimensions of semantic modules have the same as the original setup. The y parameter for the angular distance is set as 250 for all in-distribution datasets.
ReaINVP. We use the RealNVP implementation following (Izmailov et al., 2020 detail, each perspective model has 8 blocks of 8 flows and connected shift and scale networks each of which 16 dimensions of hidden layers.
affine coupling that is defined by the fully
A.3 OOD BENCHMARK
Datasets. Figure A.I shows examples of the datasets in the proposed OOD detection benchmark. To quantify the role of texture information, we use corrupted C10 dataset (Figure A.Ta). We distort images of the C10 dataset with frost, haze, motion blur, and shot noises (following C10-C set- tings (Hendrycks & Dietterich (019)). With such corrupted C10 scenarios, we try to simulate the weather change small sensing error of the vision system. Note that we separate the distortion sce- nario into mild and severe corruption cases to cover the various demands from the real applications. Specifically, the mild scenario is utilized to evaluate the textural robustness of the method. sim- ulates the environments where slight textural deviation should be ignored and the algorithm should concentrate on the semantics changes. For instance, the vision-based surveillance system requires such capability since it should be robust to minor changes (e.g., weather, dust, etc.) and detect the environment change regarding the semantics (e.g., trespassing). The severe distortion case, on the other hand, is used to measure the OOD detection performance with given textural change. A system that should notice the deviation of textural dissimilarity falls in this category.
Severe
Frest
Shot noise
Haze
Mation blur
Escape
13
Tesla
(a) Image corruptions
(b) SVTRO
4
3
3
M
K-NNIST
FANIST
46
Normal
Crack
(c) MNIST variants
(d) Rotated MNIST
(c) BTAD
Figure A.I: Examples of the datasets used in the proposed benchmark. The corresponding experiment for each dataset is illustrated Figure
displays the SVIRO dataset Cruz al. 2020), which is collection of vehicle interior images. Each car dataset is consist of images taken on the rear seats with six car brands (as shown in figure) and among them, the ID whole directions of Mercedes A-class rear seat and OOD
Figure A.
https: thw.. com zmai 1ovpave floommm
13

---

Under review as a conference paper at ICLR 2023
other datasets. With this dataset, we measure the robustness of the semantics change for the following potential scenario: The machine learning system utilizing the camera installed inside of the vehicle should not behave differently although the vehicle interior is changed. In other words, the system should be robust to the vehicle type (semantics discrepancy).
MNIST variants such as K-MNIST and F-MNIST used measure semantics-aware benchmarks (Figure (.Tp). I. In addition, we generate a rotated MNIST dataset by rotating images of the MNIST dataset as illustrated in Figure A.Id. Since MNIST-based datasets are composed of grayscale im- ages, we replicate the color channel by three times before feeding into the detection method.
To evaluate the ability detect the products' crack from the real environment, we the BTAD dataset (Mishra et al., (222I) as shown in Figure(A.Ip. In detail, we selected the cracked image from the test dataset of BTAD and reorganize it as the ID dataset contains only non-cracked (normal) product images and the OOD dataset has cracked images only.
Image resolution. When we are able to acquire the official pre-trained network or samples, we provide the x2x32 resolution images as input, which is de facto setup in this field. If not the case (such as our benchmark), we prepossess the image as to have 32x32 resolution following the official setup. In our method, except for the resolution change experiment, we resize the images to 1 112x 112 resolution to give enough information to the texture extraction module.
B MODEL ANALYSIS
C10
C100
Blend
Jigsaw
10
30
40
50
Figure B.I: Power spectrum density (PSD) analysis of the blending and Jigsaw augmentations.
Texture information analysis. Our texture extraction module T(x) is induced to lose the shape information so that we can ensure that the texture and semantic features are disentangled. To validate the texture information, on the transformed CIFAR dataset (Figure B.T). For transformation of CIFAR, we blend the C10 and C100 in half ratios with alpha-blending (denoted Blend). Also, parts and randomly arrange them as Jigsaw augmentation (denoted as Jigsaw). We observe slight differences in the mid-level frequency for Jigsaw due the hard crossing border the center of the image but no significant change. Moreover, even though the sample's shape is mixed in the alpha- blend case, it can be seen that PSD is almost identical. These inspections support that T()) forgets the semantics (especially the shape clue) as expected when embedding the features.
that (z)) concentrates
analyze the power spectrum density (PSD)
the image into four
Semantic embedding results. In Figure B.2,
we plot the embedding results of the proposed seman- tics extraction module to verify the effectiveness of using semantic information. To visualize, we apply principal component analysis (PCA) to the extracted feature and make the dimension three. Figure B.22 depicts the scenario where MNIST is used as the ID dataset and other MNIST variants are the OOD. Figure shows the case where C10 is used as the ID dataset while C100 and SVHN are OOD. For both results, we observe that the embedding manifolds of the ID datasets are easily separable from the OODs' counterparts, and our semantics extraction module S(x) concentrates on semantics information, such as labels of C10 and C100.
On the other hand, as presented in Figure B.2p and d, the semantics module S(x) cannot identify the textural change such the image distortions or resolution changes; all the embeddings lie in similar manifolds. These results are natural and expected since the semantics extraction module is designed to solely focus the semantics side, not the texture.
Power Spectrum Density (PSD) analysis. To validate our assumption that texture information could be handled via Fourier transform, visualize the PSD for all the datasets of the proposed
14

---

Under review as a conference paper at ICLR 2023
MNIST (ID)
C10 (ID)
K-MNIST (OOD)
C100 (OOD)
F-MNIST (OOD)
SVHN (OOD)
(a) ID: MNIST OOD:(K, FJ-MNIST
(b) ID: C10 OOD: C100, SVHN
C10 (ID)
C10 (ID)
Blur Mild (OOD)
Res. 36 (OOD)
Blur Moderate (OOD)
Res. (OOD)
Blur Severe (OOD)
Res. (OOD)
Res. 48 (OOD)
(c) ID: C10 OOD: C10 with motion blur
(d) ID: C10 OOD: C10 with resolution change
Figure B.2: Embedding results of the semantics extraction module. The proposed semantics module S(x) focuses the semantics information. (a, b) The embedding outputs between ID and OOD are easily separable since the ID and OOD datasets have distinct semantic information. (c, d) It fails to disentangle the textural difference cases such as motion blur image resolution change. This is a natural outcome since (zx) is not guided to follow the textural cues.
OOD benchmark. In the case of the MNIST variant datasets (Figure B.3 n), MNIST and K-MNIST show a similar spectrum because they have identical textures but only the semantics (letters) are different as illustrated in Figure A. On the other hand, F-MNIST has a different spectrum trend due the disparate texture (letters vs. cloths). In Figure B.3p, we compare C10 the C100 and SVHN datasets. Here, although C10 and C100 are separable in the embedding space (as shown in Figure B.2p)) which has semantic information, the frequency domain-based approach, in which only texture information is extracted, cannot distinguish them. The SVHN dataset, however, shows disparate PSD, implying that the texture extraction module can identify this effectively.
We visualize PSD analysis results when the image distortions are applied to the C10 dataset (Figure B.3 In many cases, the C10 dataset (blue) and mild corruptions (orange) appear as aligned spectrum, thus the texture-only mode may struggle in detecting mild corruptions OOD. However, based on the real-world applications, we assumed that the mild distortion would be treated as ID, in other words, to be the textural robustness (Table 4). In contrast, severe corruptions (green) are distinctive compared to original C10, indicating that texture information is useful for detecting the severe corruptions as OOD.
15

---

Under review as a conference paper at ICLR 2023
SN (ID)
C10
K-VNIST (COD)
C100
F-NNIST (OOD)
SVHN
30
10
30
(a) ID: MNIST OOD: (K, F)-MNIST
(b) ID: C10
OOD: C100, SVHN
C10 (ID)
C10 (ID)
Frost Mild (OOD)
Noise Mild (OOD)
Frost Severe (COD)
Noise Severe (OOD)
30
(c) ID: C10 OOD: C10 with frost
(d) ID: C10
OOD: C10 with shot noise
C10
C10
Haze Mild (OCD)
Blur Vildd (OOD)
Haze Severe (COD)
Blur Severe (OOD)
20
30
10
20
30
(e) ID: C10
OOD:
C10 with haze
(I) ID: C10 OOD: C10 with motion blur
Figure B.3: Power spectrum density (PSD) analysis. We extend the PSD analysis conducted on
Figure to all the benchmark datasets we proposed.
16

---

Under review as a conference paper at ICLR 2023
C MODEL COMPARISON
Table C.1 presents the OOD detection performance of various methods when C10 and C100 datasets set to the ID. We compare with R-Flow (Zisselman & Tamar 2020), MOOD (Lin al. 2021),
1-Dim (Zaeemzadeh et al. 2021) and G-ODIN (Hsu et al. 2020) algorithms. In the table, (c) and (r) depict the center-cropper and resized, respectively. To get standard image resolution (32x 32),
downsize high-resolution datasets by described downsizing procedure. Note that the resized datasets are similar the one presented in this work. We would like emphasize that we mainly compare the competitors to our method with the À 0.5 setting. This makes the comparison fairer because 0.5 does not require prior knowledge of which factor is important. Our method outperforms the previous algorithms in many cases and the proposed framework does not utilize any information other than the given dataset (C10 or C100) the training phase. Unlike ours, the other methods external information such class labels additional datasets boost the performance.
Ours À
ID - 00D
R-Flow
MOOD
1-Dim
G-ODIN
0.0
0.5
1.0
SVHN
98.2
96.4
98.8
86.1
99.9
99.9
TinyImgNet
99.6
65.4
99.9
99.9
C10
LSUN (c)
99.2
99.4
98.3
68.3
90.9
98.6
LSUN (r)
99.6
93.2
99.3
99.4
97.0
99.4
99.6
ImgNet (c)
98.1
98.7
85.2
98.1
98.5
ImgNet (r)
98.5
99.1
85.0
98.1
98.9
SVHN
95.1
85.8
95.9
80.9
99.9
100.
TinyImgNet
98.1
91.7
100.
100.
C100
LSUN (c)
96.8
93.8
95.3
65.6
91.9
92.2
LSUN (r)
98.9
77.6
95.7
98.7
97.0
99.8
99.6
ImgNet (c)
88.6
97.6
94.1
100.
100.
ImgNet (r)
93.7
98.6
94.7
97.7
98.2
Table C.I: Comparison on the conventional OOD detection benchmark.
We compare other
representative methods with various settings: R-Flow (Zisselman Tamar
2020). MOOD (Lin
al. 202T), 1-Dim (Zaeemzadeh et al. 202I) and G-ODIN (Hsu al. 2020). Note that (c) and indicate center-cropped and resized, respectively.
17


Reviews:
[{"review_id": "rkPHsStjqax", "reviewer": "Reviewer_W1ZC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduces a new framework for OOD detection by defining more broadly the in-distribution (ID) samples using the notion of texture and semantic. Based on this definition, they create new benchmark scenarios. Finally, they propose a new method which perform well on both old and new benchmarks.", "summary_of_the_review": "I would tend to recommend acceptance though I lack of expertise to have a strong opinion. The authors should really improve section 3.3. Apart from that my final decision will depend on other reviewers' opinion and the coming discussion.", "strength_and_weaknesses": "Strength:\n- a first attempt to clarify OOD detection task\n- use of SOTA tools to design their models (resnet +SSVD for semantic, frequency features for textures and normalizing flow to combine both)\n- achieve SOTA performances on different datasets\n- ablation study is conducted\n\nWeaknesses:\n- I guess OOD detection is also meaningful for data that are not images. As such it seems only applicable to images.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nCPLEqBK2pF", "reviewer": "Reviewer_xaty", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes a novel approach for OOD detection based on two branches. One branch operates on DFT coefficients and the other on pixels. The pixel branch combines A ResNet-18 backbone, a multi-SVDD [ghafoori20icdm] layer initialized with unit centers, and a normalizing flow. The frequency branch is directly evaluated by a normalizing flow. The authors argue that the pixel branch is suitable for detecting semantic OODs, while the frequency branch is suitable for texture OODs. \n\n", "summary_of_the_review": "As written in strengths/weaknesses, there is competitive experimental performance, interesting novelty (L_disentangle), but insufficient evidence about the contribution of the texture branch.  ", "strength_and_weaknesses": "Strengths\n1. Somewhat original approach to an important problem. \n2. Competitive performance with respect to approaches that train with class labels (Table C.1) on very small images.\n3. Initialization at the unit sphere and disentanglement loss appear to help (Table 6)\n\nWeaknesses\n\n1. \\lambda=1.0 wins in almost all conventional OOD experiments (Table 1). The exception is C10-LSUN*. However, this result does not repeat in Table C.1 where \\lambda=1 is better than \\lambda=0 in both C10-LSUN configurations. If \\lambda=1.0 always wins, then the contribution of the texture branch is unclear.\n2. Table 6 does not reveal \\lambda; it should be performed for all three values of \\lambda\n3. CIFAR-10 and CIFAR-100 have shared and related classes (dog - wolf, cat - tiger etc); this is not a suitable pair for OOD detection.\n4. OOD detection on small images has been largely solved (Table C.1). Further work should evaluate on tougher benchmarks such as Species, SMIYC or FIshyscapes. \n5. The distinction between anomalies and OOD samples does not reflect the current usage in the community. For instance, UCSD anomaly detection dataset [a] contains OOD objects. A recent review explicitly states that anomalies, outliers and novelties are synonyms but distinguishes low-level and high-level anomalies [b].\n6. How does one choose K?\n\n[a] Anomaly Detection in Crowded Scenes. CVPR 2010.\n[b] A Unifying Review of Deep and Shallow Anomaly Detection. Proc. IEEE 2021.\n\nComments\n\nThe equation in 3.1 (λ · log p(T (x)|x) + (1 − λ) · log p(S(x)|x)) is different than (9).\n\nMissing related work\n\nThe proposed approach appears related to [c]. The main difference is that [c] uses labels while the present approach appears to use a form of self-training.\n\nUnit centers appear related to [d].   \n\nDetecting semantic OOD samples with a convolutional model is not easy due to texture bias [e,f]. It would be good to discuss how to deal with that problem (eg. [c]).\n\nIt would be sensible to discuss applicability for dense anomaly detection [g].\n\n[c] Hybrid Models for Open Set Recognition. ECCV 2020.\n[d] ArcFace: Additive Angular Margin Loss for Deep Face Recognition\n[e] ImageNet-trained CNNs are biased towards texture. ICLR 2019.\n[f] Why Normalizing Flows Fail to Detect Out-of-Distribution Data. NeurIPS 2020.\n[g] SegmentMeIfYouCan - A Benchmark for Anomaly Segmentation. NeurIPS 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eafYxyMj_K", "reviewer": "Reviewer_UnLe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a definition of the in distribution data (and by extension OOD), where the authors decompose the definition of the ID into texture and semantics. This decompostion provides the flexibility to different scenarios by determining which view of the ID more suited for a given scenario. The texture pipeline is built form 2D FFT and the semantic component is build from ResNet and multi-SVDD.", "summary_of_the_review": "I am not totally sold on the efficacy of the method and I dont agree with the main thesis of the paper, i.e., separation of the texture and the semantic component of the images and use it to identify in and out of distribution data. As such I will lean toward rejecting the paper.", "strength_and_weaknesses": "The main strenghts of the paper are as follows:\n\n1) The paper is well-written and the main ideas are presented in a succinct manner.\n2) A problem of interest to the community is tackled in the paper, where better definition of in-distribution and OOD data will allow for deployment of deep learning based solutions in real-life scenarios.\n\n\nThe main weakness in my opinion is as follows:\n1) I don't believe frequency based mehtods are a good proxy for texture of a region. In fact most texture segmentation methods I am aware of don't use frequency based features.\n2) In principle if frequency components are indeed important for the OOD data, we should be able to learn it with a neural network. FFTs are just scaled dot product of the input with FFT basis.\n3) I am not totally sold on the distinciton of the semantics and the textures. How do we decouple the texture from the semantics. I did not get it from the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "REyVumW5SP", "reviewer": "Reviewer_BBXZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper outlines an approach to out of distribution [OOD] example identification. The approach makes use of the idea of explicitly separating high-level semantic content and low-level image frequency content and training separate classifiers whose outputs are combined to make the OOD decision. The semantic content extraction network uses a novel approach to data pre-processing intended to make it easier to identify OOD elements.", "summary_of_the_review": "The conceptual approach leads to a specific architecture that shows good performance on both the existing data sets and a new proposed benchmark data set. The concept of thinking about OOD detection as having separable problem characteristics enables new approaches to the problem.", "strength_and_weaknesses": "The overall analysis of the problem is interesting and provides a justification for the design choices and dual-pathway classifier. The angular projection of the training data to enhance OOD detection is also an interesting approach to capturing the concept of the training set boundaries.\n\nThe use of rectangular frequency binning, and discarding orientation, as a texture descriptor is very rough. Empirically, it seems to work well enough, but a more sophisticated set of features may improve performance. Does the process make use of an integral image to do the computation of rectangular sums? It's also not clear if the computation is being done on the base DFT or the power spectrum. The text for figure 4 implies it is the power spectrum.\n\nThe acronym SVDD does not seem to be defined in the text.\n\nThe balance of the two systems is evaluated at only lambda = {0, 0.5, and 1}.  It's not clear from the results that equal weighting will necessarily be the best option.\n\nThe new benchmark data set seems to be needed, given the performance on the prior data sets. It seems tailored to demonstrating this particular differentiation of the task (high level versus low level features).  My question is whether it has broader applicability.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject