Out-of-Distribution Detection via Conditional Kernel Independence Model
Authors: Yu Wang
Jingjing Zou
Jingyang Lin
Qing Ling
Yingwei Pan
Ting Yao
Tao Mei
Venue: NeurIPS 2022 Accept (2022)
Keywords: out-of-distribution detection
Hilbert-Schmidt Independence Criterion

Abstract: Recently, various methods have been introduced to address the OOD detection problem with training outlier exposure. These methods usually count on discriminative softmax metric or energy method to screen OOD samples. In this paper, we probe an alternative hypothesis on OOD detection by constructing a novel latent variable model based on independent component analysis (ICA) techniques. This novel method named Conditional-i builds upon the probabilistic formulation, and applies the Hilbert-Schmidt Independence Criteria that offers a convenient solution for optimizing variable dependencies. Conditional-i exclusively encodes the useful class condition into the probabilistic model, which provides the desired convenience in delivering theoretical support for the OOD detection task. To facilitate the implementation of the Conditional-i model, we construct unique memory bank architectures that allow for convenient end-to-end training within a tractable budget. Empirical results demonstrate an evident performance boost on benchmarks against SOTA methods. We also provide valuable theoretical justifications that our training strategy is guaranteed to bound the error in the context of OOD detection. Code is available at: https://github.com/OODHSIC/conditional-i.

Out-of-Distribution Detection via Conditional Kernel Independence Model
Yu Wang"! Jingjing Zou Jingyang Lin Qing Ling', Yingwei Pan', Ting Yao", Tao Mei*
1: Qiyuan Lab, Beijing, China
2. University of California, San Diego, USA
3: Sun Yat-sen University, Guangzhou, China
4; JD AI Research, Beijing, China
feather1004@@mail.. com, j2zou@ucsd. edu, yung. minjyy@aay@@ com
1ingqing5560mail1 wswp::asammmmss.sl........ sysu. edu. en, (panyw. ustc, tingyao wwt.://ttcaal....... com, tmei@jd.com
Abstract
Recently, various methods have been introduced address the OOD detection problem with training outlier exposure. These methods usually count on discrimi- native softmax metric or energy method to screen OOD samples. In this paper, we probe an alternative hypothesis on OOD detection by constructing a novel latent variable model based on independent component analysis (ICA) techniques. This novel method named Conditional-i builds upon the probabilistic formulation, and applies the Hlilbert-Schmidt Independence Criteria that offers a convenient solution for optimizing variable dependencies. Conditional-i exclusively encodes the useful class condition into the probabilistic model, which provides the desired conve- nience in delivering theoretical support for the OOD detection task. To facilitate the implementation of the Conditional-i model, we construct unique memory bank architectures that allow for convenient end-to-end training within a tractable budget. Empirical results demonstrate an evident performance boost on benchmarks against SOTA methods. We also provide valuable theoretical justifications that our training strategy is guaranteed to bound the error in the context of OOD detection. Code is available at: https:/gghhlb..oom/ooooo///////p
Introduction
The out-of-distribution (OOD) detection ability of models is at the core of achieving machine learning safety and reliability. Knowing how the classifiers perform on unseen OOD data remains as a challenging task, owing the train-test distribution divergence (3). Conventional computer vision tasks such as [8] 24 36
65| 666 77 (7)), NLP tasks [4] 75 and other machine learning tasks therefore need to carefully take into account the effect of potential distribution shift between training and test data. While deep generative models have become immensely popular, seminal OOD detection works consider applying generative modeling techniques to reconstruct the training data distribution density, so that statistical test can be implemented on the density. However, is found that such inference perform strong prediction artifact owing to various possible reasons 67 78 Instead, another line of work considers deploying discriminative methods on
47 53
54
55 64
12
48 55
38 41 where the detection is implemented by thresholding some class uncertainty measurements obtained from training inlier classifiers. However, as the train and test data i.i.d. assumption severely violated for the OOD detection task, only limited theoretical guarantees hold when deploying these discriminative models, There has been a surge of interest in the OOD detection task recently
OOD detection 28
37
9 18
59
60 69
Yu Wang and Jingjing Zou contributed equally to this work (joint first author).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unlike previous efforts that rely on canonical discrimination or generative models, we view the OOD detection problem from a new perspective of the independent component analysis (ICA). Our starting point is the hypothesis that OOD data should exhibit slight dependency on in-distribution training data, which can effectively constrain the OOD data to expose limited mutual information with the inliers. This hypothesis imposes the inliers to extract only little predictive information from outliers during training, and we distinguish OOD samples via the related independence measurements during the test. The contemporary work [39] attempted to probe the OOD problem from similar view. However, (39] merely demonstrates the promising practical values of such independence assumption based on empirical success, which lacks theoretical justification and supporting analysis. In this work, we propose a brand new motivating generative model, by incorporating additional class conditions into the latent variable model. This new work also aims to showcase the theoretical soundness through the transparent lens of ICA techniques, which provides desired technical convenience.
Our contribution to this paper includes: C1: We propose new OOD detection framework called Conditional-i model (reminiscent of fonditionnl-independene model). In comparison to (39]. Conditional-i additionally encodes the class condition into the probabilistic dependence model. C2: In order to exclusively facilitate the end-to-end training of the Conditional-i model, we construct an efficient memory bank architecture that constrains the training within a practical computational budget. C3: Conditional-i can be efficiently implemented both when training OOD data is available or not. C4: We provide theoretical justifications for new model. Empirical results demonstrate the evident superiority of our new proposal over the state-of of-the-art methods for computer vision and NLP tasks.
2 Related Work
Challenges for OOD detection. In 12 28 50, 67., empirical evidence shows that deep generative models trained on image datasets can often erroneously assign high likelihood to OOD inputs. In the meanwhile, as observed in 223 33 discriminative models also encounter a similar issue in returning high confidence on OOD samples or misclassified samples. Many contemporary works attempted to understand the principle behind this phenomenon. The work in (52) figured that the shared background statisties between in-distribution data and OOD data could interfere with the test. In [50, 67]. they discussed the impact of typicality on the the test. In (78). the work rather challenged the prevailing typical set hypothesis and the assumption that in-and out-distribution overlap. In (76). authors argued both accurate density estimation and discriminative classifier are critical, and henceforth proposed a hybrid model to address the issue. Research works also aim to alleviate the issue from the view of miscalibration [16] 19 23 44]. The work in [18] further demonstrated that deep architecture on the OOD detection performance task could have a significant impact.
OOD detection with statistical uncertainty. Adapting NNs to incorporate uncertainty and proba- bilistic methods has been of great interest in the community. The majority of this line of work rely on a Bayesian formalism, called Bayesian Neural Net (BNN). Many extensions along this path aim to estimate a posterior distribution over the DNN parameters (5) 20 42)
given some prior distribution on the DNN parameters. However, Bayesian NNs are often harder to implement and are prohibitively expensive in computational complexity. Some approach then considers reflecting a similar spirit of BNN 3 with reduced complexity. In the meanwhile, the statistical tests in differentiating OOD data are also of critical importance. In [ 577, hierarchical probabilistic models are used to estimate uncertainty on OOD data. In [38] authors found that in-distribution data are more sensitive to softmax temperature scaling and noise perturbations, under which they proposed a novel statistical test. In 34. the Mahalanobis distance is estimated to reflect the uncertainty of the test data classes. Particularly, a group of
There are many more explorations along this path (26 27 29 38
40
41
57
37 41). The approach in (28] assigned the "uniform label" on OOD training data to impose the uncertainty on OOD data. The model in [] introduced an energy- -ased score to replace the softmax score, achieving superior performance. The work in (37] investigated sampling policies of the OOD training data to improve OOD detection performance.
works assumes some OOD samples accessible during training B9
26
28
Conditional-i is the first that considers the independence assumption to address the OOD detection problem. Measuring independence and non-linear relationship between features is a classic technique, while independence component analysis (ICA) has powered conventional machine learning techniques 35 43 63). Some emerging work, such as in (79 empirically shows removing non-linear correlation

---

between features improves generalization for deep models. However, these models were built and motivated for drastically different tasks. For instance. (7] targets the specific domain generalization (DG) task, whereas Conditional-i is tailored for the OOD detection task, two entirely different tasks having distinct definitions, modeling assumptions, and evaluation metrics. In comparison to our earlier contemporary work HOOD [39], this paper is further motivated by modified conditional independence latent variable model. The fresh generative model behind Conditional-i provides unique technical convenience in delivering theoretical guarantees by allowing Conditional-i to utilize more flexible outlier regularizers during training.
3 Methodology
We introduce variable a R² to denote in-distribution (inlier) data, and let variable E R' represent the out-of--istrrbutte annotated pairs of data (æt; v!"), where indexes training samples. The label yf"h {1, C} Vin assigns each æ; one out of the C ground truth classes. A common assumption for OOD detection is that some outtt--diitribution (OOD) samples can be accessed during training, where are defined as samples from distinct classes y [28 41] We cannot access labels of outlier u, samples.
(outlier) data. The in-distribution training data set Dir consists of
We define 2; fx 0) E Rd be the features of inliers z;. Function f(: 0) the deep feature extractor with deep parameters 0 to be learned. The features of OOD sample u, denoted as q, are also learned through the same feature extractor, i.e., q. = s(u.; 0) E Ie Out-of-distribution detection is essentially binary classification task that aims to learn efficient f(. ((:;0) O) that can separate and discriminate between inliers x and outliers q at test time.
3.1 Motivating principle
In this paper, OOD samples are defined to be from the classes outside the inlier classes Vin. by strictly
following discriminative models (28] 34
41. This definition distinct from the generative model based detection methods 12 50 67 78 that samples from a different feature distribution are
38
instead defined be OOD data regardless of their classes. Mainstream discriminative OOD detection approaches usually rely on softmax metric (28) or the likes, e.g., energy score [41] to make decisions. These methods essentially compare the linear correlation between a feature and the classifier weights.
Our aim is to reduce the mutual information between training inliers and training outliers, since features representing different semantics should confine their predictive ability to each other. Note that reducing mutual information between variables implies reduced dependence, while dependence captures more general and complicated statistical relationship than linear correlation. As partial step towards this goal, our new model relies on an explicit conditional independence hypothesis that is shown to be both theoretical sound and practically superior the SOTA methods.
Pushing further than our contemporary work formulated as "HOOD" 33), our new OOD detection training strategy is built on novel generative model under particular conditional independence assumption. Specifically, we define each sample from the c class to be modeled as variable (e))
generated from a generative process glt) IRd +R... x (1):
(c) is the 7th sample realization of variable
(1) Here, w. are i.i.d. realizations of variable we)) and w(()) defines the classifier that reflects the common components shared between each sample in e class. Variable 33 denotes the latent variable defining the generative process of the outliers u through function g( Each 3, is i..d. sample with finite first and second moments. This suggests that we could model z across all classes E (1, (I,... C}.
(e) (w)
u (B))
each denoted (()), along with q variable to be jointly distributed given 0:
(2
(C) q) ~ p(2 (!) (",,,,,, (c) O)
(c)
(2)
We further define variable& & as the variable indicating the true class of feature z. The relevance of this construction then becomes apparent as per the following objective on requiring the independence between z and q given class e:
R(z q, c, 0) MMD (2 (5) qll c) p0 (= c) po(g)),
(3)
where pe denotes probability density parameterized by 0. Penalizing maximum mean dependency (MMD) [21 between the joint distribution on z (55) q and product of two conditional distributions
3

---

in Eq. (3) then explicitly regularizes the training so that inlier feature x of every class & = c, to be independent with outlier feature q. Deep parameter 0 is trained so that unseen OOD test data can generalize conditional independence given { c, and only lives in the constrained subspace subject to small loss incurred from Eq. (3). Contradiction against independence assumptions for any class ç = c would increase dependence metric between q and the generative process defined by { c.
The model above ultimately allows us to incorporate the class condition and the associated generative model into an integrated loss function, one that can be directly exploited and practically penalized. is of pivotal importance to note that E3.(3) strongly contrasts with the sampling strategy in HOOD [39]. It also manifests that since po(2; q) E. Po .g/5 = c)p(( c), then sampling from po (5) glk c) here by conditioning on class & further eliminates the sampling artifact introduced by p(5 c) of the training dataset. In this regard, conditioning on each specific avoids implicit marginalization over pl5 = c) as in [3] and instead encourages all in-distribution classes to produce independent features from outlier classes. Another compelling reason is that, such additional conditioning also provides technical convenience delivering our theoretical guarantees by linking to the generative model of each specific class, see Theorem
3.2 Conditional independence model for OOD detection
As proposed in [22]. the Sillbert-Schmitt Independence Criterion (HSIC) is effective measurement that quantifies the variable dependence through the eigenspectrum of covariance operators in repro- ducing kernel Hilbert spaces (RKHSs). In 222. it is guaranteed that the HSIC metric is zero if and only if the random variables are independent. We therefore anchor our modeling to this principled criteria, and employ the HSIC metric to encourage the conditional independence between and given each observed class e of z. To reach this goal, during each training iteration, we sample
pairs of samples S2,N {(21
(c) .q1), (x (c) .2....., (z (c) ...)..., (2 (c) 4rx) from each conditional ...,
distribution pe(z(6), glc = c) given each class e and current 0. Here, inlier samples z ..... =e() out
of set Sc,N are from the same ground truth class c. Then, for each class c, Eq. (3) can be empirically estimated via the estimator modified from 22]
R(2 0,
(N 1I tr ( HQ
(4)
Here the matrices ( H. By E R NXN are defined as follows. The (hh entry of matrix Φ is computed as $ (c) k(g.: qj)) Here (z.
2,6.j k(z, (c) (c) and for matrix p, the ( entry is denoted as og
(c) 9.) and (z) qq)) are two samples from S,,v. Matrix H I 111,, and
notation tr() computes matrix trace. In principle, the kernel function k(-,.) can take any legitimate form of characteristic kernel in the RKHS, such as Gaussian kernel (c) 2, = k(z (c)
(1 2r (1))
exp
linear kernel. The conditional independence loss sums over all classes is: 1 C C
R 1/2 Σ R(z,
(5)
In order to facilitate the practical training, we design a specific memory bank architecture in Section 3.3 under which the loss R in Eq. (5) is penalized in closed form. In the meanwhile, it is crucial
to include the conventional softmax classification loss train for semantically useful features for
labeled in-distribution data, where = is label of (r). exp
(m 2
1 Lin = CN
log
(6)
DC exp (w)(1)
The classifier variable w(() to be estimated was defined in generative process Eq. The overall loss that backpropagates through the function f(: 0) is:
L Lin + XR,
where X is a pre-defined penalty hyperparameter. The parameters 0 are optimized during the training.
3.3 Implementation of condition independence
In contrast to HOOD [39], Conditional-i model particularly incorporates the additional class condition into loss Eq. 5). Such new assumption introduces nuanced algorithmic complexity: Computing

---

R(z, q. c, 0) over all classes с as in Eq. 5 is prohibitively expensive and impractical for batch- wise training. To ensure Eq. ( stays practically useful, we propose an auxiliary memory bank architecture that allows for convenient end-to-end training. The general idea is to form C' number memory banks for computing each class specific matrix Φ During each training iteration, dynamically maintain and update memory banks V = {u1, u2...., Ve: ..... vcJ. All memory banks ve are of the size N (i.e., N features in each v2)) and are formed under a queue structure with "first in first out" policy. Features x from the current arriving in-distribution mini-batch is enqueued to each corresponding memory bank v, indexed by class c y ( and the oldest sample feature (im) is removed. During each training iteration, we randomly sample
(e) in the queue Ve with = Vj
C < C classes from V. and use the feature values stored in to compute (c) in Eq. (4) We then can compute Eq. with the current arriving batch of OOD feature q of size N, at the current O.
3.4 A novel statistical test
During test time inference, natural choice would be to reuse the independence metric of Eq. for test data. But we unfortunately cannot sample multiple i.i.d. samples to construct b, during test, since we have access test data distribution and By would be a constant. We resolve this problem by resorting to a surrogate function that can correctly reflect independence alternatively. As discussed in [22 62). HSIC independence metric be further reduced for standardized features if linear kernel used to construct Qy and (C yielding the independence metric:
tr( H |z"Q||
(8)
where Z Nxd has features zJ as rows, and Q E TR. Nxd has features q. as rows. We then follow this thread and modify the Eq. 8 into a novel practical test metric that suits Conditional-i training (different from )) during the test, we replace Q in Eq. (8) by a test feature constructed matrix S, where each row in S; E [g-Vxd repeats the identical test feature 8; gg We then concatenate all the training inliers in each class into each matrix (()) E BRdNN and compute:
4 max ( (c) S:llF.
(9)
Here || ||| is Frobenius norm. Eq. searches for the inlier class that returns the maximum conditional dependence with the test feature s;, We determine test feature s; to be OOD if t, < a for predetermined hyperparameter owing to significant independence with any possible class c; and classify it to be inliers if I 2 a, showing evident dependence with certain inlier class. By ranging across various threshold values implement the detection, obtaín the conventional FPR95, AUROC, and AUPR evaluation metrics by definition 28
4
Analysis
The analytic results presented here fully justify Conditional-i's validity as an advantageous OOD detection algorithm. Let F denote the function indexed by parameters θ such that for given input
(a
(C) uz). F(x. (1)
x (C) uu; 0) returns output features (z ,..., 2 (C) 9.)), where
2 (c)
f(x (c) Q) for 1, ..... C. indicating the feature extracted from incidence in class c. and q: flu;i 0) is the feature extracted from an outlier, and f is defined in Section 3. Our objective
Eq. 7 over the parameter 0 then can be viewed as the Lagrangian of the following constrained optimization problem over F:
min F
1
Σ> Σ log
exp((wo(0) (2))
(10)
CN
c exp((ur(()) 20
{z|0) (c) :i = 1, ..... N) 1 :q: 1,..., N),
s.t.
(11)
| |/ = 1, Vi,
(12)
where symbol 1. denotes independence between variables.
4.1 Condition-i versus HOOD
The distinction between HOOD and Conditional-i approach is made apparent by comparing the
HSIC metrics for dependence in the optimization problem. HOOD, the metrie for dependence
is HSICHOOD tr H/ 1)². where L24i k(2,, zz) with 21, žy being samples
5

---

from the mixture distribution Pmx()) = Ecs1 po(z [ c)p(5 = c). On the other hand, the metric for dependence in Conditional-i is HSICcond-i = き > tr tr
H 1)) where ( زييي - kz/ ) are samples from the conditional distribution of inliers given class ç = c. The outlier features that minimize HSICHOOD are those independent of inlier features from the mixture distribution. The outlier features that minimize HSICcond-1 are those independent of inlier features from distribution of each of the C' classes, and thus also minimize HSICHOOD. but the opposite is not necessarily true. A heuristic is given below in the scenario of d and linear kernels: As shown in [62] given the features are standardized, the HSIC metrics can be written as || ||=> and thus the minimizer q of Conditional-i orthogonal (inner product defined by the correlation) the space spanned by features of all classes {2(() c 1..... C}. Minimizers q of HOOD are those orthogonal to the linear combination Σe1 pl5 = c)z(1), which are not necessarily independent of each z (c), In fact, by choosing proper weights, can be a weighted combination of {z(() and still be a minimizer of HOOD (see more evidence and illustration in supplementary file).
4.2 Inlier and outlier independence guarantee
In terms of the optimum of optimization problem Eq. 10
adapt the technique from 43 to
support our proof (see supplementary file). But unlike previous efforts as in [43] our approach is anchored in a principled loss function Eq. (10) with drastically distinct objective, modeling and bounds tailored for OOD detection task.
We firstly introduce the used assumptions and definitions to support our analysis:
Assumption 1.
Admissible generative functions {g(()
: cc OJ and feature extractors {F(:;)) are third-
order differentiable.
ii) The Rademacher complexity of the function class (F((): for each and l is bounded by R
given N samples.
iii) Any third order derivative of function h1 = = (g()) ...... g(O), ((())) is in [-Ca, Cal, where
Ca is a finite constant. In addition, h( is invertible with respect w(e)
iv) There exists a positive constant v such that sup (1(()) (w."" (1) w (C)
B,) - w 1c S almost
surely for all
v) suppp
(f (1)
Cz (C)
u:)) Iuy (o) S Cj almost surely for all and I, where Cj is a constant.
Under the above assumptions and definitions, we propose the following theorem:
Theorem 1. Suppose F is solution the optimization Eq. - In addition, let p(s) denote xC), u) = =z(=) and let F( denote the thh element of the output of Fle) for = 1,... d. Suppose Assumption holds, then the following holds with probability of at least 1 8: V d1, d2 [l....,d) we have
the function such that F(o) (xx(1, "..,() x()),
( (x(1), ,...., (1),,,(0),,)) [
KC!/ ./33
E
(13)
O()),,
where V KNRN + 107 (og((/0)/(2N) and K and K₁ are positive constants.
Theorem Texplains why the independence assumption is beneficial for OOD detection. If we could achieve perfect independence between z and q through the network / conditional on each E = e, then under mild conditions, we obtain accurate disentanglement between the in-distribution feature and the true generative variable 3 that have genuinely generated OOD data, up to limited error. In other words, the variation in OOD generative /3 does not cause change in z, literally described in Theorem []] As the Rademacher complexity increases, the required number of training samples to achieve accurate independence also increases, in order to effectively bound our disentanglement error between the in-distribution feature and the true generative process OOD samples.
In this way. Theorem I alllws us to quantitatively describe how well the inherent generative process of in-distribution data and that of OOD data are disentangled up to an error. we were using a conventional softmax classifier to train for the in-distribution data, in the presence of the OOD data, the best disentanglement we can achieve bounded by Theorem [] This justifies the legitimacy and relevance of our proposed method for the OOD detection task, where the detection goal is to

---

differentiate the samples belonging to different generative classes from the known in- in-distributional data. This hopefully may better generalize the training to unseen test OODs as long as the generative process of the test OOD is genuinely different. Theorem | Jonly considers the scenario where perfect independence is achieved through penalizing Eq. A Certainly, we could further introduce an error term that bounds how much Eq. (4) is deviated from zero. We nevertheless choose to present the most simplified version here for clarity. We defer the more complicated derivation that Eq. is nonzero to future work.
5 Experiments
In this section, we evaluate the performance of the Conditional-i method against the state-of-the-art OOD detection methods both computer vision and NLP (natural language processing) applications. Across all the experiments, the Gaussian kernel is used to compute loss Eq. (5). We evaluate the models under two setups: 1. The training inliers data and training outliers are assumed available, but the labels of training outliers are unknown. 2. The training outliers are no longer available, and manually generate fake outliers by applying strong augmentation [14] and CutMix [72] on the training inliers. All models are tested on mixture of test inliers and outliers with the goal of correctly identifying outliers. We compare all methods in terms of FPR95, AUROC, and AUPR metrics [28]
5.1 Datasets and baselines
Datasets. In-distribution training data: We respectively use the CIFAR- 100 and ImageNetIK as the in-distribution training data for image OOD detection tasks. We employ the 20 Newsgroups data as the in-distribution training data for the NLP OOD detection task. OOD training data: By
following the protocol in (28), we use the 80 Million Tiny Images 61 as the OOD training data when the In-distribution data is CIFAR- 100 (3). We let the ImageNet21K (54] be the OOD training
data when ImageNetIK K [54] is the in-distribution training data. We define WikiText-2 [46] to be training OOD, when 20 Newsgroups is in-distribution data. In-distribution test data: We the validation data from the corresponding in-distribution dataset as the in-distribution test data as defined
in (88) OOD test data: We employ the benchmarks: DTD 13), SVHN [ST), Places69 (8), LSUN 7T1. and CIFAR- 10 [32] as the OOD detection test data for computer vision tasks; We use SNLI [7]. IMDB (45), Multi30K [1]. WMT16 (6) Yelp [21, and EWT 4 as OOD test data for NLP tasks. The specific training in- and out- distribution pairing strategy is (illustrated as inlier training data: OOD test data) CIFAR-I00- DTD, SVHN, Places365, LSUN, and CIFAR-10; ImageNetIK: DTD, SVHN, Places365, and LSUN; 20 Newsgroups: SNLI, IMDB, Multi30K, WMT16, Yelp, and EWT.
Baselines:
The alternative SOTA baselines are K-Base 27. ODIN 38 Mahalanobis distance (3]
OF (28) Energy [41], ReAct [59]. GradeNorm 30 and HOOD 39 Here, K-Base [27] corresponds
to the model trained through simple softmax loss using in-distribution data, while the OOD detection test is implemented by thresholding the maximum softmax prediction score.
5.2 Training architecture and optimization
We use the WideResNNet28--U 73 and ResNet18 25| as the training backbone for vision tasks, with the final deep feature dimension d 640 for WideResNett22-110 and d 512 for ResNet18. We
employ the SGD optimizer with IT = 0.1, Nesterov Momentum 0.9, for both Table and 2 We use two-layer GRU [10] for producing Table 3 with Adam (31) optimizer. For Table I and
adopt
batchsize = 256 and 512 respectively for in-distribution data. For Table 3 we use batchsize = 64
for in-distribution data. Every benchmark method in Table 1 and 2 is trained for 100 epochs w.r.t the size of the in-distribution training data. For NLP experiments, we train 2-layer GRUs [1] for 5 epochs for Table 3 (See supplementary material for details). We train each algorithm on P40 GPU. In contrast to the sampling as in (39] and (28) which sequentially sample random blocks of data from 80 Million Tiny Images, we firstly randomly shuffle the training dataset before sampling. The shuffling is expected to eliminate the sampling bias owing to the inherent correlation between the sample ordering and the categories. We found that such shuffling has improved all the baselines. We believe we have tuned all the baseline models to our best. By following the protocol in (88) we
"This dataset seems to be suspended recently. We therefore will supplement the result of Table by training Conditional-i on ImageNet22K/300K Random Images (28) (A subset of 80 Million Tiny Images) instead, and shortly release the results code link
7

---

Table I: Averaged OOD detection performance. Performances on each specific test dataset are in sup- plementary material. D CIFAR-100, D 80 Million Tiny Images, D (DTD, SVHN, Place365, LSUN, CIFAR-10). "True Out. Expo." " means known outlier training data exposure.
Drvos
Method
True Out. Expo. FPR95(S))
AUROC(S)
AUPR(S)1
K-Hase 12
65. 65.842077
74.7620.3
13,252004
ODIN 98
62.64=15
79.06-03
38.46:12
Mahalanabis
n0
53.77
82.72.01
48. 48.39207
(WidcResNeO)
CLEAR-100
ReAct (+Energy]
Energy [4]
B0
62.25
59.37.00
79.90=u2
79.04:04
35.79:06
36.62.205
OF: geneeetttiveee
25
53.29
83.44:0.3
$1.25::04
HOOD geDD-ggnerttvee
54,91=03
83.12:0.4
18
Conditionall-geeerriee (ours)
00
51.09
83.74-1.5
56.69
Lnergy 41
yes
33.
89.57=0.1
56.92:04
OE 28
yes
=3
89.75=0.4
57.74
HOOD
33. 06:06
89.81
19.18:110
HOOD bank
yes
32.97=0.3
89.87=022
60.16::06
Conditional- (ours)
y05
32.66:422
90.03-011
61.48:005
Table Averaged OOD detection performance. Performances on each specific test dataset are pro- vided in supplementary material. D ImageNet- IK, D = ImageNet-221K, Diod (DTD, SVHN, Place365, LSUN). "True Out. Expo." " means known outlier training data exposure.
Periin
Method
Out. Expo.
HPR95(%)
AUROC(S) It AUPR(S))
K-Base
no
66.05=12
72.98
32,95+009
ODIN
52. 18:08
80.41
44.53105
Mahalanohis
14
no
75.80=226
61.10200
23.79
ImageNetIK
Energy
4
54, 14.15
78.73=0.3
42.36
(ResNet18)
ReAct (+Fnegyy)
nO
14.592.8
78.9620.1
42.94..0.
OE OE-generative
25
63
74.11
38.13++20
HOODDgenzrative
BD
$2.28::09
80.16
44,38+077
Connditionll--genneraive (ours)
B0
50.20
81 81.20
44.55
Energy 41
48, 03
84,5620.3
46.06110
OE 28
yes
17.32
81 81.47=00
45.37
HOOD
30
46, 30..0
84, 14.28
50,50.a..2
HOOD-band bank
yes
42:00:
84.67.04
47.31
Conditional-i (ours)
42.84:09
86.79:002
55.00
report the average performance by using the consistent hyperparameters, without tuning for each specific test dataset. The detailed results of each specific test can be found in the supplementary file.
5.3 Benchmark results
Image OOD detection tasks. We first present the OOD detection results in Table D where the CIFAR-100 dataset serves as in-distributional data, with 80 Million Tiny Images being the OOD training dataset. DTD, SVHN, Places365, LSUN, and CIFAR-10 are tested OOD data. We tuned all the compared methods based on the validation set defined in (28). It manifests from Table [ that Conditional-i generally outperforms all the existing SOTA OOD methods in terms of FPR95, AUROC, and AUPR metrics. Particularly, Conditional-i surpasses our earlier HOOD method by an evident margin. justifying the unique advantage of the Conditional-i model discussed in 4.1 We found that the Conditional-i approach is especially efficient in differentiating near OOD samples, such as distinguishing CIFAR-10 from CIFAR-100, while it also provides decent performance against large domain gaps, e.g., CIFAR-100 vs. DTD. In Table 2 where ImageNet KK is inlier and ImageNet-21K is outlier, the performance of Conditional-i is even further improved when finer classes of OOD training data are used for training. This verifies that, by capitalizing on the conditional-indepenee
assumption, Conditional-i exploits unique constraints when searching for the OOD feature parameterization. and henceforth shows advantageous generalization ability under the finer classes' OOD discrimination. This also best supports the relevance of our Theorem [ that Conditional-i can distinguish inliers and outliers depending the true generative process of test samples given their class conditions.
Training without access to known true OOD training data. Proceeding further, we compare all models when we lose access to the oracle training OOD data. In this scenario, we use generated fake OOD training data by applying strong augmentation [14] and CutMix (72) (for CV tasks) or EDA (68) techniques (for NLP tasks) on each in-distributional data (see supplementary material). Each "XX" baseline trained with fake OOD training data is referred to as "XX- "XX-generativee" (e.g.. Conditional-i generative). As displayed in Table [ and 2 annditionali-generatiie performs relatively worse than the vanilla Conditional-i model, owing to the absence of true OOD data. However, Conditional-i- generative not only beats all SOTA models without OOD training data exposure, including K-Base, ODIN, Mahalanobis distance, Energy, and ReAct(+Energy), but also exhibits evident gain over other exposure methods, i.e., OF-generative and HOOD-generative, Here, ReAct(+ (nnegyy) corresponds to the ReAct [59] method based on Energy
8

---

Table 3: Averaged OOD detection performance on NLP tasks. Table Ablation on the number of classes
DI 20 Newsgroups, D WikiText-2, Doot IMDB, Multi30K, WMT16, Yelp, EWT).
(SNLI, C' for computing Eq. (5)
C
5
10
100
1000
Method
True Out. Expo.FPR95($5)4)
FPR95
47.08
47.18
46.83
45.43
42.84
K-Base 22
no $2.84231
84.27.17
41.41:67
AUROC
84.98
85.14
85.63
86.79
ODIN [5
m1 46.71=355
16.90112
50.98+511
AUPR
47.42
47.73
49.87
49.24
55.00
Mahulannbi
70.09 215
65:37a21
15.70443
Energy
41,000:41
19.95116
54.64174
ReAd (+Energy
I10
40,24+443
90:21116
55.28+74
GrddNNom
OE-generative
10
23
I10
19.89=11
39 1
86.20=m1
K5. 36:n3
61.95+14
79.02:44
Table 5: Ablation against in-distribution
HOOD-gonetaive 19
I00
4
92.51=04
batchsize N.
Eegeeerive (ours)
I00
33.17
93.12:84
78.94110
Energy 21
yes
2.79
99.25=m02
93.67+20
16
32
64
128
256
DE
yes
2,99
99. 16:n]
94,74+03
FPR95
33.37
34.02
33.54
33.02
32,66
HOOD-bank
y2s 1.83:03
yes
2,03:04
99.45=00
99.14=01
95.87+122
AUROC
87.43
89.79
89.91
89.95
90.03
Conditional-i (ours)
y08
1,52:93
99.56:4J
96.45:114
AUPR
47.30
57.28
61.09
61.30
61.48
NLP OOD detection tasks. We observe from Table 3 that Conditional-i offers the strong OOD detection performance on texts across all the evaluation metrics. The advantage of Conditional-i is particularly evident in the AUPR metric, that the Conditional-i model is more than 1% better than the HOOD model, and is 2% better than the OE method. Note although GradeNorm 30 shows superior performance in AUPR metric when training OOD is unavailable, it nevertheless performs poorly in terms of the other metrics, showing the unique benefit of Conditional-i on NLP applications.
5.4 Ablation study
Ablation on memory bank architecture. The proposed memory bank architecture is necessary and exclusively designed to facilitate the Conditional-i training. However, explain the advantage of conditional independence modeling hypothesis against HOOD, it is crucial to ablate the effect of the additional memory bank that is absent HOOD training. We refer to this new baseline as HOOD-bank, which is constructed with the identical memory bank architecture of Conditional-i. The critical difference here is that each out of C memory bank for HOOD-bame with random in distriiibbttinn data regardless of their class category. Each arriving in in-iitribution data randomly chooses 1 out of C memory banks with a probability 1/C and joins the corresponding queue. We compute the batch-wise loss function as in Eq. (5) by ranging across Ò number of memory banks of inliers data, where E < C' classes are sampled from C' total memory banks as defined in Section 3.3 In this way, samples in Φ are no longer sampled i..dd by conditioning the class variable e for HOOD-bank. Rather, samples in each bank follow the marginal distribution of p(z10) by margining out the classes. Tables 2 3 clearly demonstrate that, the memory bank architecture alone, ie., HOOD-bani method, can hardly boost HOOD. The advantage of Conditional-i is mainly attributed to the new conditional latent model. This result directly justifies Section 4.1
instead filled
Ablation on number of classes Ć. We investigate the impact of sampled class number é' included for computing Eq. (5). by following the procedure in Section 3.3 The training setup is the same as for producing Table 2 We still keep using the total number of classes present in the in-distributional data to train the inlier classifier through Eq. (6. In the meanwhile, we reduce the number of classes
included in Eq. (5) to Ò < C' and examine the regularizing impact of the additional condition on the OOD detection performance. As shown in Table 4 as more in-distributional data join the loss Eq. (5) the network gradually learns to shrink the subspace in which the OOD training sample can live in, so that the OOD features respect the independence condition posed by each in-distributional class. While we increase the value of Ć, the performance keeps improving across all the metrics, until all classes are included.
Ablation on in-distributional batchsize N. We vary the in-distributional data batch size during training, with the training setup identical to Table Since computing Conditional-i loss involves the estimator construction as in Eq. 4. the number of samples used to compute the kernel matrix $5 and & would affect the estimator bias in terms of the actual independence. As displayed in Table 3 the OÔD detection performance significantly drops when the in-distribution batch size is below 32, owing to the increased estimator bias introduced through Eq.
Ablation on weight X. The weight X balances the classification loss and the strength of the conditional independence loss. When = O. Conditional-i degenerates to the K-Base method. When À = 0.03, the OOD detection performance reaches sweet point owing to the inclusion of the kernel based

---

$2.5
22.5
0.006
0.11
011
14.0
0.500
1.00
0.33
0026
601
6.50
(a) FPR95
(h) AUROC
(c) AUPR
Figure 1: Ablations against weight
Table Ablation study applying different test method on different training models.
Table Ablation study against augmentation for
Conditional-is -nneitionall-geenaaaiv...
Train (Test)
FPR95 1.],
AUR0
AUPR %
Train (TeR))
FPRO5 S1
AUR AUROC(977
AR %77
K-Base (SFM)
72.
72.0%
K-Base (Cond-i)
CCnmix
Conditional-i (SFM)
56,76
81.20
48.37
KandAug
56.15
79.31
42.47
Conditional -i (Condi 6)
42.84
86.79
55.00
-RandAngg-Cttii
50.20
81.20
44.55
independence calculation. However, beyond À 0.03, the performance starts to drop again, partially because this overemphasized independence starts to hurt the in in-distribution semantic feature extraction that relies the accuracy of inlier classification. Learning independence from the noisy semantics having high classification loss would then lose the benefit.
Ablation on test method. Note our proposed test method Eq. (9) was originally motivated from the training objective Eq. 77) which exclusively couples with Conditional-i training loss. Nevertheless, it still worths trying testing the Conditional-i test/train alone independently. In the Table we use Train (Test) to indicate the model trained with "Train" " method and tested with "Test" method. Other training details are the same Table 2 We apply the Conditional-i test the model trained via conventional cross entropy loss, which is denoted as K-base (Cond-i). In contrast to the model K base (SFM) which employs softmax score (SFM) for test, K-base trained model may hardly benefit from the new test. In the meanwhile, we also observe that Conditional-i (SFM) may hardly beat Conditional-i (Cond-i). showing the critical role of coupled train/test pair. This again verifies that Conditional-i a holistic procedure with principled motivation and theoretical support.
Ablation augmentation for aanditional-i-geeeraate This section evaluates the impact of applying different augmentation for the proposed Conditionnlllrgeecratiie model. The ablation study is shown as below (other training details are the same as Table In Table 7 "+Cutmix" indicates Cutmix augmentation (72) and "+RandAug" indicates the strong augmentation [14] As can be seen from Table strong augmentation along with the Cutmix (i.e.. (Rand/Au+Cuutill would lead to the best results, whereas other augmentation methods would also help Conditional-i to surpass SOTA methods.
Conclusion and Broader Impact
We propose novel OOD detection approach referred as Conditional-i. Inspired by the ICA techniques, Conditional-i is shown to be effectively penalizing the deep feature dependency between inliers and outliers. Conditional-i models conditional independence by incorporating the condition information e into the probabilistic model. Such independence can be explicitly penalized through the use of the HSIC metric. We constructed an efficient memory bank architecture critical for the practical implementation of the method in an end-to-end training manner. Empirical evidence justifies the superior performance of the proposed method on both computer vision and natural language processing benchmarks. We believe OOD detection is an important task regarding Al safety, The accuracy of OOD detection directly impacts the reliability of many Al applications, including healthcare, anomaly detection, autonomous driving, and others. The negative impact is that the development of the Conditional-i method is subject to the limitations of current SOTA scores we can achieve, which may limit the actual safety of the AI applications. Whereas Conditional-i achieves the SOTA performance, another limitation though, is that Conditional-i needs be trained on (fake) OOD training data, that might potentially encounter generalization issues, by virtue of (3]
10

---

References
[1] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Danicla Rus. regression. In NewrIPS, 2020.
Deep evidential
2] Nabiha Asghar. Yelp dataset challenge: Review rating prediction. arXiv:1605.05322. 2016.
[3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 2010.
[4] Ann Bies, Justin Mott, Colin Warnr,, and Seth Kulick. English web treebank Idc2012t13, web download. Philadelphia: Linguistic Data Consortium, 2012.
[5] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty neural network. In ICML, 2015.
[6] Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana L.. Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin M. Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Workshop on Statistical Machine Translation, 2016.
[7] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015,
[8] Qi Cai, Yu Wang, Yingwei Pan, Ting Yao, and Tao Mei. Joint contrastive learning with infinite possibilities. In NeurIPS, 2020.
[9] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In ECML PKDD, 2021.
[10] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. EMNLP. 2014.
[11] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. EMNLP, 2014.
[12] Hyunsun Choi, Eric Jang, and Alexander A. Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv:1810.01130 2019.
[13] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014.
[14] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
[15] Jacob Devlin, Ming-Wos Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI, 2019.
[16] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models, In NeurIPS, 2019.
[17] Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30k: Multilingual english-german image descriptions. In Proceedings the 5th Workshop on Vision and Language, ACL, 2016.
[18] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. In NeurIPS, 2021.
[19] Will Grathwohl, Kuan-Chieh Wang. Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In ICLR, 2020.

---

[20] Alex Graves. Practical variational inference for neural networks. In NeurlPS, 2011.
[21]
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. Journal of Machine Learning Research, 2012.
[22] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In ALT, 2005.
[23] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[26] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In CVPR. 2019.
[27] Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and -ut-of-diittiiuution examples in neural networks. In ICLR, 2017.
[28] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
[29] Y. C. Hsu, Y. Shen, H. Jin, and Z. Kira. Generalizedodin: Detecting out-of-distribution image without learning from dut-i--distiitution data. In CVPR, 2020.
[30] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurlPS, 2021.
[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[32] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[33] Balaji Lakshminarayanann Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurlPS, 2017.
[34] Kimin Lec, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018
[35] Yazhe Li, Roman Pogodin, Danica J. Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. In NeurlPS, 2021.
[36] Yehao Li, Ting Yao, Yingwei Pan, and Tao Mei. Contextual transformer networks for visual recognition. IEEE Trans. on PAMI, 2022.
[37] Yi Li and Nuno Vasconcelos. Background data resampling for outlier-aware classification. In CVPR, 2020.
[38] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
[39] Jingyang Lin, Yu Wang, Qi Cai, Yingwei Pan, Ting Yao, Hongyang Chao, and Tao Mei. Out- of-distribution detection with hilbert-schmidt independence optimization. arXiv:2209. /2807, 2022.
[40] Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distributtin detection. In CVPR, 2021.
[41] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based disti---ittooottiio detection. In NeurlPS, 2020.
12

---

[42] Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. ICML, 2016.
[43] Qi Lyu, Xiao Fu, Weiran Wang, and Songtao Lu. Understanding latent correlation- multiview learning and self-supervision: An identifiability perspective. In ICLR, 2022.
[44] Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In NeurIPS, 2019.
[45] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011.
[46] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv:1609.07843, 2016.
[47] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021.
[48] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In ICML, 2021.
[49] Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alex Alemi, and Joshua Dillon. Density of states estimation for out of distribution detection. In AISTATS, 2021.
[50] Eric Nalisnick, Akihiro Matsukawa, Yee whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't Ion't know? In ICLR, 2019.
[51] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[52] Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for -fitt-riiitiitttin detection. In NeurIPS, 2019.
[53] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Trans. on PAMI, 2017.
[54] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
[55] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning point clouds by recon- structing space. NeurlPS, 2019.
[56] Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self- eepf-supe outlier detection. In ICLR, 2021.
[57] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty.
NeurIPS, 2018
[58] Alireza Shafaei, Mark Schmidt, and James Little, A less biased evaluation of out-of-distribution sample detectors. In BMVC, 2019.
[59] Yiyou Sun, Chuan Guo, and Yixuan Li. ReAct: Out-of-distribution detection with rectified activations. In NeurlPS, 2021.
[60] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In NeurlPS, 2020.
13

---

[61] Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Trans. on PAMI, 2008.
[62] Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov. A note on connecting barlow twins with peggativessamle-feee contrastive learning. arXiv:2104.13711
2021.
[63] Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In NeurlPS, 2021.
[64] Xueyang Wang, Xiya Zhang, Yinheng Zhu, Yuchen Guo. Xiaoyun Yuan, Liuyu Xiang, Zerun Wang, Guiguang Ding. David Brady, Qionghai Dai, and Lu Fang. Panda: A gigapixel-level human-centric video dataset. In CVPR, 2020.
[65] Yu Wang, Jingyang Lin, Qi Cai, Yingwei Pan, Ting Yao, Hongyang Chao, and Tao Mei. A low rank promoting prior for unsupervised contrastive learning. IEEE Trans. on PAMI, 2022.
[66] Yu Wang, Jingyang Lin, Jingjing Zou, Yingwei Pan, Ting Yao, and Tao Mei. Improving self-supervised learning with automated unsupervised outlier arbitration. NeurIPS, 2021.
[67] Ziyu Wang, Bin Dai, David Wipf, and Jun Zhu. Further analysis of outlier detection with deep generative models. In NewrIPS, 2020.
[68] Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP. 2019.
[69] Jingkang Yang, Haoqi Wang. Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent -iittr-diitiiuttionn detection. In ICCV, 2021.
[70] Ting Yao, Yingwei Pan, Yehao Li, Chong-Was Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In ECCV, 2022.
[71] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv: 03365, 2015.
[72] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Scong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[73] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[74] Chenbin Zhang, Congjian Luo, Junyu Lu, Ao Liu, Bing Bai, Kun Bai, and Zenglin Xu. Read, attend, and exclude: Multi-choice reading comprehension by mimicking human reasoning process. In SIGIR, 2020.
[75] Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. Demograph- ies should not be the reason of toxicity: Mitigating discrimination text classifications with instance weighting. In ACL, 2020.
[76] Hongjee Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In ECCV, 2020.
[77] Jinzhi Zhang, Mengqi Ji, Guangyu Wang, Zhiwei Xue, Shengjin Wang, and Lu Fang. SurRF: Unsupervised multi-view stereopsis by learning surface radiance field. IEEE Trans. on PAMI, 2022.
[78] Lily Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures in out-of- distribution detection with deep generative models. In ICML, 2021.
[79] Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen. Deep stable learning for out-of-distribution generalization. CVPR, 2021.
[80] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Trans. on PAMI, 2017.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results? [Ys]] See Supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] See Section and the code URL in abstract.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 5 and supplementary file.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See the code URL in abstract.
(c) Did you include any assets either in the supplemental material URL? [Yes]
See the code URL in abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] See the code URL in abstract.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "ybbKnwzsOxZ", "reviewer": "Reviewer_LynZ", "summary": "This paper proposes a novel method to address OOD detection by independent component analysis (ICA) techniques. This novel method named Conditional-i builds upon the probabilistic formulation, and applies the Hilbert-Schmidt Independence Criteria that offers a convenient solution for optimizing variable dependencies.\n\nBasic Idea:\nFor different ID class-conditional distributions, the paper uses MMD to estimate the dependence between ID for c-class and OOD. Main formulas are in Eqs. 3 and 4. \n\nTheorem 1 explains why the independence assumption is beneficial for OOD detection.\n\nExperiments have shown Conditional-i is better than some classical methods. \n", "questions": "(Major) 1. Poor writing. Many notations are not standard. I spend many times to understand your notations, like lines 127-128. What is p_{theta}? it is a mixture of ID and OOD distributions? You should revise your notations to be more readable.\n\n(Major) 2. In Eq 11, do you assume that the number of OOD data is N, which is same with the number of c-class ID data? \n\n(Major) 3. If we have not any OOD data, how you train your model? It is really difficult to find where you have described it. Could you tell me which line you have introduced it?\n\n(Major) 4. Please compare the SOTA method Gradenorm (On the importance of gradients for detecting distributional\nshifts in the wild.)\n\n(Minor) 5. The idea of  HOOD is similar with this paper. You use an extremely similar idea to submit two papers?\n\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Experiments show Conditional-i is better than some classical methods. \n2. The idea that uses independent component analysis (ICA) techniques seems to be novel.\n\n\nWeakness:\n(Major) 1. Poor writing. Many notations are not standard. I spend many times to understand your notations, like lines 127-128. What is p_{theta}? it is a mixture of ID and OOD distributions? You should revise your notations to be more readable.\n\n(Major) 2. In Eq 11, do you assume that the number of OOD data is N, which is same with the number of c-class ID data? \n\n(Major) 3. If we have not any OOD data, how you train your model? It is really difficult to find where you have described it. Could you tell me which line you have introduced it?\n\n(Major) 4. Please compare the SOTA method Gradenorm (On the importance of gradients for detecting distributional\nshifts in the wild.)\n\n(Minor) 5. The idea of  HOOD is similar with this paper. You use an extremely similar idea to submit two papers?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VIXnjZUjduw", "reviewer": "Reviewer_FRYP", "summary": "This paper proposes a new out-of-distribution (OOD) detection method that makes use of known OOD data. Given a classifier with an encoder that outputs features, and a linear classification layer, the idea is to decorrelate the features of in-distribution and OOD samples in a class-wise manner. In order to achieve this, the authors use the Hilbert-Schmidt Independence Criterion (HSIC) to measure the independence between in-distribution (of a given class) and OOD features. A classifier is then trained on the usual classification loss plus the independence objective. However, since computing the HSIC metric involves computing the features of $N$ in-distribution samples in every class, which can get expensive with large $N$, the authors use a buffer to reuse old features and save compute. The method, which is called Conditional-i, is evaluated on OOD detection benchmarks in image classification and NLP, and achieves better performance than other baseline methods.", "questions": "- As mentioned previously, how were the values of $\\lambda$ and $\\tau$ decided? Was a separate validation OOD dataset used?\n- How reasonable are the assumptions for Theorem 1, especially the ones involving $g$’s?\n- The authors mention in lines 267-268 that “This hopefully may better generalize the training to unseen test OODs as long as the generative process of the test OOD is genuinely different”. What does a different generative process mean, and how can we define it? I’m not sure I believe this statement since Theorem 1 doesn’t guarantee anything about the unknown OOD case.\n- In the experiments, have the authors tried using just the test scoring method from Equation 9 on a classifier trained normally with cross entropy?\n- It doesn’t seem fair comparing Conditional-i-generative with methods that don’t use any auxiliary data (even if it’s synthetic). Have the authors tried outlier exposure with the generated OOD data?\n- Have the authors removed overlapping images between train in-distribution, train OOD and test OOD? For example, the tiny images dataset contain CIFAR-10 and CIFAR-100.\n- What $\\hat{C}$, $N$ values were used for each experiment?\n- What were the test classification accuracies achieved by Conditional-i on the in-distribution dataset?\n\nOther comments:\n- Line 204: “matrics” -> “metrics” \n- Line 218: Typo in $p_M(\\mathbf{z}) = \\sum_{c=1}^C p_\\theta(\\mathbf{z}^{(\\xi)},\\mathbf{q}|\\xi=c)p(\\xi=c)$ -> $p_M(\\mathbf{z}) = \\sum_{c=1}^C p_\\theta(\\mathbf{z}^{(\\xi)}|\\xi=c)p(\\xi=c)$\n- Table 1 and 2 caption: “trainnig” -> “training”\n- It would be nice to include the setup that was used to compute the values in Table 4, 5, and Figure 1.\n", "limitations": "The authors used the Tiny Images dataset, which was withdrawn for ethical reasons. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method is very similar to HOOD, as the authors mentioned, however, the differences (class-wise computation of HSIC, and usage of memory bank), are made clear, and there is evidence of improvement in the experiments.\n- The proposed method seems to consistently perform better than other OOD detection methods.\n\nWeaknesses:\n- There is no guarantee that decorrelating the features of in-distribution and known OOD data will lead to improved feature distinction between in-distribution and unknown OOD data.\n- There is no mention in the paper how hyperparameters ($\\lambda$ and $\\tau$) were tuned for each of the methods. However, it seems from the ablation experiments that the used values conveniently match up with the best performance on the test OOD datasets. If it is true that the hyperparameters were tuned on the test set, the results presented in the paper aren’t trustworthy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HrQ318OFFM8", "reviewer": "Reviewer_1WG2", "summary": "The paper proposes a novel training method for improving the out-of-distribution (OOD) detection performance of a classification model. It is motivated by the hypothesis that the mutual information (dependency) between OOD data and in-distribution (ID) data should be small. Specifically, the proposed method relies on the concept of class conditional independence between ID features and OOD features and utilities maximum mean discrepancy (MMD) to enforce this condition, leading to a composite loss function and training strategy. The method evolves into two variants where one requires unlabeled OOD data and the other uses generated OOD data through strong data augmentation. ", "questions": "1. Could the authors comment on the accuracy of this method? \n2. Could the authors comment on the sensitivity of the method using different OOD data during training or different augmentation strategies? \n", "limitations": "The authors addressed potential negative impact in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength \n1. The paper is very written, especially on addressing the similarity to a contemporary work and descriptions of small implementation details that are important for the success of this method in practice. \n2. The method is thetically motivated and empirically verified on large datasets including both CV and NLP tasks. \n3. The paper includes ablation study on various aspects of the algorithm such as the number of classes for enforcing independence during training and the hyperparameter for adjusting the contribution of the loss component for enforcing independence.\n\nWeakness: \n1. As with all methods requiring OOD data during training, the proposed method necessarily entails overhead on storing training-data, sampling/generating OOD data and additional computation such as the Gaussian Kernel.\n2. The performance against a specific type of OOD data of the proposed method might be highly dependent on the type of OOD exposure or augmentation used during training. \n3. It’s not clear how the proposed composite loss function would affect the discriminativeness (accuracy) of the model on in-distribution data. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fuwn1yD8Pl3", "reviewer": "Reviewer_62WK", "summary": "This paper proposes a conditional independence model called Conditional-I for OOD detection. The authors introduce an integrated loss function to reduce the dependence between inliers features of each class and outlier features, so as to distinguish the OOD samples by a relevant independence measure during testing. The authors provide theoretical guarantees of the conditional independence hypothesis. The experiments demonstrate the better performance of the proposed method on CV and NLP tasks in the cases that training OOD data are available or not.\n", "questions": "1.In Section 4.2, the authors propose that Theorem 1 explains the independence assumption is beneficial for OOD detection.  Is there any clue to prove that conditional independence within a class is more effective than restricting independence between inliers and outliers? Are there any empirical experiments to support this idea?\n\n2.The impact of using different augmentation methods on the result should be explored in the ablation. In Image OOD detection tasks, why use two different methods of generating fake outlier samples? Why not generate them in a uniform way?\n\n3.How about the HOOD results on the unseen OOD training data setting?\n\n4.How much computational cost can be saved by using the memory bank architecture?\n", "limitations": "The authors seemed not to  discuss the limitations of the proposed model.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe work relies on an explicit independence hypothesis, i.e., the inliers  extract only little predictive information from outliers during training. The authors extend the assumption to the conditional independence which encourages each IND class to produce independent features from outliers. Moreover, theoretical arguments and supporting analyses are provided for this independence assumption. The theoretical derivations basically follow the work from [2] and [38], which seem to be correct and convincing to me.\n\nWeaknesses:\n\n1.The innovation of the article seems limited to me,  mainly since the work shares the same perspective as [2]. Both the models build upon the probabilistic formulation and applies the Hilbert-Schmidt Independence Criteria (HSIC).  It may be good to clarify a bit more on how novel the paper is compared from [2].\n\n2.There is a lack of qualitative experiments to demonstrate the validity of the conditional independence model. \na)It is better to provide some  illustrative experimental results to demonstrate that minimising HSICcond-i could indeed perform better than minimising HSIC_HOOD. Possibly, one toy dataset can be used to demonstrate the separability of inlier features and outlier features.\n\nb)The authors propose a new test metric, however, lacking the correctness test and comparative experiments with other metrices. \nIt may be better to provide some visualization results or schematic diagram, which could  make  readers easier to understand.\n\n3.Current experimental results seem not very convincing to me. Some critical comparative results are missing. \na)Under the setting of unseen OOD training data, DIN [34], Mahalanobis distance [31], Energy [36], their original papers did not use fake/augmented OOD training data. These settings need to be clarified in the paper. Moreover, the impact of using different augmentation methods on the result could  be explored in the ablation.\n\nb)In CIFAR-100, the experimental setup appears to be consistent with that of HOOD [2]. However, in Table 1 (unseen OOD training data), the HOOD’s results are missing.  In [2], the results of HOOD are superior to those of Conditional-I's. \n\nc)In Table 2 (unseen OOD training data), the HOOD’s results are also missing. \n\nd)There are missing both Conditional-i-generative and HOOD results for the NLP OOD detection tasks.\nAs missing the results of the most relevant methods,  the present experiments could not  convince me of  the validity of the improvements.\n\n4.The memory bank architecture is one contribution, but the authors do not provide quantitative results of introducing the memory bank architecture.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept