A Variational Information Theoretic Approach to Out-of-Distribution Detection
Authors: Sudeepta Mondal
Zhuolin Jiang
Ganesh Sundaramoorthi
Venue: ICML 2025 poster (2025)
Keywords: out-of-distribution detection
information theory
variational calculus
Primary area: general_machine_learning
TL;DR: A theory and new explainable framework for Out-of-Distribution Detection.

Abstract: We present a theory for the construction of out-of-distribution (OOD) detection features for neural networks. We introduce random features for OOD through a novel information-theoretic loss functional consisting of two terms, the first based on the KL divergence separates resulting in-distribution (ID) and OOD feature distributions and the second term is the Information Bottleneck, which favors compressed features that retain the OOD information. We formulate a variational procedure to optimize the loss and obtain OOD features. Based on assumptions on OOD distributions, one can recover properties of existing OOD features, i.e., shaping functions. Furthermore, we show that our theory can predict a new shaping function that out-performs existing ones on OOD benchmarks. Our theory provides a general framework for constructing a variety of new features with clear explainability.

A Variational Information Theoretic Approach to Out-of-Distriuution Detection
Sudeepta Mondal Zhuolin Jiang'
Ganesh Sundaramoorthi
Abstract
2024; Xu al., 2023; Zhang al., 2024) has focused on identifying descriptors of the data that can distinguish be- tween OOD and ID data. In particular, feature-shaping functions have been shown to be promising. In feature shaping, features, usually from the penultimate layer of a pre-trained network, are input to a shaping function and then used to score incoming data. Examples of these ap- proaches include ReAct (Sun et al., 2021), FS-OPT (Zhao et al., 2024), VRA (Xu al., 2023), ASH (Djurisic et al., 2022). While these approaches have provided advancements to SoA on several benchmarks, most of these are empirically driven rule-based methods, and may not generalize well over new unseen datasets (Zhao et al., 2024). is thus benefi- cial to understand under what conditions that these methods will work so that we may understand when to employ one method over another. One way to do this is to develop theory where one can derive features for OOD detection (henceforth called OOD features for brevity) as function of underlying assumptions (e.g., statistical distributions). This could potentially offer a way to critically examine existing methods. It could also potentially enable the systematic development of new features that may generalize better.
We present theory for the construction of out- of-distribution (OOD) detection features for neu- ral networks. We introduce random features for OOD through a novel thforoiiion-theoreiicc loss functional consisting of two terms, the first based on the KL divergence separates resulting in-distribution (ID) and OOD feature distributions and the second term is the Information Bottleneck, which favors compressed features that retain the OOD information. We formulate a variational procedure to optimize the loss and obtain OOD features. Based on assumptions on OOD distribu- tions, one can recover properties of existing OOD features, i.e., shaping functions. Furthermore, we show that our theory can predict new shaping function that out-performs existing ones OOD benchmarks. Our theory provides general frame- work for constructing a variety of new features with clear explainability.
1. Introduction
Towards this goal, we develop a theory to formulate OOD features based on underlying statistical distributions of ID) and OOD distributions. We develop a novel loss functional, based on information theory, defined on the set of OOD fea- tures whose optimization yields OOD features as a function of the underlying statistical distributions. Unlike current approaches, our OOD features are random and thus fol- low statistical distribution. The mean value models the deterministic shaping features in the literature.
Machine learning (ML) systems are typically designed un- der the assumption that the training and test sets are sampled from the same statistical distribution. However, this often does not hold in practice. For example, during deployment, test data may include previously unseen classes. In such cases, the ML system may produce incorrect results with high confidence (DeVriee & Taylor, 2018). Therefore, it is crucial to develop methods that enable ML systems to detect out-of-distribution (OOD) data. Detecting OOD data allows users to be alerted of potentially unreliable predictions and enables the system to adapt accordingly. OOD detection has gained considerable attention recently (Yang et al., 2022).
Our loss aims to determine the OOD feature that maxi- mally separates resulting ID) and OOD feature distributions through the Hallback-Leiber (KL) divergence. As separat- ing distributions by itself is ill- posed, we propose novel use of the Information Bottleneck (IB) (Tishby et al., 2000) as regularization. In use, IB seeks compressed features that preserve the information the data has about OOD, aiming for feature representation that contains only the information necessary for OOD detection. As this loss functional is de- fined on probability measures (representing the distribution of the OOD feature), is an infinite dimensional optimiza- tion problem, and thus we use the calculus of variations (Troutman, 2012) to derive the optimization procedure. Our
Recent state-of tet-the-nn art (SoA) (Sun et al., 2021: Djurisic et al., 2022; Ahn al., 2023; Sun & Li, 2022; Zhao al.,
'RTX Technology Research Center (RTRC), East Hart- ford, CT 06118. Correspondence to: Ganesh Sundaramoorthi cuansh.sundarrmoorttiennr
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
theory offers an explanation of several techniques employed in SoA rule- based approaches, and suggests new shaping function that out-performs other shaping functions in SoA.
softmax probability. (Ren al., 2019) proposes to use the likelihood ratio, which has been proposed over likelihoods, which do not work well (Kirichenko et al., 2020). (Lee et al., 2018) leveraged Mahalnobis distance to compute the distance between features and classes. KNN (Sun et al., 2022) uses a non paraaaaramricc approach. Energy methods (Liu et al., 2021) present an alternative to softmax scores by employing the Helmholtz free energy. Energy scoring has been adopted by several OOD feature-shaping approaches: feature-shaping is the focus of our work.
There have been recent theories for OOD detection (Zhao et al., 2024; Xu et al., 2023). These works have intro- duced the novel idea of formulating OOD features through a loss function rather than empirically driven rule-based approaches of the past, and motivates our work. In contrast to the aforementioned works, our theory employs novel informatiionthhoreet loss function, which offers several ad- vantages. Our theory shows how different assumptions on the OOD distribution lead to different OOD feature shaping approaches. Our theory is able to more accurately offer an explanation for properties of several SoA rule-based approaches as being from different underlying OOD distri- butions and different regularization (see next section for more detailed discussion).
Feature-shaping approaches to OOD detection: Several methods perform OOD detection by computing features of the output of layers of the neural network (Sun et al., 2021; Kong & Li, 2023; Djurisic et al., 2022; Fort et al., 2021b; Zhao et al., 2024) before being input to score, In ReAct (Sun et al., 2021), the penultimate layer outputs are processed element-wise by clipping large values. It is empirically noted that OOD data results in large spikes in activations, which are clipped to better separate the ID and OOD distributions. BFAct (Kong & Li, 2023) uses the Butterworth filter to smoothly approximate the clipping. ASH computes features by sparsifying intermediate outputs of the network by flooring small values to zero and passing larger values with possible scaling. DICE (Sun & Li, 2022) is another approach to sparsification. Different than element- wise approaches, ASH then does vector processing of the shaped feature before input to a score. VRA (Xu et al., 2023) and (Zhang et al.) derive element-wise shaping functions by an optimization approach.
In summary, our contributions are as follows: 1. We intro- duce a novel theory and framework for deriving OOD fea- tures from neural networks. This involves the formulation of OOD features as a variational problem that formulates OOD features as random features through a novel loss func- tional that contains two terms, one that maximizes the KL divergence between the random feature under ID and OOD distributions and another term, the Information Bottleneck, which extracts the information from the data that is relevant for OOD detection. 2. We develop the techniques to op- timize the loss functional using the calculus of variations, and specificallyy derive a computationally feasible algorithm in the one-dimensional data case. 3. Using our framework, we show how the OOD shaping functions change based on various data distributions. We relate the mean value of our OOD features to existing OOD shaping functions. 4. We introduce a novel piece-wise linear OOD feature shaping function predicted through our theory, and show that it leads to state-of-the- art results on OOD benchmarks.
Optimization-basee approaches for feature shaping: (Xu et al., 2023) formulates a loss function for deterministic OOD features that aims to separate the means of ID and OOD feature distributions, and regularization is added to keep the OOD feature near the identity through the L2 norm. (Zhao et al., 2024) analyzes a similar loss function but with point-wise rather than L2 regularization. They further offer simplifications remove the reliance the OOD distri- bution. These works have introduced the novel idea of formulating OOD features through a loss function. Our ap- proach offers several advantages. Over (Zhao et al., 2024), we present a framework in which we can study the OOD fea- ture as a function of the underlying OOD distribution. This shows the implicit assumptions in several existing methods. In contrast, (Zhao et al., 2024) aims to remove dependence the OOD distribution. Our results show that feature shaping can vary as a function of the underlying OOD distri- bution. Over (Zhao et al., 2024; Xu et al., 2023), our theory offers an explanation of qualitative properties of existing SoA methods. For instance, clipping of large values in OOD features (of ReAct (Sun et al., 2021)) is associated with a higher Information Bottleneck (IB) regularization which is needed for noisier OOD datasets. Negative slope at large values in (Zhao et al., 2024; Xu et al., 2023) is associated
1.1. Related Work
We briefly review related work; the reader is referred to (Yang et al.. 2022) for a survey. Post-hoc approaches of OOD detection, which are applied to pre-trained models without additional training, have focused on constructing scoring functions to differentiate OOD from in-distribution data, leveraging confidence scores (Hendrycks & Gimpel, 20; Zhang& & Xiang, 2023; Liang et al., 2020), energy- based metrics (Liu et al., 2021; Wu et al., 2023; Elflein et al., 2021) and distance-based measures (Lee et al., 2018; Sun et al., 2022). For example, MSP (Hendrycks & Gimpel, 2018a) used the maximum softmax probability as a con- fidence score. ODIN (Liang et al., 2020) improved OOD detection by applying temperature scaling and adding small perturbations to input data before computing the maximum
2

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
with low IB regularization. Also, pruning of small feature values in (Xu et al., 2023; Djurisic et al., 2022) is associated with OOD distributions with heavier tails. See Section 4 for more technical details.
where
p(xx) log p(x) dx, and (2) g(x)
DKLlp |1
p(āly) - p(z1z))(z(1)) dz.
(3)
2. Variational Formulation of OOD Features
Note that we have used that p(ž)2, v) p(2]2) the feature is constructed the same for both ID and OOD data. These equations shows the dependence of the OOD feature distributions on p(ā|z). The KL divergence is a natural choice for separating distributions and a standard information-theoree quantity.
We formulate OOD features as an optimization problem. For the sake of the derivation, we will assume that the probabil- ity distributions of ID and OOD features from the network are given in this section. In practice the ID can be estimated by training data. In Section 4, we will then study the OOD features under various distributions to show how features vary with distribution and offer plausible assumptions made by existing feature shaping approaches. We will also make reasonable assumptions on the OOD distribution to derive new prescriptive OOD features for use in practice.
Unconstrained maximization of KL divergence is ill-posed, and regularization is needed. Also, it is possible to reduce the dimensions of Z to a few dimensions that are maximally separated but remove information necessary to fully char- acterize OOD data. Therefore, we need to ensure that 2 contains all the information relevant to accurately determine OOD data. With these considerations, we aim to compress the dimensions of Z to form a simple/compact feature, but in a way that preserves the OOD information (contained in the variable Y). To achieve this, we adapt the Information Bottleneck (Tishby et al., 2000). In the Information Bottle- neck method, the quantization of a random variable X is considered to form the random variable in such a way to preserve information about a random variable Y, where Y forms a Markov Chain with X. A functional is formulated such that when minimized forms T. This is precisely the functional we would like to determine 2 (where 2 is anal- ogous to T and Z is analogous to our functional, following from (Tishby et al., 2000), is
Current OOD features in the OOD literature are computed by processing features from the neural network through a deterministic function (c.g., clipping). In contrast, we propose to generalize that approach by allowing for random functions. Let Z denote the feature (a random variable) from the network (penultimate intermediate layer feature). We denote by Z the random OOD feature (a random variable) that we seek to determine. The distribution of Z is denoted p(()2). Thus, rather than solving for a deterministic function f(Z), we instead solve for a random feature 2 represented through p(ž(z) as in Information Theory (Cover, 1999). Thus, given a feature z, the OOD feature is Z ~ p()]z). We will primarily be concerned with the mean value of the distribution in this paper to relate to other feature shaping methods. Let X be the random variable indicating the data (e.g., image, text), and Y be the random variable indicating in- (Y = 0) and out-of- (Y = 1) distribution data. Note this forms a Markov Chain Y + X + Z + Z. The Markov Chain property is needed to construct one of the terms of our loss function, discussed next.
X). The second term of
IB [p( = I(z 2) -T(
(4)
where I indicates mutual information, and 8 > 0 is a hy- perparameter. The first term of (4) is the compression term that measures the mutual information between Z and Z; this term is minimized and thus the term favors to be a compressed version of Z. The second term maximizes the mutual information between 2 and Y, and thus favors 2 to retain OOD relevant information.
We propose novel loss functional design the OOD ran- dom feature. This loss functional is defined on p(ā|=). The first term aims to separate the ID and OOD distributions of the random feature Ž. This is natural since we would like to use the OOD feature to separate the data into in or duitr--diittibution. To achieve this separation, we pro- pose to maximize the symmetrized KL-divergence between p(ž)Y = 0) and p(ž)Y= = 1). Note recent work (Zhao et al., 2024) also seeks to separate distributions, however, differ- ently than our approach as only the means of the distribution are separated. Also, note that p(ž = y) is a function of p(ā)z). the variable of optimization, and thus the KL term a function of p())2). This term is defined as follows:
Thus, our combined loss functional is
L(p(E|=)) - -Dx (=x((((((E))) + aIB (p(E)))
(5)
which is minimized to determine the conditional distribution of Ž, p()]2), and >> 0 is hyperparameter. Our goal is to determine the optimal p(ä]z), which can then be used with a score function to determine whether data is OOD or not. Note that we are seeking to optimize over the set of continuous probability distributions, which forms an infinite dimensional optimization problem. To gain intuition into the loss functional above, in particular to see that it forms a well-posed problem and that IB regularization needed, we
DKL(p(E12)) D = 1)
(1 1)
= +]]+
DKIpp = 0) ||al = 1)], (I)
3

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
analyze a simple case with ID Gaussian distributions that result in closed form solution in Appendix A. We verify in the next section that the loss functional, for more complex distributions/feattres, yields well-posed problems and hence result in an optimal solution.
3.2. Gradient of Loss Functional
We will use gradient descent to optimize the loss functional. Since the problem is non-convex, gradient descent is a natu- ral choice. Given the infinite dimensional problem, use the calculus of variations to compute the gradient.
3. Optimization for OOD Features
We perform the computation for the gradient of (5) in Ap- pendix C and summarize the result in the following theorem:
In this section, we discuss the optimization of the loss func- tional (5). The loss functional is defined on continuous probability density functions p(ž]=), where z, 2 are continu- ous. This is an infinite dimensional optimization problem, and to find the optimal feature one can use the calculus of variations to determine the gradient of (Troutman, 2012). Setting the gradient to zero and solving for the probability distribution that satisfies the equation gives the necessary conditions for the optimizer. For our loss, that does not yield a closed form solution and so we instead use the gradient to perform a gradient descent.
Theorem 3.1 (Gradient of Loss). The gradient of DKL (p(ā)2)) (1) with respect to p(ž)2) is given (up to an additive function of 2) by
V PKL = p(z10) [1(z) log (() [[2)]
p(=|1) [1(=) (() + ((2) (())) (8)
where p(zlv) = p(= = v), p(2l)) y = v) and
1(x) = p(=[1)
and (()) = p
(9)
p(=10)'
p(
The gradient of IB(p(2)2) (4) is given by V PB
3.1. Loss Under Element-wise Independence of Feature
p(ā|z) 3 Blog
p(žly)
Σ p(y)p(=l)) (E
log p(z)
(10)
p(3)
Because formulating numerical optimization for general multi-dimensional distributions is difficult, we make some simplifications to gain insights to our theory and approach. Even with these simplifications, we will show that the ap- proach can explain popular approaches in the literature and lead to a new state of the art approach. Our first simplifica- tion (which is similar to element- wise processing assump- tions made in cxisting methods, c.g.. (Sun et al., 2021; Zhao et al., 2024)) is to assume that the conditional feature distri- bution 2)] can be factorized as p(ā|=) = II pl [z), which assumes conditional independence of the components of E and that each component has the same conditional dis- tribution. We also assume that p(zly) = II= p(2; ly), that is, the components of z are independent conditioned y. Under these assumptions, the optimization of the loss functional (5) reduces to the optimization of several opti- mization problems defined on one-dimensional probability distributions from each feature component (see Appendix B for details):
The gradient of the full loss L in (5) is then
Va(ale L = - VV Veaas DEL + a IB.
(11)
To simplify further and study a model that more closely resembles OOD feature shaping functions in the literature, we make the following assumption:
-(E|=)~ ~ (((()),
(12)
where N' indicates Gaussian distribution, ă, E R and H4, Te : IR + I are the mean/standard deviation. We use the sub-script e to denote "conditional" to distinguish from other sigmas used below. We can think of this model as random perturbations of deterministic feature shaping function μa. The OOD's feature mean value for given network feature is puz). The closer de is to zero, the closer the approach is to deterministic feature shaping. Note if the optimization turns out to result in a, 0, then deterministic functions would be optimal. In our numerous simulations, this does not happen and thus random OOD features appear to be more optimal. We now compute the gradients with respect to p and o.:
arg mig L(((/e))))) i [l. p(ā))1))
(6)
where
Theorem 3.2 (Loss Gradient Under Gaussian Random OOD Feature ( (12)). The gradient of the loss (5) under (12) is
[x(((E)(=)))= = = -Dxxb((E)))))ll D ( ((()0))) + aall(2; 2,) ()) (7)
V p(āl)) [(2, 2)
V,,((=)= =
[3 m(=))p(E(=)dd
(13)
0²(())
(()) 2)
[( [())(2))²
Va. a. [(2)) =
1 p p((lz) dž,
de(2)
02(2)
Thus, we next provide an optimization procedure for the loss functionals above, defined on -e-dimensiooal distributions. For simplicity of notation, we now omit the i subscripts.
(14)
where V Vp(Rle I is given in (11).
4

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
3.3. Numerical Optimization of Loss
Algorithm ID Gaussian Random Feature Computation
We implement a gradient descent algorithm using a dis- cretization of the continuum equations above. We choose uniform discretization of the space of 2, i.c., { c R. We represent and de through their samples: 14 = (M(2)) and oe.i = or (z,). We specify formulas for p()) and p(2ly) under the discretization, which will be required in the com- putation of the approximation to the gradient:
Input: IN/OOD Distributions p(zly), a, and learning rate r/ Output: Converged mean (4, std de.s for each
Initialize: μ4, = 24, C,,i const for rs itcrations do
for 2, do
Compute a discretization of 2 in its likely range: Ef (n4 ko Mi + koa,s) where l 2 3
for 3 do
p(ž = Σ p(()2))((211))
Compute (E),)))
Gr _ (5 (i-m))(=l))
Σ Oet
(15)
p(z]0)) [(x)) (()) [(E})]
p(=
[(x log (E)) + +(())=
p(2) E((t)p(Ely).
(16)
Σ p(u)p(z (v) loog p(E)|z))
lloog ((3llV)
a
p(z))
p(25)
Thus, p(lly) is approximated as mixture of Gaussians. The gradient descent is shown in Algorithm which as- sumes ID and OOD distributions and determines the Gaus- sian random feature parameterized through μ and oe.
ve(0.1)
end for Compute V v=((z)= =
(( (M)))))))))
The complexity for this optimization (which is done off- in training) is O(N where N is the number of samples of p(2ly), M is the samples of p()]2) and K is the number of gradient descent iterations. On a single 100 GPU, this
0
Compute V., Vo.)(z))
took less than a minute.
Σ V
(2) - 14)²
p(z))z))
Cc.
4. A Study of OOD Features vs Distribution
end for for do
In this section, we study the resulting OOD features based on various choices of distributions using the algorithm in the previous section, and relate these choices to OOD feature shaping techniques that are present in the literature. Note that while in practice the OOD distribution is unknown, our theory nevertheless suggests the underlying distributional assumptions of existing methods. This is useful to under- stand when these methods will generalize as a function of the type of OOD data. We will also derive a generic OOD shaping function, encompassing properties of several distri- butions, and show that this shaping funetion can lead SoA performance in the next section. Note in practice, have observed that distributions from OOD datasets to exhibit similarities to the distributions studied, see Appendix H. We will further rationale on studying these distributions below.
1/1 + ph -V
Je, 4- Jc,t - VV, L(z)
end for end for
Gaussian OOD: First, we study the case of a Gaussian for the OOD distribution, as its the most common distribution in probabilistic analysis. Let p(=[1) lustration, we choose μl = -0.5, μ = 0.5 and = 0.5 and a = 1.0, 8 = 10. The resulting converged result of the optimization for p and is shown in Figure I (positive part). No feature shaping would mean that μ is the identity map, and a. = O; this solution is plotted in dashed blue. Notice that the optimal mean value is not the identity. The mean indicates that the feature has positive slope for small values of |z| (similar to (Sun et al., 2021)) and negative slope for large values of |=| (similar to (Zhao et al., 2024)). In Appendix E.2, we show that under different distribution parameters, one can get negative values for small |>| as in (Zhao et al., 2024). Interestingly, the optimal standard deviation (=(2) is non-zero, indicating randomness in this
N(nn, For il-
For this study, we will assume the assumptions of Sec- tion 3.3, i.e., that the OOD features are element-wise independent and that the OOD feature is Gaussian, i.c., p(ā)2) ~ (m(=)), (z))). We will further assume that the ID distribution is Gaussian, i.e., p(z(0) (N(p,, σ). We make this assumption for simplicity and that features in network layers can be approximated well with a Gaussian, as evidenced empirically in (Xu et al., 2023). We will study three OOD distributions next: Gaussian, Laplacian and a distribution we propose based on the Inverse Gaussian.
5

---

Inverse Gaussian OOD Distribution: Next, we consider a distribution that may be a distribution that generically holds for OOD data and can be used in the absence of prior information of the OOD distribution. If the ID distribu- tion is Gaussian, we can formulate a distribution that has high probability outside the domain that the ID distribu- tion has high probability. To this end, one can consider a variant of the Inverse Gaussian defined as follows. Let d(=) ž - Hol/a where μoo a are the mean and standard A Variational Information Theoretic Approach to Out-of-Distribation Detection
Meen ( of OGD Teature
Std. Dev. [d:] of (00DFeeturs
deviation of the ID distribution. This is distance the ID distribution. We would like the OOD distribution to have high probability when d(z) is large, and thus we consider p(z[1) ~ [G(((z); fl, X) where IG denotes the inverse Gaussian distribution:
1.5
2.0
5.0
1.8
2.5
1.6
0.0
1.4
-5.0
J
Identity map
1.2
PIG(E; = V 2 exp
1(x 12) 211xx
(17)
-7.5
which is plotted in Appendix D. Note that there is some overlap of this distribution with the ID Gaussian. As noted in Figure 2(c), the Inverse Gaussian distribution results in qualitatively similar OOD feature compared to the Lapla- cian distribution: suppression of small |a] values and clip- ping/flattening of large |2| values and positively sloped linear function for intermediate values of |z]. For a large we have flattening similar to clipping and a smaller results in a negative slope similar to the other distributions.
Figure 1. OOD Gaussian Feature Under Gaussian ID/OOD Distri- butions. Mean (left), standard deviation (right) the feature.
case is beneficial in terms of the loss. In fact, in all of our simulations across distributions and their hyperparameters, we've ve observed non-zero standard deviation.
In Figure 2(a), we show the effects of the Information Bot- tleneck weight a. The impact of 3 on the shape is studied in Appendix E.1. For larger (higher regularization), the mean of the feature becomes flat for |a| large, similar to clipping that is used in popular methods (Sun et al., 2021; Xu et al., 2023).² See Figure 3 for a plot of existing fea- ture shaping methods. Even under the simplifying Gaussian assumptions, we see that the our shaping functions have properties of existing methods.
We summarize key observations. Clipping as done in Re- Act seems to be a universal property across all the OOD distributions for large regularization. In the next section we show that for noiser OOD datasets larger regularization is beneficial, and so the clipping mitigates noise, which is noted in (Sun et al., 2021). Next, the OOD distributions that are heavier tailed result suppression (zeroing out) of low |a| values. This is consistent with the VRA and ASH methods. All distributions yield positively sloped region for intermediate values of |a]. Our results suggest ReAct and FS-OPT may be operating under an implicit Gaussian OOD assumption for high regularization (ReAct) and low regularization (FS-OPT), VRA and ASH seem to implicitly assume heavier tailed OOD distributions.
Laplacian OOD Distribution: Next, we consider the Lapla- cian distribution for the OOD distribution, i.e., p(=[1) = 26 exp (-12- - p1l/ The intuition for choosing this dis- tribution is that it has a heavier tail than the Gaussian, and thus, is better able to model outliers, and it seems reasonable OOD data would be considered outliers. We show the result of the mean of the feature in Figure 2(b). We notice that when |=] is small, the mean OOD feature is zero, which indicates suppression of low values (this is used in VRA (Xu et al., 2023) and ASH (Djurisic et al., 2022)). Note that this is consistent across a values, larger values increases the suppression region. We also see that large values of |z| are being clipped or suppressed (with larger a) approaching a zero slope. The jump discontinuity is also present in VRA and ASH. There also appears to be positively sloped linear function for intermediate values of |z|. similar to VRA.
1
The above mean shaping func- tions (from Gaussian. Laplace and Inverse Gaussian OOD distributions) all approximately fit in a particular piece- wise linear function family as shown in Figure 4, where R1, #2, yo: Va,, V15, mi, m2 are hyperparameters. Therefore, in practice, if the distribution is unknown, one can choose this family of shaping functions that would implicitly as- sume any of the aforementioned three distributions. Be- cause many existing SOA methods implicitly make one of the three distributional assumptions, this family makes more general distributional assumptions than existing SOA, thus potentially offering generalization to more OOD datasets while not being too general so as to lose discriminability. In the experimental section we explore this family of shaping functions, and show we can obtain SoA results.
Piecewise Linear Shaping:
5. Implementation of New OOD Detection
In this section, we provide the implementation details for our new approaches to OOD detection, using the simplifying assumptions presented in Section 3. We provide the details for two cases where the ID/OOD distributions are known
Note our approach does not assume energy scoring like SoA, which changes the scale of the features. However, our approach uncovers and explains similar shapes irrespective of the score.
6

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
Gaussian OOD
Laplace 00D
Inverse Gaussian OOD
10
15.0
12
8-
12.5
10
10.0
8
6-
(12)
7.5
6
4-
5.0
4-
a=0.1
am 3.0
2-
2.5
2-
0-10.0
100.0
0
0.0
Ident ty map
0.0
0.5
1.0
1.5
2.0
2.5
2
4
6
2
(a)
(b)
(c)
Figure 2. The mean of the OOD Gaussian Feature under the Gaussian (left), Laplace (middle) and Inverse Gaussian (right) OOD distributions for varying weights on the Information Bottleneck, For all plots, 8 = 10. For the Gaussian case, p(z(0) ~ WV-0.5, 0.5) and p(=[1) ~ (0.5, 0.5). For the Laplace case, p(=[0) ~ (00, 0.66) and p(=[1) ~ Lap(0, 1). In the Inverse Gaussian case, p(z/0) (0, 0.66) and p(z[1) IG(d(=);3333 15). For visualization purpose, we only show the positive part.
VRA
PS-Opl
ASH
the feature shape fits the piecewise family in the previous section (i.e., the OOD distributions are one of Gaussian, Laplacian or IG). The hyper-parameters for the piecewise family are tuned by e.g., minimizing the false positive rate at true positive rate of 95% (FPR95) metric on the validation set this gives the optimal shaping function je".
AS
3-
ASI
1-
1 3
2
3
2
Figure 3. Plot of existing feature shaping functions from SoA meth- ods: ReAct (Sun et al.. 1021), VRA (Xu et al., 2023). FS- Opt (Zhao et al., 2024), and variants of ASH (Djurisic et al., 2022).
Online Operation: During operation, the network feature z is extracted, and then shaped via the function / (z) (μ² (21), ..... .....,"" (2,)). Subsequently, is input scoring function (e.g., in this paper, energy score (Liu et al., 2021)), which is then thresholded to produce the ID/OOD label.
Yo + Yoo
0 S Zz
f()
V1b.
I 21
m12
Y18
m
Yıb + m, (z 21),
<
6. Experiments
Yib + m2 (Z2 21)= +
Yie
yo
0
Z1
Z2
mz - 2:2),
z> Z2
We validate our theory by comparing our new shaping func- tion to SoA for OOD detection on standard benchmarks.
Figure 4. piece-wise linear family of functions that approxi- mately encompasses the mean value of our OOD feature shaping functions across OOD distributions examined in this paper.
Datasets and Model architectures. We experiment with ResNet-50(He et al., 2016), MobileNet-v2 (Sandler et al., 2018), vision transformers V6T-B-16 and ViT-L- 16 (Dosovitskiy et al., 2021) with ImageNet-Ik 1k (Rus- sakovsky et al., 2015) as ID data, and benchmark on the OOD datasets/methods used (Zhao et al., 2024). For the ImageNet benchmark. we evaluate performance across eight OOD datasets: Species (Hendrycks al., 2022), iNaturalist (Horn al., 2018), SUN (Xiao al., 2010), Places (Zhou et al., 2018), OpenImage-O (Wang et al., 2022), ImageNet- O (Hendrycks et al., 2021). Texture (Cimpoi et al., 2014), and MNIST (Deng, 2012). Moreover, we also experiment with CIFAR 10 and CIFAR 100 as ID data, for which we use a ViT-B-16 (Dosovitskiy et al., 2021) finetuned on CI- 1AR10/100 consistent with (Fort et al., 2021a), and a MLP- Mixer-Nan model trained on CIFAR 10/100 from scratch.
and unknown. In the latter case, we apply the piecewise family of feature shaping derived in the previous section (Figure 4). We assume that validation of ID and OOD data is available (as in existing literature) and the choices given in experiments section. A trained neural net- work is also provided. The network feature vectors and their ID/OOD label for the validation set are {z,, }. Consis- tent with our simplifying assumptions and literature, each component of the network feature z is processed indepen- dently and for this paper, they will be processed by the same shaping function μ.
Off-line-Training Under the case that the forms of the ID and OOD distributions are known, the hyperparameters of the distributions are estimated from the validation set (rasterizing the vector data). Using the fitted distributions, Algorithm is run to compute the optimal μ², o. In the case that the distributions are unknown, we assume that
We evaluate eight OOD datasets: ImagenaeNett (Torralba et al., 2008), SVHN (Netzer et al., 2011), 1), Texture (Cimpoi et al., 2014), Places365 (Zhou et al., 2018), LSUN-Cropped (Yu et al., 2016), LSUN-Resized (Yu et al., 2016), iSUN

---

A Variational Information Theoretic Approach to cut-of-Distribbtiin Detection
(Xu al., 2015), and CIFAR CIFARIO (CIFAR 100 treated OOD for CIFAR 10, and vice-versa).
through intermediate layer pruning and rescaling of features, which are then routed back into the network for further pro- cessing, and thereby is not an element-wise feature shaping approach. Our theory currently does not address this case.
We compare our results against the SoA methods across two categories penultimate layer element-wise feature shaping approaches, which our theory currently applies to, and other methods. Penultimate layer feature shaping ap- proaches involve element-wise feature shaping functions applied directly to the penultimate layer of the model be- fore computing the energy score for OOD detection. Ap- proaches in this category are: Energy (Liu et al., 2021), ReAct (Sun et al., 2021), BFAct (Kong& & Li, 2023), VRA- (Xu et al., 2023) and FS-OPT (Zhao et al., 2024). The second category, which are not directly comparable to our approach because they may not involve feature shaping or additions to feature matching, but are included for reference, are softmax- based confidence scoring (MSP (Hendrycks & Gimpel, 2018b)), input perturbation and temperature scaling (ODIN (Liang et al., 2020)), intermediate-layer shaping and subsequent processing by following layers (ASH-P, ASH- B. ASH-S (Djurisic et al., 2022)), or weight sparsification (DICE (Sun & Li, 2022)).
For the vision transformers ViT-B-16 and ViT-L--1,, our method achieves the lowest FP among all competing meth- ods. providing evidence of generalization across different architectures. Overall, our results demonstrate that our fea- ture shaping is highly competitive with the latest SoA, along with providing theoretical explanation.
The inference cost of our feature shaping method is on the order of microseconds for a 256x256x3 image, using PyTorch on an NVIDIA A A100- 80GB GPU. This is comparable to other piecewise linear shaping approaches such as ReAct and VRA.
Computational Time:
Regularization as a Function of OOD Data. We study how IB regularization should be chosen with respect to prop- erties of OOD data. This is important in practical scenarios. In particular, we conduct an experiment to suggest higher IB regularization is beneficial for "noisier" OOD datasets.
As in ReAct (Sun et al., 2021), for ImageNet- Ik bench- marks use a validation set comprising the validation split of ImageNet-1K 1k as II) data, and Gaussian noise im- ages as OOD data, generated by sampling from NN0,, 1) for each pixel location, to tune the hyperparameters of our piecewise linear activation shaping function. For CIFAR 10/100 benchmarks, following ODIN (Liang et al., 2020), we employ a random subset of the iSUN dataset (Xu et al., 2015) as validation OOD data for our hyperparameter tun- ing, As ID validation data for CIFAR10/100 we use the test splits of the corresponding datasets. The hyperparam- eters are optimized using Bayesian optimization (Frazier, 2018), by minimizing the FPR95 metric on the validation set. Resulting hyperparameters are reported in Appendix G.
To this end, we conduct series of controlled experiments using ResNet-50 trained on the ImageNet- Ik dataset. We aim to determine the structure of optimal feature-shaping functions as a function of noise. We apply additive Gaus- sian noise (00 a) to the ImageNet- 1 validation set and consider them as OOD data. The standard deviation a used are {25, 50, 100, 150, 255] to create 5 OOD datasets. Visu- alizations of this data and activation patterns are shown in Appendix F. We observe that this data closely resembles the high variance of activation patterns in OOD datasets as noted in (Sun et al., 2021), and so our noisy data serves to mimic OOD data with varying noise levels. By examining how the learned features adapt under different noise levels, we gain insights into the relationship between the OOD data and the IB regularization term for optimal shaping.
Metrics. We utilize two standard evaluation metrics, follow- ing (Sun al., 2021; Zhao et al., 2024): FPR95 the false positive rate when the true positive rate is 95% (abbreviated as FP), and the area under the ROC curve (AU).
In Figure 5, we plot the IB term of the obtained optimal shap- ing function optimized over hyperparameters at each noise level. Note we have used the Laplacian OOD distribution to estimate the IB term (Inverse Gaussian also results in similar results). It is seen that higher noise levels result in optimal shaping functions with lower IB values, which correspond to higher degree of regularization of the IB term in our loss functional. Thus, noisier OOD datasets require higher IB regularization for best OOD detection performance.
Results. The results on the ImageNet- Ik benchmarks (Table 1) and the CIFAR 10/100 benchmarks (Table 2) demonstrate that our approach achieves state-of-the-art per- formance among comparable feature-shaping methods in the previously mentioned category of methods. Specifically, when compared to pointwise feature-shaping techniques such as ReAct, BFAct, VRA-P, and FS-OPT, our method consistently outperforms these approaches, yielding the best overall results in this category.
7. Conclusion and Future Work
We have presented a novel theory for OOD feature construc- tion. OOD features were computed as the optimizer of a loss functional consisting of term maximizing the KL distance between resulting features under ID and OOD distributions
While ASH variants marginally outperform our method in some cases, it is important note that ASH employs fun- damentally different approach. ASH modifies activations
8

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
ResNer50
MobileNeiV2
ViT-B-16
ViT-L-166
Method
FP4
AUT 1
FP4
AUT
FP
AU 1
FP 1
AUT
Energy
60.97
81.01
61.40
82.83
73.96
67.65
74.89
70.11
Feature Shaping
Element-wrie
ReAct
42.29
86.54
54.19
85.37
73.82
76.86
76.16
81.07
BFAct
43.87
86.01
52.87
85.78
77.64
80.16
84.02
81.12
VRA-P
37.97
88.58
49.98
86.83
98.39
35.66
99.58
16.70
FS-OPT
39.75
88.56
51.77
86.62
69.52
82.66
72.17
83.23
Ours
35.82
89.36
46.97
87.49
67.73
81.06
66.67
83.92
Other methods
MSP
69.30
76.26
72.58
77.41
69.84
77.40
70.59
78.40
ODIN
61.56
80.92
62,91
82.64
69.25
72.60
70.35
74.51
ASH-P
55.30
83.00
59,41
83.84
99.36
21.17
99.18
20.27
ASII-B
35.97
88.62
43.59
88.28
94.87
46.68
93.72
38.95
ASH-S
34.70
90.25
43.84
88.24
99,48
18.52
99.42
18.61
DICE
45.32
83.64
49.33
84.63
89.68
71.32
72.38
67.08
Table 1. ImageNet-I Ik Benchmarking Results. Lower FP is better, higher AU is better. Bold values highlight the best results among clement-wise feature shaping methods. Underlined values indicate best results among other methods.
and regularization based on the Information Bottleneck. We have related the optimal features to several element- wise OOD shaping functions that are used in existing practice, offering theoretical explanation and suggesting underlying distributional assumptions made in these often empirically motivated approaches. Our theory was shown to lead to shaping function that perforrrorms existing shaping functions on benchmark datasets.
CIFARIO
CIFARI00
DenscNer101
MLP-N
DenseNet101
MLP-N
Method
FPL
AUf
FPL
AUT
FPI
AU1
FPJ.
AUT
Element wineen-wwee wise
Feature Shaping
Energy
31.72
93.51
63.95
82.84
70,80
80.22
79.90
75.30
ReAct
82.00
76.46
64.34
81.85
77.00
78.30
79.99
75.87
BFact
84.40
74.39
78.02
72.68
80.27
73.36
80.05 76.58
VRA-P
92.77
65.95
67.75
87.19
There are several areas for future investigation. Firstly, we have developed the concept of random features, whose mean value models OOD shaping functions, but only exploited the mean value algorithmically. Future work will aim to exploit the OOD feature distribution. Secondly, our theory far only explains the OOD feature and not the score function. We wish to incorporate scores into our theory. Finally, we wish to explore more general vector shaping functions.
FS-OPT
28.90
94.12
83.87
71.83
65.20
82.39
81.35 74.67
Ours
95.26 24.08
62.67
81.67
64.15
82.00
78.33
73.53
Other methods
67.01
82.86
83.97
73.07
ODIN
32.84
91.94
69.20
69.53
62.03
82.57
78.71 66.47
MLS
31.93
93.51
64.50
82.97
70.71
80.18
82.41 74.52
ASH.P
29.39
93.98
84.39
66.93
68.21
81.11
86.73
65.27
ASH-B
28.21
94.27
93.93
53.00
57
83.80
93.63 $7.20
ASIES
23.93
94.41
82.57
68.02
$2.41
84.65
89.39
59.63
DICE
93.27
96.64
52.17
59.56
95.78 44.48
Table Performance comparison on CIFARI0/100 datasets. Lower FP is better, higher AU is better. Bold values highlight the best results among clement-wise feature shaping methods. Un- derlined values indicate best results among other methods.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Ahn, Y. H., Park, G.-M,, and Kim, S. T. Line: Out-of- distribution detection by leveraging important neurons, 2023.
0.08
IB Value
0.06
0.04
0.02
Cimpoi, M.. Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014. doi: 10. 10../C/P..001.....0.
0.00
25 50
100
150
255
Noise Level (a)
Figure 5. IB for the optimal hyperparameter optimized shaping function as a function of the noise level of OOD data. Lower IB corresponds a more regularized shaping function.
Cover, T. M. Elements of information theory. John Wiley & Sons, 1999.
Deng, L. The mnist database of handwritten digit images
9

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-142, 2012. doi: 10.1009/MSP:221.2221111
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song. D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 17262-15271, June 2021.
DeVries, T. and Taylor, G. W. Learning confidence for out-of-distributtin detection in neural networks, 2018.
Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. Scaling out-of-distributie detection for real-world settings, 2022, URL https: arxiv. www.ii.irgii.. .rgg abs/ 1911. 11111.1113 11132.
Djurisic, A., Bozanic, N., Ashok, A., and Liu, R. Extremely detec-
simple activation shaping for tion. 2022. URL https: /tt///iivvvv//////////aaa////aav wwtp./.ww.//bb/////0000 09858.
Horn, G. V., Aodha, O. M., Song, Y., Cui, Y., Sun, C., Shep- ard, A., Adam, H., Perona, P., and Belongie, S. The inat- uralist species classification and detection dataset, 2018. URL https:ttpp........tt http:s://///...ro//a/aa abs 1100170000000 06642.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition scale, 2021. URL https: // //arxiv..rg/ abaa 201002000.00 11929.
Kirichenko, P., Izmailov, P., and Wilson, A. G. Why normal- izing flows fail to detect
data. Ad- vances in neural information processing systems, 33: 20578-20589 2020.
Elflein, S., Charpentier, B., Zügner, D., and Günnemann, S. On -ut-of-distribution detection with energy-based models. arXiv preprint arXiv:2107.08785, 2021.
Kong, H. and Li, H. Bfact: cut-of-distributtin detection with butterworth filter rectified activations. In Sun, F., Cangelosi, A., Zhang, J., Yu, Y., Liu, H., and Fang, B. (eds.). Cognitive Systems and Information Processing, pp. 115-129, Singapore, 2023. Springer Nature Singapore.
Fort, S., Ren, J., and Lakshminarayanan, B. Ex- ploring the limits of out-of-distribution detection. In Ranzato, M., Beygelzimer. A., Dauphin, Y., Liang, P. and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 7068-7081. Curran Associates, Inc.,
Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting dutto--distriuttin samples and adversarial attacks. Advances neural information pro- cessing systems, 31, 2018.
2021a. URL https://proceedinge
Liang, S., Li, Y., and Srikant, R. Enhancing the relia- bility of out-ofddistribution image detection in neural networks, 2020. URL https: http://wwxvv..././///// 1706. 02690.
co/paper_ //apppe_fflle//pppr///0////ee 99014338001774022335ccffffaaa333 pdf.
Fort, S., Ren, J., and Lakshminarayanan, B. Exploring the
Liu, W., Wang, X., Owens, J. D., and Li, Y. Energy-based -fist--iistiiuution detection, 2021. URL https:// arxiv. arriv.org/abs/22.
limits of -itrr--distribution detection. Advances in neural information processing systems, 34:7068-70$3 2021b.
03759.
Frazier, P. tutorial on bayesian optimnnatiiioninnnni 2018. URL ht pettps:/aaree http:iiiworr/ab////eeebee
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsuper- vised Feature Learning 2011, 2011. URL http:
abs alrrg//bs//88/.//. 02811.
He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), Computer Vision ECCV 2016, pp. 630-645 645, Cham, 2016. Springer International Publishing.
//ufldl. sww.sord....... wwwp:/ussssshbbbbbrr...... nips2011 htwp2001_hhusseumberrs.oo pdf.
Ren, J., Liu, P. J., Fertig. E., Snock. J., Poplin, R., Depristo, M., Dillon, J., and Lakshminarayanan, B. Likelihood ra- tios for information processing systems, 32, 2019.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018a. URL https: org/ abs/ 1610. 02136.
detection. Advances in neural
Russakovsky, O., Deng. J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge, 2015. URL https:/ a wrxiv.org/abs//
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018b. URL https: // //arxvvv../////////////////aaavr org/ abs/ 1610. 02136.
0575.
10

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 JEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4510-4520, 2018. doi: 10.1109/0
Yu, F,, Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, Lsun: Construction of large-scale image dataset using deep learning with humans in the loop, 2016. URL https: / aaxii.iiiiiv//..a orq/ abs /10..150...0 03365.
Zhang, Y., Lu, J., Peng, B., Fang, Z., and Cheung, Y.-m. Learning shape in-distribution feature space for out- of-distribution detection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Sun, and Li, Y. Dice: Leveraging sparsification for out- of-distribution detection. In European Conference Computer Vision, 2022.
Sun, Y., Guo, C., and Li, React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144-157, 2021.
Zhang, Y., Lu, J., Peng, B., Fang, Z., and ming Cheung, Y. Learning shape in-distribution feature space for out- of-distribution detection. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https:://///.mc bettps::/openre///// http///ppnneeeeeeeee/ee/eeeoeeep net forum? id=1Du 3MMP 5YN.
Sun, Y., Ming, Y., Zhu, X., and Li, Y. Out-of-distribution de- tection with deep nearest neighbors. In International Con- ference on Machine Learning, pp. 20827- 20840. PMLR, 2022.
Zhang. Z. and Xiang, X. Decoupling maxlogit for out-of- distribution detection. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. $388-3397, 2023. doi: 10.1109/VVPS2222920000 00330.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004057, 2000.
Torralba, A., Fergus, R., and Freeman, W. T. 80 million tiny images: large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11):195--1970, 2008. doi: 10.1 00.1099/PAM1.2000. 128.
Zhao, Q. Xu, M., Gupta, K., Asthana, A., Zheng, L., and Gould, S. Towards optimal feature-shaping meth- ods for -ut-of-diitri detection. arXiv preprint arXiv:2402.0086. 2024.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6))1452-1464, 2018. doi: 10. 109/TPAMI 2017.223009.
Troutman, J. L. Variational calculus and optimal control: optimization with elementary convexity. Springer Science & Business Media, 2012.
Wang, H., Li, Z., Feng. L., and Zhang, W. Vim: Out- of-distribution with virtual-logit matching, 2022. URL https://arxte
abs /2003.223.. 10807.
Wu, Q., Chen, Y., Yang, C., and Yan, J. Energy-based out- of-distribution detection for graph neural networks. arXiv preprint arXiv:2302.029) 2023.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485- 3492, 2010. doi: 00.1109/CVRR.20005553999
Xu, M., Lian, Z., Liu, B., and Tao, J. Vra: Variational rectified activation for out-of-distribution detection, 2023. htth:t::: //t////aaaa aaaxiviiii..///ai///aaaaaaaaaava org/ /bts/230.... 11716.
Xu, P., Ehinger, K. A., Zhang, Y., Finkelstein, A., Kulkarni, S. R., and Xiao, J. Turkergaze: Crowdsourcing saliency with webcam based eye tracking, 2015. URL https:
//arxiv. orgo abs 504.0044400 0..44..... 06755.
Yang, J., Zhou, K., Li. Y., and Liu, Z. Generalized out-of- distribution detection: A survey, 2022.

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
A. Analysis of Loss Function in 1D Gaussian Case
We analytically study the optimization of our loss functional the case where the ID/OOD distributions of z are Gaussian, ie.. p(zly) ~ N Ty)s y E {0, 1}. and the feature is Gaussian with mean being a linear function and the standard deviation constant, i.e., p(2]z) ~ NW
ae) where W, 6 E. The W, are parameters that are to be optimized, which specify the OOD feature. This is relaxation of deterministic shaping function being linear function. The loss function shape and various components of the loss are shown in Figure 6. The plot shows the loss terms versus W; as will be shown below the loss does not depend on b. It shows that the separation between feature distributions (KL term) increases as [W] + 20. On the otherhand, the information bottleneck term increases as |W + O. Thus, these terms compete with each other and the optimal solution is well well-defined at a finite W > This simple example suggests that the loss functional is well defined (i.e., has finite optimum). Notice that without the IB term, there is no optimal value of W.
Next, we derive the components of the loss in analytic form. Based on the assumptions specified in the previous paragraph, the formulas for the assumed probabilities are:
1
1
p(zly)
exp
22y2
(18)
V2
20
1
(ž - W - 6)
p(āl=)=
exp
(19)
V2TO.
20
Under these assumptions, we can calculate the feature distributions:
1
(-
1
(2 - W - 6)2 -b)2) ))²)
((
exp
2(02 + W2 (-
(20)
2 + 72
and note that the joint distribution is
(
W WWy (Wbg+bb )
(
W
- 02 Wo²
Wo d²
p(ž 2ly) ~ Nu Ee,s,, Me =
En.5=
(21)
Bly
Let us compute the loss of both the KL term and the information bottleneck under this case. First note the formula: p ~ N(A,, ~ ~N 02) then
07 + (fi
P22)
1
DKL(Plla) log
+
(22)
262
2
Choosing
A1 = WH1 + b.
(23)
A = WHO 40 + b,
(24)
6f. 0 = 0f + 2
(25)
we find that
o: + w2 + W Mo)2
( - -m0)²
( 1) || 0))
(26)
2(02 + W2
2(0² + W220)
Let us now compute the information bottleneck term. Note the following result: if X,Y ~N(,,) where
ox ==
(
of ogxox) )
paxax
(27)
paxoy
then
1
I(X;Y)= = log (1 - p²²).
(28)
-
Note that
Σz, = (
of + W202
W
(29)
2
o²
12

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
therefore,
Wo
P2,2
(30)
0
Thus,
1
W²0²
-²
1(:2))
log
1
log
(31)
2
02 W2
2
0
Next, we compute
(11Y=1) 11 11/ p(a log p(a =
p(a =0) /
p = 0)
1(2;Y) = p(Y
dē + p(y =0)
0) log
d3
(32)
p(2)
p(2)
-ppYY 1)b(p(E1Y - -1)) - -p(Y - =0)((((E|Y - -0)) + ((p(z))
(33)
1
= - [log (2+(0² + 02)) + 1] + h((())))
(34)
where we used that
h(N(1,))) = 2 (ogg(200 + +1]
(35)
Note that
= - = 0) p(ž) p(Y = =
(36)
p(Y = (oo P - O]G(('; +b,0z)
(37)
lp(Y = =
(38)
=
where
ī - No -
o² W y'= W(u1 - po)
(39)
02
Thus,
Mx((3) = - / p( (og))
(40)
-00 / p(z') logp(z') de'
(41)
/ log)
[ dz'
(42)
0z
logo: - / G(()))
G(zz))d
(43)
llogo: + +(G)),
(44)
where
G(=')=) 1)G((2;0,1) p(Y 0)G(=':p',
(45)
Therfore,
1(2;Y) = 1
2 (log(2 + 1] + log o5 + +())
(46)
1
= 2
[log(2T)
+ 1] 1| (())
(47)
Therefore,
B [log(2r)
IB(p(E|z)) = log
+
(2T) + 1] 1111 Bh(
(48)
2
Finally,
[(p(3l=)) = W2(M - m2
+ log
a=
+
3
[og(2Tr) + +1] 1 (G)
(49)
20
2
13

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
-D KL + a
total loss
-D_KL KL
2.0
I(Z:Z)
-I(ZY))
IB
1.5
loss
1.0
0.5
0.0
-0.5
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
W
Figure Loss function ID Gaussian case. Note No - - 1, 7
0g = 1, and
Note DXL + alB where a = 0.5.
where
G(z) p(Y 1),(3;0,1) + O)G(2;p/,1
(50)
of o: + + p² W W(mn 411
(51)
(52)
We show the plot of this loss function in Figure
B. Simplifying the Loss With Independence Assumptions
We simplify our loss functional under independence assumptions. Specifically, we assume that
(()
p(2 (z)
(53)
p(=ly) II p(z Y y)-
(54)
14

---

A Variational Information Theoretic Approach to Out-of-Distribatton Detection
Note that
p(žly) / p(2)2)p(21)) dæ
(55)
/ II p(2)l2))p(2 d21
dzn
(56)
II / p(2))))((2))))
(57)
II p(2.))).
(58)
Therefore,
(xrlb((()))ll(( - f p(z10) log II p( da
(59)
Σ/ p(z log p( dz p(2.11)
(60)
(61)
and by similar computation, we get that
1(2:2)- Σ ((2622)
(62)
1(;;)) Σ ((2x1))
(63)
Thus, we see that
L(p(a|z) Ll( +
Σ L((p(E))))),
(64)
where
[.((EE/2:)
-
3T(Z,Y))
(65)
Therefore, we just need to solve n independent problems:
mig (p()))))), i (1....,)) -(a+))))
(66)
C. Gradient of Loss Computations
We review the definition of gradient functionals, which are functions defined on functions, in order to compute gradients of our loss. First, define the directional derivative. Let ôp(ā]z) denote perturbation of p( p()22), which is function of 2 with integral zero that p(2)2) + 20p(2)2) defines a valid probability (i.e., integrates to 1) for small. The direction derivative is defined as
d
(p(()))) 5p(E12) de
L'p(E)2) =0p(E
(67)
The gradient V VPCE ((((2)) is the perturbation of p(212) that satisfies the relation:
dl(p(E]>)) 6p(2)2) / (((E))) dp(E)z) dz.
(68)
15

---

A Variational Information Theoretic Approach to Out-of-Distribatioo Detection
C.I. KL Loss Gradient
We look into the optimization of
mAx 1)ll(( 0)}, p(
(69)
where D is the KL-divergence or relative entropy:
p(a) ((x)
D = p(x) log
dx,
(70)
that is we would like to compute such that the resulting distributions under in/out data are maximally separated.
We compute the optimizing conditions by computing the variation. First note the following:
p(žly) = / p(2lz, p((lai) = p(2)z)p(z1)) dz,
(71)
where the equality the right hand side is by assumption do not want our feature 2 to be dependent whether the data is OOD not. Now compute the variation:
/ 6 =) 6p(()=) log
p =1 1) p = 0)
6DKL 6p(212)
(72)
+ p(ā = 0)8 [p(2|== -11 1)
5p(312) dě.
(73)
Lp(ElY 0)
Let's compute
8p(31y) 65((1=) / (p(3)=)p(2l))dd= dz.
(74)
Therefore,
p(ž 1) p(ā = 0)
-8p(8)2)= 5 1)p(2 0) x 1)0p(E
(75)
p( = 0)
[ 1)dz p(ž =0) (p((3))p(=)1 = 0) p(3
(76)
p = =0)
Therefore,
p =1) 1) p(ā = 0)
-8p(E1z) / f (x()) = L())p(22/ = =0] dz,
p(Z = 0)8
(77)
where
plēty = ((3) D( 0)
(78)
is the likelihood ratio of the distributions of 2. Thus,
6DxL 5p(512) = / 6p(2)=) [(1 + log p(g() =1) 1(3))(())) = 0)] dz dz
(79)
where
1(2) p(² =1
Pin (in(2)
(80)
p(= 0)
Pout (2)
Therefore,
V P(aa)) DEL Pome(2) - 1(1 + log ((8))((2) 1(3))
(81)
16

---

A Variational Information Theoretic Approach to Nuttof-Distribution Detection
C.2. Information Bottleneck Gradient
We consider the information bottleneck term:
[B(((1==)) = - BT(
(82)
where I denotes mutual information. Note that
1(2;2) - / pl2, 8) log p(z, 2) p(z)((2)
dē dz = p(=))(212) log p(2)2)
dā dz.
(83)
p(2)
Also,
/
p(2,y) d=
/
((ēl))p(() log p(2 p(ž)
1( Σ
p(2; log
E
dč.
(84)
p(2)p(z)
8E{0,1}
vE(0.1)
Let us compute the variation of these terms:
8I(Z; 2) 6p(š|=) / p(=))p(2(=) log p(2)2) p(3)
p(zl2)6p(2) 6p(212)p(2) -((2))))) 5p(212) da
+ p(z)
(85)
p(#)
/ 6p(21=)p(=) +og log p(2[2)
p(ε)=)p(=)
/ 5p(2)= de' dsds.
(86)
p(3)
p(2)
Let us evaluate the term after the minus sign:
/S] 6p(E)=)) p(2)=)((=) p(2)
/ 8p( 60(()2 pla))
/ (d) p())
p( p(=') dz' dē
(87)
/ / dā de'.
(88)
Therefore,
61(2; 2) ó0(((z) = / ))p({2))((2) log p(E(z)
dē dz,
(89)
(=²)
V I = p(z) log p(ā]2)
(90)
p(2)
Let us compute the variation of the second term in IB:
8I(Z;))= Σ / 6p(2lu)p(u) log p(īly)
5p(31))p(2) - p(zl))6p(3)
+ +(v)
dz.
(91)
p(2)
p(ž)
(E(0.
Note that
6p(2) = / 6p(El=)p(=)d dz
(92)
6p(Ely) / (p(E)=))(Ely)dE
(93)
Therefore,
81(2:8)= Σ / [00(332)000)) (((ll)) log
p(āly)
p((())p(z)
p(2) + p(=lv)
dădz
(94)
p(2)
vE(U,1)
= =/ {OO(ET)a
5p(E1=) Σ p(v) (p(=lw) log
p(8ly)
+ +p(zly) p(21))p(=)
dž dz
(95)
p(2)
p(2)
8E(0,1)
= -fsop(1e)) Σ p(y)p(zl)) log p(īyy)
dā dz.
(96)
p(ž)
vE(0.1)
17

---

A Variational Information Theoretic Approach to Outtof-Distribattion Detection
Gaussian (M= 0.0 o= 0.66)
0.6
3 15.0)
(u= 3.3 30.0)
0.5
IG (μ 4.75 15.0)
Probability Density
0.4
0.3
0.2
0.1
0.0
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
Figure 7. Plot of Inverse Gaussian (IG) distribution, p(=[1) ~ IG(((z); A), under different parameters with a Gaussian (blue). Note that IG has high probability where the Gaussian does not.
Therefore,
p(6|2)
Σ p(y)p(=lv) log p(Ely) vE(U.1}
(() IB = p(2) log
-3
(97)
p(2)
p(2)
log p(ālz)
3 log p(žly)
Σ
p(u)p(zlu)
(98)
p(ž)
p(2)
(0.1)
D. Plot of Inverse Gaussian Distribution
In Figure 7, show plot of our Inverse Gaussian for various parameters along with Gaussian. Notice that the IG has mass complementary to the Gaussian, and thus represents natural distribution for OOD if the ID is Gaussian.
E. A Study of Our Feature Shaping Over Parameters
In this section, we perform an extended study feature shaping as function of parameters of the distributions and the parameter in the Information Bottleneck, extending the study in Section of the main paper.
E.1. Feature Shaping Versus /
In Figure 8, we explore how our feature shaping function varies function of 3 for various distributions.
E.2. Feature Shaping Versus Distribution Parameters
In Figure 9, we explore how our feature shaping function varies with different distribution parameters.
F. Visualizations for Additive Gaussian Noise Experiment
Figure 10 shows visualization of an example ImageNet image and various noise levels added to it to simulate OOD data for the experiment in Section 6.
Figure visualizes the activations of images from the validation set of ImageNet- with different levels of additive Gaussian noises. From figure 1la, with more noises, the activations are more instable and have more larger spikes. From figure 1 1b, the activations shrink more and peak more at small values.
18

---

A Variational Information Theoretic Approach to Out-of-Distribation Detection
Gaussian OOD
Laplace OOD
Inverse Gaussian OOD
7.5
10
10
5.0
2.5
5
5
u(z)
ulz)
u(z)
9=3.0
$=5.0
==0.1
-2.5
$=5.0
-5
6=7.0
6=5.0
$=10.0
==20.0
B=20.0
-5.0
9=100.0
B=100.0
B=100.0
-10
identity map
identity map
identity map
-7.5
1
2
4
4
2
2
2
Figure 8. The mean of the OOD Gaussian Random Feature under the Gaussian (left), Laplace (middle) and Inverse Gaussian (right)
distributions for the OOD distribution. Different curves on the same plot indicate differing weights on the / ((; Y) component of the Information Bottleneck term, B. The weight on the IB is fixed and p(=[1) ~ (0.55 0.5). For the Laplace case, p(= 0) 0.5) and p(z(1) ~ Lap(0, 1). In the Inverse Gaussian case, p(=|0) (0, 0.5) and p(z[1) IG(d(=))005. 15).
3.0. For the Gaussian p(2|0) WV-0.5, 0.5)
Gaussian OOD
Laplace OOD
Inverse Gaussian OOD
15
7.5
10:
10
5.0
5-
2.5
u(z)
μiz)
μ(z)
0.0
μ= 0.25 a= 0.78
μ=0.00 b=100
3=3.3 A-13.00
u=0.5 0=0.5
-5
==0.0 b=1.25
-5
==3.5 A=22.75
-2.5
0-0.75 0-0.78
p-0.0 b-1.62
p-4.75 A-15.5
-5.0
w=1.0 0=1.0
-10
==0.0 b=2.0
-10
μ=6.0 A=22.75
deetitty map
Identity map
Identity map
-15
2
2.5
5.0
7.5
10.0
Figure 9. The mean the OOD Gaussian Random Feature under the Gaussian (left), Laplace (middle) and Inverse Gaussian (right) distributions for the OOD distribution. Different curves the same plot indicate different OOD distribution parameters. The weight the IB term and weight of its I I(Z;Y) component are set as = 3.0 and 8 = 10.0. For the Gaussian case, p(z(0) ~ NV.-0.5,0.8 For the Laplace case, p(2(0) (O, 0.5). In the Inverse Gaussian case, p(=(0) (0000..5)
(a) σ = 25
(b) a = 50
(c) 100
(d) = 150
(c) a = 255
Figure 10. Visualization of a sample image from ImageNet validation split under different levels of noise corruption (a values).
19

---

A Variational Information Theoretic Approach to Detection
without noise
without noise
20
Activations
Activations
0
05
Aciivations
Activations
20
(min,
50
Activations
Activations
20
Activations
Activations
20
14
0-150
Activations
Activations
20
umin, max)
052
Activations
Activations
20
imin, maxi
500
1000
1500
2000
500
1000
1500
2000
Activation range of μ
(b) Activation range of (min, max)
Figure 11. The network activations of ImageNet with different levels of additive Gaussian noises. The shaded regions in (a) represent standard deviation above and below the mean, while those in (b) represent the range of min and max values of activations. With more noises, activations lend to instable, smaller range of activation values.
20

---

A Variational Information Theoretic Approach to Out-of-Distriaation Detection
G. Optimal hyperparameters used in ImageNet and CIFAR benchmarking
report the optimal hyperparameters for the experiments studied Section Table 3.
Hyperparameters
Model
ID Data
y/o
21
V18
7
Y112
ResNet-50
ImageNet-Ik kk
0.0
0.0
0.52
0.73
0.61
1.2
-0.3
MobileNet-v2
ImageNet-IK
0.0
0.0
0.55
0.5
0.79
1.49
-0.74
ViT-B-16
ImageNet-1k
0.0
0.0
0.05
1.58
2.0
2.0
-1.0
ViT-L-16
ImageNet-Ik 1
0.0
0.0
0.06
1.76
1.79
2.0
-0.32
DenseNett01
CIFAR 10
0.0
0.0
0.51
0.41
1.18
1.1
0.37
MLP-N
CIFAR 10
-0.3
0.25
0.73
0.40
0.10
3.54
1.76
DenseNet101
CIFAR 100
0.0
0.1
1.0
2.0
0.17
1.8
-0.18
MLP-N
CIFAR 100
0.0
0.3
0.59
0.4
0.1
4.0
2.0
Table Our optimal hyperparameters for different models and datasets.
H. Empirical Distribution of Network Feature Outputs
In this section, we show the empirical distributions of ID and OOD data for various ImageNet- Ik benchmarks. As is shown in the subsequent figures, some henchmarks/architeeturss resemble the distribution assumptions analyzed in this paper (e.g.. Gaussian ID/Gaussian OOD Figure and Gaussian ID/Laplacian OOD Figure 13 thus showing that these may be realistic assumptions. On the otherhand, some datasets (Figure 14 and Figure 15) exhibit distributions that do not fit the distributional assumptions analyzed in this paper. Nevertheless, benchmarks, showing that our shaping function may well work even when the data differs from the assumed distributions, which is important in practice as exact distributions may not be known.
novel shaping function still performs well on these
21

---

A Variational Information Theoretic Approach to Detection
In-D OOD: inaturalist ood features
In-D OOD: ood features
In Distribution
n-Distribution
1.50
1.50
OOD
OOD
1.25
1.25
Density
1.00
Density
1.00
0.75
0.75
0.50
0.50
0.00
1.5
0.5
4.0
Feature Value
Feature Value
In-D vs OOD: places ood features
In-D vs OOD: texture, ood features
In-Distribution
In-Distribution
1.50
1.50
OOD
OOD
1.25
1.25
Density
1.00
Density
1.00
0.75
0.75
0.50
0.50
0.25
0.25
1.0
1.5
3.0
3.5
0.5
1.0
2.0
2.5
3.5
4.0
Feature Value
Feature Value
Figure 12. Distribution of features from the penultimate layer of ViT-B-16. Comparison with In-distribution data (ImageNet- Ik) and different test OOD datascts the ImageNet- 1k benehmark (Zhao al., 2024). The ID and OOD distributions resemble the positive part of a Gaussian with different variance.
22

---

A Variational Information Theoretic Approach to Detection
In-D vs OOD: inaturalist ood features
In-D vs OOD: sun ood features
3.5
Distiriibution
In-Disribbttin
OOD
OOD
3.0
2.5
Density
Density
2.0
2.0
1.5
1.5
1.0
0.5
0.0
2
1
Feature Value
Feature Value
In-D vs OOD: places ood features
In-D vs OOD: texture ood features
2.5
In In-Distribution
In-Distribution
0OD
2.0
Density
Density
2-
1.0
0.5
0.0
2
5
1
Feature Value
Feature Value
Figure Distribution of features from penultimate layer ResNet-50. Comparison with In-distribution data (ImageNet-1kk and different test OOD datasets in the ImageNet- Ik benchmark (Zhao et al., 2024). The ID distribution resembles the positive part of Gaussian and OOD resembles the positive part of Laplacian distribution.
23

---

A Variational Information Theoretic Approach to Detection
In-D vs OOD: inaturalist ood features
In-D vs OOD: sun sun_ood features
Distribution
In-Disribbutio
OOD
OOD
Density
Density
3-
Feature Value
Feature Value
In-D vs OOD: places ood features
In-D vs OOD: texture ood features
7-
In-Distribution
In-Distribution
6-
Density
Density
4-
3-
2-
4
5
Feature Value
Feature Value
Figure Distribution features from the penultimate layer MobileNet-V2 Comparison with In-distribution data (ImageNet-Ik) different test OOD datasets in the ImageNet-Ik benchmark (Zhao et al., 2024). The ID and OOD both appear Laplacian; although this does not fit the ID assumption analyzed in this paper, our method nevertheless works well on this benchmark.
24

---

A Variational Information Theoretic Approach to Detection
In-D vs OOD: inaturalist ood features
In-D OOD: sun ood features
In-Distribution
In-Distribution
3.0
OOD
OOD
2.5
Density
Density
1.0
1.5
Feature Value
Feature Value
In-D vs OOD: places_ooe oodoooodo_oo features
In-D vs OOD: texture_oor features
In-Distribution
In-Distribution
OOD
OOD
Density
Density
0.5
0.5
1.0
1.5
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
Feature Value
Feature Value
Figure 15. Distribution of features from the penultimate layer of ViT-L-16. Comparison with In-distribution data (ImageNet 1k) and different test OOD datasets in the ImageNet- Ik benchmark (Zhao et al., 2024). The ID and OOD distributions appear Gaussian but with heavy weight on zeros; although these distributions don't fit the distributional assumptions in the cases analyzed in the paper, our method nevertheless performs well this benchmark.
25


Reviews:
[{"review_id": "QOcFO08Yc4", "reviewer": "Reviewer_4N3p", "summary": "The paper presents a novel theoretical framework for constructing out-of-distribution (OOD) detection features in neural networks using a variational information-theoretic approach. The key contribution is a novel loss functional that consists of a KL divergence term that maximizes the separation between in-distribution (ID) and OOD feature distributions, and an information bottleneck (IB) term that favors compressed features that retain relevant OOD information while discarding unnecessary details. \n\nThe loss functional is optimized using variational methods, leading to the derivation of OOD features as random variables. The mean of these random features corresponds to deterministic shaping functions used in existing OOD detection methods. By carrying large experiments, the authors show that their proposed piece-wise linear shaping function outperforms existing methods on standard OOD benchmarks, including ImageNet-1k and CIFAR datasets. The authors show that their method generalizes well across different architectures (e.g., ResNet, MobileNet, Vision Transformers) and datasets.\n\n---\n### Update after rebuttal\n\nI thank the authors for their responses. I will maintain my initial score.\n\n---", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Key Question 1\n\n**Response 1**: Thanks for the ref, we’ll cite.  We agree with the use of likelihood ratio (LR) as a score over the likelihood.  However, our work focuses on feature shaping not scoring (L60-61, 2nd column).  Our work separates the distributions of the resulting OOD feature shape distributions under ID/OOD through the KL divergence (& IB) to derive optimal shaping features, while [1] uses the LR of the original neural net feature as a score.  So while the LR relates to the KL div. (as the expectation of the LR), [1] defines a score, whereas our approach determines feature shape.\n\nKey Question 2\n\n**Response 2:** Our theoretical framework does not assume specific distributions: it is applicable to any ID/OOD distribution - no assumptions are needed for optimality.\n\nTo concretely illustrate the theory, example distributions are explored.  Section 4 explains our choice of distributions: Gaussian is common in probabilistic analysis; Laplace models heavy-tailed outliers typical of OOD; and Inverse Gaussian serves as a reasonable assumption when no prior knowledge is known - large when ID distribution is low and vice versa. These examples reveal that current SOA heuristics may implicitly assume properties of these distributions.  Having said that, our examples have similarities to distributions consistent with OOD data in the real-world.  VRA (Xu et al.) plots distributions of various OOD data in the ImageNet benchmark – OOD qualitatively have heavier tails and sharper peaks than ID data.  Gaussian ID / Laplace OOD choice match these properties.  In e.g. VIT-B-16, the distributions ID/OOD appear Gaussian – see https://drive.google.com/file/d/1XANDdWfXdy4MONY0MnQfaVOBoVNy7r0a/view?usp=sharing\nIn practice, OOD data will change based on application so it isn’t desirable to match too closely to any real-world dataset otherwise it would not generalize.\n\nKey Question 3\n\n**Response 3:** Ablations are not shown as, without IB, the loss is ill-posed (L114-116, 2nd column) - the optimization does not converge since distributions can be arbitrarily separated, increasing the KL term indefinitely. Appendix A explains this (L153-159, 2nd column) with a case that can be analyzed analytically - linear mean feature with Gaussian ID/OOD distributions. Without IB, the slope becomes unbounded. In the more general non-linear case, which cannot be solved analytically, the optimization also diverges without IB. This is consistent with literature – FS-Opt and VRA that separate distributions (though through mean separation) impose constraints by effectively restricting the slope of the feature shaping function. While restricting the slope does produce a well-posed problem, it may not be addressing the underlying cause of the ill-posed-ness. IB is more naturally suited to the OOD problem – retaining OOD information while compressing the feature.\n\nKey Question 4\n\n**Response 4:** See Section 5 Lines 342-350 (2nd column). Yes, consistently chosen over all datasets.\n\nKey Question 5\n\n**Response 5:** Our method has been demonstrated on high-dim data - the shaping functions are applied on each component of the penultimate layer feature (Sec. 5), as SOA does.  We experimented with two vision transformers (Table 1: ViT-B-16 and ViT-L-16).\n\n*Superiority of random features over deterministic …*\n\n**Response 6:** We formulated general random features; a question is whether they result in a better loss.  The answer is yes - maybe not for all cases, but all cases that we examined (all the distributions and hyperparameters in all the plots shown have non-zero standard deviation, we didn’t show them because of space and didn’t find it insightful).  We will clarify the statement in the paper – that our intent is not to claim random features are always more optimal, just that there is reason to consider them.\n\n*Scalability to high-dimensional data …*\n\n**Response 7:**  Our current methodology applies to high-dim data with independence assumptions (L185-190) as SOA.  Generalization of the independence assumptions is challenging as numerical methods presented won't scale - grid representations of multi-d probabilities are infeasible. While we have preliminary ideas to address this, it is future work. The current paper's goal is to establish foundational theory and demonstrate its utility. Despite focusing on 1D shaping functions, consistent with SOA, our theory successfully explains these methods and predicts new shaping functions. This underscores the promise of our approach and lays groundwork for future exploration.\n\nSee Reviewer Ghj6 Response 2 for complexity.\n\n*[2] limitations of information-theoretic methods like normalizing flows*\n\n**Response 8:**[2] shows why normalizing flows that produce exact likelihoods do not work well as a score for OOD detection. [2] relates to the scoring function, not feature shaping – which is the focus of our work. We’ll cite [2] in our discussion on scores.\n\n*[3]…should cite*\n\n**Response 9:** Thanks; will do.", "claims_and_evidence": "While the paper presents a strong theoretical framework with insightful connections to existing methods, certain claims, however, lack empirical validation, particularly those related to:\n\n1. OOD distribution assumptions: The proposed theoretical framework assumes specific statistical distributions for in-distribution (ID) and OOD data (Gaussian, Laplacian, and Inverse Gaussian). Real-world OOD data may not necessarily follow these distributions, and no empirical evidence is provided to justify these assumptions. The method's generalization to arbitrary OOD distributions is unclear.\n\n2. Superiority of random features over deterministic ones: The paper claims that random features (rather than deterministic shaping functions) are more optimal for OOD detection. However, this is only based on the assumed specific statistical distributions where the choice of the parameters seems handpicked and the dimensionality of the probability space reduced to 1D.\n\n3. Necessity of the Information Bottleneck term: The IB term is used for regularization, ensuring that features retain only the necessary OOD information while being compressed. However, the necessity of the IB term is not experimentally tested. Put differently, we don't know how good the loss functional would be without the IB term. An ablation study could help determine the effectiveness of the loss.\n\n4. Scalability to high-dimensional data: The optimization problem is inherently infinite-dimensional, requiring numerical approximation techniques. While they propose a computationally feasible method in 1D, the complexity of extending it to high-dimensional feature spaces is not addressed. Unfortunately, this is discussed extensively in the paper, even the complexity of the approach in a 1D setting is not discussed.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria mostly make sense for the problem of out-of-distribution (OOD) detection. The models and datasets they considered are widely used in the OOD literature as benchmarks.", "theoretical_claims": "I didn't check in depth the proofs of the theoretical claims. In fact, I mostly focused on understanding the proposed loss, the variational framework, and its effectiveness in detecting OOD samples.", "experimental_designs_or_analyses": "Yes, I examined the experimental design and analyses focusing on methodology soundness, validity of comparisons, and potential issues. Below is a structured review. \n\nStrengths: The authors compare their method to several state-of-the-art (SoA) OOD detection approaches, including:\nfeature shaping methods (e.g., ReAct, FS-OPT, VRA-P, ASH), Softmax-based methods (e.g., MSP, ODIN), and energy-based methods (e.g., Energy, DICE). This makes their results contextualized within the broader OOD detection field. \n\nTheir experiments were conducted on widely accepted datasets. They considered ImageNet-1k as the in-distribution (ID) dataset, and \nSpecies, iNaturalist, SUN, Places, etc. as the OOD datasets. They considered CIFAR-10/100 as ID datasets with common OOD datasets such as TinyImageNet, SVHN, Texture, Places365, LSUN-Cropped, LSUN-Resized, iSUN, CIFAR100/10.  All these datasets seem appropriate for natural image distribution shifts, making them relevant for evaluating OOD detection.\n\nThe paper evaluates many widely accepted neural networks models using metrics such False Positive Rate at 95% True Positive (FPR95) for OOD detection, and AUROC to measure overall separability between ID and OOD samples. These are standard and valid metrics for OOD detection performance evaluation.", "relation_to_broader_scientific_literature": "OOD detection is a widely studied problem in machine learning, and various techniques have been proposed, including confidence-based, energy-based, distance-based, and feature-shaping methods. The paper's contributions build upon these methods while introducing a variational formulation using information theory.\n\nWith respect to Confidence-based OOD Detection methods, the paper moves away from softmax-based methods, which rely heavily on the output space rather than feature space. The authors argue that feature-based approaches offer better generalization than confidence-based methods.\n\nFor Energy-based OOD Detection methods, the variational loss function in this paper can be seen as an extension of energy-based approaches, since KL divergence maximization between ID and OOD features implicitly encourages separation. However, the proposed method focuses on shaping feature distributions rather than directly modifying the energy function.\n\nWith respect to Distance-based OOD Detection methods, the proposed method does not rely on distance metrics, but instead optimizes a variational loss functional to encourage separation between ID and OOD distributions. However, the KL divergence term serves a similar function to distance-based methods by ensuring OOD features are statistically different from ID features.\n\nFor Information-Theoretic OOD Detection methods, the proposed loss explicitly incorporates IB regularization into the OOD feature optimization process. The authors justify several existing OOD shaping methods (e.g., ReAct, FS-OPT) as special cases under their framework.", "essential_references_not_discussed": "The paper introduces a variational formulation for OOD feature extraction but fails to cite prior works that have applied variational or information-theoretic techniques to OOD detection. There are existing variational formulations for OOD detection that should be referenced for proper context. \n\n[1] proposes an information-theoretic approach for OOD detection by using likelihood ratios instead of raw likelihoods. [1] is similar to this paper in the sense that it formulates OOD detection from an information-theoretic perspective. The likelihood ratio idea relates to KL divergence-based feature separation in the current work. It would be useful to know how the KL divergence optimization proposed in this paper compares to likelihood-ratio-based OOD detection.\n\n[2] Examines the limitations of information-theoretic methods like normalizing flows for OOD detection. Since the proposed method is information-theoretic, the authors should discuss known pitfalls from this prior work. For instance, does the variational approach in this paper avoid the same failure modes?\n\n[3] Investigates how pre-trained models learn OOD features, showing that deeper layers contain more useful information. This work suggests that OOD detection improves when using deep feature representations. The current paper should cite this when justifying why it focuses on feature-based OOD detection.\n\nReferences:\n[1] Ren et al. (2019) – \"Likelihood Ratios for Out-of-Distribution Detection\" (NeurIPS 2019)\n[2] Kirichenko et al. (2020) – \"Why Normalizing Flows Fail for Out-of-Distribution Detection\" (NeurIPS 2020)\n[3] Fort et al. (2021) – \"Exploring the Limits of Out-of-Distribution Detection\" (NeurIPS 2021)", "other_strengths_and_weaknesses": "Overall, the paper is well-written and insightful. Providing answers to the questions I asked above would bringing more clarity about the key contributions of the paper.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Key questions:\n\n1. How does the variational optimization framework compare to likelihood ratio-based OOD detection (e.g., Ren et al., NeurIPS 2019)? This method maximizes KL divergence to separate ID and OOD feature distributions. Likelihood ratio-based OOD detection also exploits divergence between ID and OOD distributions but does so directly in a Bayesian framework. Understanding the difference in theoretical properties would clarify whether this approach is more generalizable or computationally efficient than likelihood-based OOD methods.\n\n2. What specific assumptions about OOD distributions are necessary for the proposed feature shaping functions to be optimal? This method assumes Gaussian, Laplacian, or Inverse Gaussian distributions for OOD features. However, real-world OOD data may not follow these assumptions. It is unclear how the method would generalize to cases where OOD features do not fit these distributions.\n\n3. Why didn't the authors include an ablation study comparing performance with and without the Information Bottleneck (IB) term? The IB term is a key part of the proposed framework, but there is no direct evidence that it improves results. Without an ablation study, it remains unclear whether IB contributes meaningfully or if similar results could be achieved without it.\n\n4. How were the hyper-parameters (e.g., IB weight $\\alpha$) selected, and were they tuned consistently across all datasets?\n\n5. Since the use case that was presented in the paper focuses on 1D data, I am curious to know how well this method scales to high-dimensional data (e.g., Vision Transformers, large-scale multi-modal datasets)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HaDjld1nlZ", "reviewer": "Reviewer_Ghj6", "summary": "The paper introduces a variational information-theoretic framework for OOD detection. It models OOD features as random variables by optimizing a loss function that balances KL divergence for feature separability and Information Bottleneck (IB) regularization for compactness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Question 1: The method assumes some prior knowledge of the OOD distribution (e.g., Gaussian, Laplacian). How does it perform when the OOD distribution is completely unknown or highly complex?*\n\n**Response 1:**  Our experiments show the performance of our piecewise linear family that encompasses Gaussian, Laplacian, Inverse Gaussian perform well across datasets, even when the OOD distributions do not closely fit some of these datasets (see Reviewer 4N3p Response 2).  Our piecewise linear form makes more general distribution assumptions than SOA techniques as several SOA methods are related to our framework under one of the three distributions above, which is the reason for its out-performance of existing SOA.\n\n*Question 2: The proposed optimization involves solving an infinite-dimensional problem using variational calculus. How does this impact computational efficiency compared to simpler heuristic methods like ReAct?*\n\n**Response 2:** The inference cost is about the same as existing methods like ReAct. Since this optimization is done offline in training, the cost is not important.\n\nThe complexity of optimization for 1D is $\\mathcal O (NMK)$ where N is the samples of $p(z)$, M is the samples of $p(\\tilde z|z)$ and $K$ is the number of gradient descent iterations.  In our experiments with $N=M=200$, the algorithm took between 5-10mins on a standard desktop. \n\n*Question 3: minor typos: Section 3.1: \"we make some simplifications to gain insights to our theory and approach.\" → Should be: \"we make some simplifications to gain insights into our theory and approach.\"?*\n\n**Response 3:** Thanks; will do.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have browsed the proof but have not examined it line by line.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This work builds upon and extends previous research by enhancing OOD detection, a crucial aspect of ensuring the reliability of deep learning models in real-world applications.", "essential_references_not_discussed": "Related work appears to be discussed.", "other_strengths_and_weaknesses": "Strengths:\nThe method presented in the paper improves upon a certain method (e.g. ReAct) by providing a theoretical framework rather than relying on heuristic rules like clipping activations. Unlike ReAct, which applies a fixed threshold to activations, the proposed approach learns an optimal OOD feature distribution using variational optimization. This allows the model to adapt dynamically to different OOD distributions, whereas ReAct may require manual tuning of the clipping threshold. \n\nThe paper’s method explains why different feature-shaping techniques (like ReAct, ASH, VRA, and FS-OPT) work under specific assumptions, offering a more general and theoretically grounded solution. \n\nEmpirically, it also achieves good results on benchmark datasets.\n\nWeaknesses:\n\nThe method assumes some prior knowledge of the OOD distribution (e.g., Gaussian, Laplacian). How does it perform when the OOD distribution is completely unknown or highly complex?\n\nThe proposed optimization involves solving an infinite-dimensional problem using variational calculus. How does this impact computational efficiency compared to simpler heuristic methods like ReAct?", "other_comments_or_suggestions": "minor typos:\nSection 3.1: \"we make some simplifications to gain insights to our theory and approach.\" → Should be: \"we make some simplifications to gain insights into our theory and approach.\"?", "questions_for_authors": "Please see weaknesses part above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5QWpmn9LmQ", "reviewer": "Reviewer_p7gJ", "summary": "### Background\n\n- This paper works on the OOD detection task.\n- Previous methods use different feature shaping functions to reshape features from the penultimate layer of a pre-trained network. They achieve SoTA performance on OOD detection benchmarks, but lack in theoretical evidence and may not generalize to unseen datasets.\n\n### Method\n\n- The authors develop a theory to formulate OOD features, and they propose a new loss functional consisting of two terms: “the first based on the KL divergence separates resulting ID and OOD feature distributions and the second term is the Information Bottleneck, which favors compressed features that retain the OOD information.”\n- Their theory can recover properties of existing feature shaping functions, based on different assumptions on OOD distributions.\n- They also design a new shaping function.\n\n### Results\n\n- Their new shaping function outperforms previous methods on OOD detection benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Question 1: The proposed method is a piecewise linear shaping function, which contains 7 hyperparameters. According to the supplementary materials, the values of these hyperparameters are varying with different models on different benchmarks. For example, \nis 0.73 when using ResNet-50 but 1.76 when ViT-L-16. How do the authors choose the values of hyperparameters?*\n\n**Response 1:** See Experiments Section 5 Line 342-350 (2nd column).  Yes, the optimal shaping function will depend on the specific architecture and its weights (hence the ID datasets) since each will have different statistical distributions – these can all be determined in training as only the ID dataset is needed.  Note the hyper-parameters are fixed across all OOD datasets.\n\n*Question 2: I think the proposed method is not related with the theoretical analysis. I appologize if I miss something important. I expected that, the authors can \"infer\" some good shaping functions based on their theory. For example, maybe the theory can tell us what family of functions is good or help us to decide the values of hyperparemeters. But I don't see how the authors utilize their theory to design the function. It seems that the proposed function is mannually designed on different datasets.*\n\n**Response 2:**  The reviewer’s expectation is correct our theory should predict a good shaping function, and it does.  The piecewise linear family used in the experiments encompasses piecewise linear approximations of the resulting optimal shaping functions from the Gaussian, Laplacian and Inverse Gaussian OOD distributions, as discussed in Section 4 Lines 344-350.  Notice that shaping functions for the Gaussian, Laplace, IG  (Figure 2) all can be approximated well with a function in the family in Fig.4.  So rather than tuning hyper-parameters for the given distribution and searching over the three distributions, we instead equivalently tune the hyper-parameters for the piecewise linear family – as this is more convenient practically.  Note this not unlike previous literature (e.g., VRA) where hyper-parameter tuning of the shaping function is required.\n\nThe proposed shaping function family is not manually designed on different datasets, see Response 1.\n\n*Question 3: How to understand OOD feature is \n? Why does the OOD feature depend on feature \n? What does this \n mean, the neural network or the shaping function?*\n\n**Response 3:** $p(\\tilde z|z)$ is the conditional probability of the OOD feature ($\\tilde z$) given the input standard NN feature (z) – see Lines 125-132.  $p$ is a probability.  Note that just as conventional deterministic shaping functions (e.g., f(z)) depend on the input feature z, so does our random feature; i.e., one needs to input the NN feature to determine the corresponding OOD feature.\n\n*Question 4: Eq (4), how to understand \"where \\tilde{Z} is analogous to T and Z is analogous to X\"?*\n\n**Response 4:** In traditional IB (from Tishby et al 2000) – $T$ is a compressed representation of the data/feature $X$ that we want to solve for, and $Y$ is the random variable whose information is to be preserved from $X$.  The traditional IB objective function is then $IB = I(X;T) – \\beta  I(T;Y)$.\nIn our formulation, $\\tilde Z$ is a compressed representation of the feature $Z$, and $Y$ is the information that we would like to preserve from $Z$. Since $\\tilde Z$ is the compressed variable, it is analogous to $T$; and since $Z$ is the original feature/data, it is analogous to $X$. Our IB term is $IB = I(Z;\\tilde Z) – \\beta  I(\\tilde Z;Y)$.\n\n\n*Question 5: I'm not sure if the proposed theory aligns with the real practice. Under different OOD distribution assumptions, the theory can recover some properties of previous methods, but normally, these methods are all good on ImageNet benchmarks. So, what is the real OOD distribution of the benchmark? Which method is the optimal one?*\n\n**Response 5:**  See Reviewer 4N3p Response 2.  There is no “real” distribution of the benchmark as there are many datasets in the benchmark with different distributions.\n\n*Question 6: Algorithm 1 looks very sparse. Maybe you can modify it somehow to make it more compact.*\n\n**Response 6:** Thanks, will do.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "- The proposed method is a piecewise linear shaping function, which contains 7 hyperparameters. According to the supplementary materials, the values of these hyperparameters are varying with different models on different benchmarks. For example, $y_{1b}$ is 0.73 when using ResNet-50 but 1.76 when ViT-L-16. How do the authors choose the values of hyperparameters?\n- I think the proposed method is not related with the theoretical analysis. I appologize if I miss something important. I expected that, the authors can \"infer\" some good shaping functions based on their theory. For example, maybe the theory can tell us what family of functions is good or help us to decide the values of hyperparemeters. But I don't see how the authors utilize their theory to design the function. It seems that the proposed function is mannually designed on different datasets.", "theoretical_claims": "Sorry that there are some parts I don’t quite understand. Could the authors provide more explanation?\n\n- How to understand OOD feature is $\\tilde{Z}\\sim p(\\tilde{z}|z)$? Why does the OOD feature depend on feature $z$? What does this $p$ mean, the neural network or the shaping function?\n\n- Eq (4), how to understand \"where \\tilde{Z} is analogous to T and Z is analogous to X\"?\n\n- I'm not sure if the proposed theory aligns with the real practice. Under different OOD distribution assumptions, the theory can recover some properties of previous methods, but normally, these methods are all good on ImageNet benchmarks. So, what is the real OOD distribution of the benchmark? Which method is the optimal one?", "experimental_designs_or_analyses": "The experiments are conducted on standard OOD detection benchmarks. The authors validate their method with different models and different datasets.", "relation_to_broader_scientific_literature": "Feature shaping methods achieve the SoTA performance in OOD detection, but lacks in theoretical analysis. This paper provides a new theory, which can recover some properties of previous shaping functions. I believe their paper can inspire more ideas in the future.", "essential_references_not_discussed": "I think all related works are included and discussed.", "other_strengths_and_weaknesses": "### Strengths\n- Provide theoretical analysis, which is very interesting. The theory can recover properties of previous methods.\n\n- FS-OPT only considered the mean of ID and OOD features, while this work considers the distribution.\n\n- Their proposed method achieves better performance on different benchmarks.\n\n### Weaknesses\n\n- Another major drawback of FS-OPT is the assumption of independence between features, so their framework cannot explain vector-based shaping functions like ASH. This paper also assumes independence of features, so I think it can be extended in the future.", "other_comments_or_suggestions": "Algorithm 1 looks very sparse. Maybe you can modify it somehow to make it more compact.", "questions_for_authors": "I think the theoretical analysis is very interesting but the proposed method is confusing. I will raise my score if the authors can address my concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QEUwBDmgTE", "reviewer": "Reviewer_q3hx", "summary": "The paper introduces a variational and information bottleneck-informed framework for performing feature shaping for  out-of-distribution detection. Namely, the proposed objective maximizes the KL distribution between the distribution of ID features $Z$ and OOD features $\\tilde{Z}$ subject to an information bottleneck regularization which seeks to maximize the mutual information between $Z$ and the ID / OOD label $Y$ while minimizing the mutual information between $Z$ and $Z$. The work show qualitatively that the feature-shaping approaches proposed by past work are similar to their objective under the assumption of independent features and different choices for the distribution of the OOD features (e.g., Gaussian with a different mean from ID, Inverse Gaussian, etc.) and regularization coefficient on the information bottleneck term. Then, the work considers a general piecewise linear shaping function for feature shaping and shows that fitting this function to validation OOD data outperforms previously proposed feature shaping OOD detection methods on test OOD data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Question 1: It's not clear to me how the proposed theory predicts the generalized piecewise linear feature shaping function proposed later in the work, as mentioned in the abstract.*\n\n**Response 1:** See Response 2 to Reviewer p7gJ.\n\n*Question 2: Where does the claim \"Note this forms a Markov Chain $Y\\rightarrowX\\rightarrowZ\\rightarrow\\tilde{Z}\" (line 137-138) come from? It seems more like an assumption the authors are making (a reasonable one to me), but might be better written that way.*\n\n**Response 2:** The reviewer is correct, it is an assumption (not a claim) that Y (ID or OOD class) produces data/image X from which the feature Z is computed, and then from the feature Z, the OOD feature Zt is computed.  The reason for spelling this out is to use the Information Bottleneck, which requires this setup.  We will add a comment in the manuscript.\n\n*Question 3: Experimental design seems sound. The qualitative analysis comparing the mean of \n under the proposed objective with different assumptions vs. existing feature shaping methods is quite interesting, but ultimately seems to be an approximate relationship based on the shape of the resulting curves; for instance, the actual functions are not the same if you look at the x and y axes' values.*\n\n**Response 3:** Yes, the reviewer is correct – the relation between our shaping functions and SOA is qualitative in shape and approximate as discussed in Section 4 (e.g. Line 294 – that under Gaussian OOD the approach is similar to clipping in ReAct; Line 324 – that under Laplacian OOD a positively sloped region for intermediate values is similar to VRA, etc).  It isn’t our goal to precisely numerically match SOA methods – as many are heuristically derived and there is no reason to believe that the optimal solution based on principles will precisely match empirically driven heuristics.  We aim to understand whether SOA practice, though heuristic-driven has similarities of what an optimal principled approach would predict (hence justifying and understanding these heuristics by showing that the heuristics have traits of a principled approach), but this will not be a precise match in numerical values.  Our work is interested in qualitative understanding of SOA such as, why/when should one clip values of the feature (like ReAct)?, why/when is it a good idea to suppress small values (like ASH)?   We believe the answers to these questions give intuition and insights that will help researchers design new methods.\n\nHaving said that, the primary reason that our shaping functions are not on the same scale as current SOA shaping functions is because current SOA’s scale is chosen to fit the energy score function (this is mentioned in the footnote of page 6), while our theory does not involve the score since our goal is to understand the feature making minimal assumptions.  It is nevertheless interesting to see that our shapes (in Fig 2), which resemble SOA, do not depend on choice of score, even though SOA has made the energy score assumption and tuned according to that score.  This indicates the shapes to be a universal property regardless of score.  Note to match scales in practice with the energy score as we do in experiments, our piece-wise linear function family encompasses functions that also have similar scale values to SOA methods.\n\n\n*Question 4: Can the authors explain what they say that the proposed theory predicts a new shaping function that can outperform existing ones? And explain the evidence for the claim?*\n\n**Response 4:** The piece-wise linear family of shaping functions introduced are piece-wise linear approximations of the optimal shaping functions in Fig 2 (see Response 2 to Reviewer p7gJ).  The results of the optimal piecewise linear function have out-performed SOA as shown in the Experiments Section 5.\n\n*Question 5: Could the authors update Figure 2 to show curves that would closely match those in Figure 3 if overlaid using the same axes?*\n\n**Response 5:** Please see Response 3.", "claims_and_evidence": "1. It's not clear to me how the proposed theory predicts the generalized piecewise linear feature shaping function proposed later in the work, as mentioned in the abstract.\n2. Where does the claim \"Note this forms a Markov Chain $Y\\rightarrowX\\rightarrowZ\\rightarrow\\tilde{Z}\" (line 137-138) come from? It seems more like an assumption the authors are making (a reasonable one to me), but might be better written that way.\n3. Other claims seem reasonable.", "methods_and_evaluation_criteria": "The ImageNet-1k and CIFAR-10/100 OOD detection benchmarks make sense for comparing OOD detection methods. The authors also test 4 different pretrained models encompassing convolution- and transformer-based architectures.", "theoretical_claims": "I skimmed but did not carefully check the algebra for the Gaussian example and gradient derivations.", "experimental_designs_or_analyses": "Experimental design seems sound. The qualitative analysis comparing the mean of $\\tilde{z}$ under the proposed objective with different assumptions vs. existing feature shaping methods is quite interesting, but ultimately seems to be an approximate relationship based on the shape of the resulting curves; for instance, the actual functions are not the same if you look at the x and y axes' values.", "relation_to_broader_scientific_literature": "The paper proposes a unifying framework to think about feature shaping methods for OOD detection, enabling more critical examination of existing and new methods in this space.", "essential_references_not_discussed": "To my knowledge, the essential references are discussed.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper provides a unifying and principled framework to view many disparate recent methods in OOD detection.\n2. The analysis from the framework leads to some observations about the properties that seem to work well for feature shaping.\n\nWeaknesses:\n1. The link between the settings discussed in the framework and existing methods is qualitative, as the curves match only in approximate shape.", "other_comments_or_suggestions": "One way to strengthen this connection mentioned in weaknesses above would be to choose assumption values that would result in functions that are close in distance, not just in shape.", "questions_for_authors": "My primary questions are:\n1. Can the authors explain what they say that the proposed theory predicts a new shaping function that can outperform existing ones? And explain the evidence for the claim?\n2.  Could the authors update Figure 2 to show curves that would closely match those in Figure 3 if overlaid using the same axes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel theoretical framework for out-of-distribution (OOD) detection in neural networks. Reviewers generally acknowledge the novelty of the proposed framework and commend several positive aspects, particularly its innovative theoretical perspective and demonstrated experimental efficacy. However, the reviewers also raise several critical concerns that warrant attention: (1) insufficient empirical validation of the key theoretical assumptions underlying the framework; (2) insufficiently clarified connections between the proposed methodological implementations and their corresponding theoretical analyses. Notwithstanding these limitations, the work constitutes a substantive contribution to the International Conference on Machine Learning (ICML) research community. The authors demonstrate strong potential to address the identified issues through systematic revisions. Therefore, a moderate-priority acceptance is recommended, provided that the authors adequately address the aforementioned concerns in their revision.
Decision: Accept (poster)