Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the Role of Model Complexity
Authors: Mouin Ben Ammar
David Brellmann
Arturo Mendoza
Antoine Manzanera
Gianni Franchi
Venue: NeurIPS 2025 poster ()
Keywords: OOD
Uncertainty quantification
Double descent.
Primary area: probabilistic_methods

Abstract: **Out-of-distribution (OOD) detection** is essential for ensuring the reliability and safety of machine learning systems. In recent years, it has received increasing attention, particularly through post-hoc detection and training-based methods. In this paper, we focus on **post-hoc OOD detection**, which enables identifying OOD samples without altering the model's training procedure or objective. Our primary goal is to investigate the relationship between **model capacity** and its OOD detection performance. Specifically, we aim to answer the following question:
*Does the Double Descent phenomenon manifest in post-hoc OOD detection?* This question is crucial, as it can reveal whether overparameterization, which is already known to benefit generalization, can also enhance OOD detection.
Despite the growing interest in these topics by the classic supervised machine learning community, this intersection remains unexplored for OOD detection.
We empirically demonstrate that the Double Descent effect does indeed appear in post-hoc OOD detection. Furthermore, we provide theoretical insights to explain why this phenomenon emerges in such setting. Finally, we show that the overparameterized regime does not yield superior results consistently, and we propose a method to identify the optimal regime for OOD detection based on our observations.

Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the Role of Model Complexity
Mouïn Ben Ammar ist David Brellmann' Arturo Mendozal, Antoine Manzanera', Gianni Franchi
U2IS Lab ENSTA Paris Palaiseau, FRANCE Safran Tech', Chateaufort 78117, FRANCE
ffirst.last/ Saetssaaaaate safrangroup.com com
Abstract
Out-of-distribution (OOD) detection is essential for ensuring the reliability and safety of machine learning systems. In recent years, it has received increasing attention, particularly through post-hoc detection and training-based methods. In this paper, we focus on post-hoc OOD detection, which enables identifying OOD samples without altering the model's training procedure objective. Our primary goal is to investigate the relationship between model capacity and its OOD detec- tion performance. Specifically, we aim answer the following question: Does the Double Descení phenomenon manifest in post-hoc OOD detection? This question is crucial, as it can reveal whether overparameterization, which is already known to benefit generalization, can also enhance OOD detection. Despite the growing interest these topies by the classic supervised machine learning community, this intersection remains unexplored for OOD detection. We empirically demonstrate that the Double Descent effect does indeed appear in post-hoc OOD detection. Furthermore, provide theoretical insights to explain why this phenomenon emerges in such setting. Finally, we show that the overparameterized regime does not yield superior results consistently, and propose method to identify the optimal regime for OOD detection based on our observations. Code available here
Introduction
Since the breakthrough of AlexNet in 2012 (Krizhevsky et al., 2012), deep learning has seen rapid progress and has become the foundation for solving wide variety of complex tasks across domains such as vision, language, and robotics (LeCun et al.) 2015 [Bngioo] 2009). While Occam's 's Razor suggests favoring simpler models with fewer parameters (Blumer al. 1987), deep neural networks (DNNs) with massive overparameteriization have nonetheless demonstrated remarkable generalization ability practice.
Traditionally, generalization performance has been characterized by a U-shaped test error curve with respect to model complexity (Geman al. 1992; Hastie 2009). recommending "sweet spot" where the model is expressive enough to avoid underfitting but not so complex that it over- fits. However, this classical view has been challenged in 2019 by the emergence of the double descent phenomenon (Belkin et al. 2019), which reveals a second descent in test error after the interpolation threshold. This insight has reshaped our understanding of the generalization behavior of overparameterized models.
Generalization typically refers in-distribution (ID) performance- to be drawn i.i.d. from the same distribution as training data. However, this closed-world assump-
data are assumed
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tion (Scheirer et al. 2012) rarely holds in real-world scenarios. In open open-world settings (Bendale Boult 2016; Drummond & Shearer 2006). models frequently encounter out-of-dist (OOD) inputs that differ significantly from training data, either due to semantic shift (e.g., new classes) (Hendrycks & Gimpel| 2017) or covariate shift (e.g., domain changes) (Ben-David et al.
2010 Wang & Deng, 2018: Li et al. 2017).
In this work, we focus on OOD detection under semantic shifts as defined in Yang et (0024) where the goal is to identify inputs from unseen classes. More precisely, we investigate post-hoc OOD detection methods VDeVsses & Taylor 2018 Liu et al. 2020 Sun al. 202I). which operate on top of pretrained classifier without altering its training process. These methods are widely used due to their practicality and modularity.
Our central research question is: How does model capacity affect the effectiveness of post-hoc OOD detection? While double descent has been studied in the context of ID generalization, its role in OOD detection remains unexplored. We empirically demonstrate that post-hoc OOD detection also exhibits a double descent curve with respect model complexity. This observation is important as it suggests that overparameterization is not universally beneficial for OOD detection.
We conduct extensive experiments across different architectures (CNNs, ResNets, ViTs, Swin) and post-hoc detection methods to confirm this trend. Furthermore, under simplified Gaussian mixture model, derive theoretical insights using tools from random matrix theory that mirror the empirical findings. Surprisingly, we also observe cases where the underparameterized regime achieves better OOD detection performance, prompting to investigate these cases through the lens of Neural Collapse (Papyan et al. 2020), which provides geometric interpretation of the learned representations convergence.
Contributions. We advance the theoretical and empirical understanding of model complexity in post-hoc OOD detection:
1. We are the first to empirically highlight a double descent phenomenon in OOD detection across both CNN and transformer architectures.
2. We provide theoretical insight using random matrix theory, showing that both ID and OOD risks peak at the interpolation threshold.
3. We show that overparameteriizaion is always optimal for OOD detection, and propose Neural Collapse-based criterion to detect when simpler models may perform better.
2 Related work
OOD Detection, OOD detection research focuses on two primary directions: Training- and post-hoc methods. We will focus latter ones. These be categorized based the essential feature used for the scoring function. First, logit- or confidence-basee methods leverage network
logits to derive confidence measure used an OOD scoring metric (Hendrycks & Gimpel 2017
Vries & Taylor 2018 Liu et al. 2020:
Huang & Li 2021 Hendrycks et al. 2022). A common
OOU NSE That Lem
90
80
70
60-
50-
Inarpoation
10a
10r
width (Feature Count)
(a) OOD Risk and MSE Test Loss
(b) OOD detec. confidence score
(c) Input space projection
Figure 1: Illustration of the double descent phenomenon in a Random ReLU Feature model as a
function of model width (log scale). (a) Evolution of the in-distribution (ID) Mean-Squared Error
(MSE) and OOD detection risk. (b) Confidence-based OOD score defined in equation [4] (c) Thrree-iimenioonll t-SNE projection of the input space, visualizing the separation between ID and OOD samples. The ID samples (n = 1,000) are drawn from a Gaussian Mixture Model (GMM) fitted to a subset of CIFAR-10, while OOD samples are drawn from an independent GMM.

---

baseline for this methods is the softmax score (Hendrycks & Gimpel, (017). which simply uses the model softmax prediction as the OOD score. Then, Energy (Liu et al. (020) elaborates on by computing the LogSumExpp on the logits, thus offering empirical and theoretical advantages over the Softmax confidence score. Second, feature-based and hybrid methods (Tee et al. 201 Wang et al., 2022; Sun et al.] [022 Ming et al. 2023; Djurisic et al. 2023 Ammmar et al., 2024) exploit the model's final representation to derive the scoring function. Hybrid methods further augments the scoring metrics defined on the features using logits as weighting factors. Mahalanobis (Lee et al.) 2018b) estimates density on ID training samples using a mixture of class-conditional Gaussians. NECO (Ammar et al. 2024). on the contrary, leverages the geometric properties of Neural Collapse to construct a scoring function based on the relative norm of a sample within the subspace defined by the ID data. Besides OOD detection methods, Fang et al. (2024) have used risk formulations closely related to the one we use in this work to study PAC bounds for OOD detection.
Double Descent. The double descent risk curve was introduced by Belkin et al. (2019) to explain the good performance observed in practice by overparameterized models (Zhang et al. 2021 Belkin
al., 2018 Nakkiran et al. 2021) and to bridge the gap between the classical bias variance trade-off theory and modern practices. Theoretical investigation into this phenomenon mainly focuses on
various linear models in both regression and classification problems through the Random Matrix
Theory (Louart et al. 2018: Liao et al.] 2020; Jacot et al., 2020 Derezinski et al. 2020; Kini &
Thrampoulidis 2020: Mei & Montanari 2022 Deng al. [0022] Bach, [024]] Brel
al.
2024). techniques from statistical mechamics (D Ascoli et al., 2020; Canatar et al. 2021). the VC
theory (Lee & Cherkassky 2022; Cherkassky & Lec|[002] or novel bias varianee decomposition in
deep neural networks (Yang et al.
2020). The double descent phenomenon has also been observed in
experiments with popular neural network architectures (Belkin al. 2019 Nakkiran et al., (021).
In addition to depending on the model complexity, the double descent phenomenon also depends
on other dimensions such as the level of regularization (Liao et al.) 2020 Mei & Montanari 2022),
the number of epochs (Nakkiran et al. 2021 Stephenson & Lee 2021 Olmin & Lindsten 2024).
or the data eigen-profile (Liu al. 2001 la). Finally, the theoretical background on double descent
and benign overparametrization developed by Bartlett et al. (2020) inspired subsequent works that focused on generalization under dataset shifts (Tripuraneni et al.) 2021b; Hao et al.) 2024 Kausik iet al. 2024: Hao & Zhang 2024). It is important to note that these dataset shifts concern scenarios where the model can generalize, e.g., the class labels are the same as in the training. These studies do not address OOD detection.
3 Preliminaries
Notations. For a real vector v. we denote by ||vl|2 the Euclidean norm of v. When the matrix
A is full rank, we denote by A+ the Moore-Penrose inverse of A. We depict by [d] == {1..... d} the set of the d first natural integers. For subset T S [d]. we denote by T == [d] its complement set. For subset T C [d]. d-dimensional vector v E Rd and x d matrix
A = |... (")) E Raadd we use uT = [to :j E T] to denote its [T]-dimensional subvector of entries from T and AT = [[a]]] .... aa (m)7 T from T. For subset T C [d]. we define R. (T) == { E Rd 2T = -
to denote the n x [T| design matrix with variables
}. We use Amin and Amax (A) to depict the min and max eigenvalues of A, respectively. A (O Ia) denotes the standard multivariate Gaussian distribution of d random variables.
Supervised Learning Problems. In supervised learning, use training dataset D (x)),)), ...,...),............... (wn, vn)] of n independent and identically distributed (i.i.d.) samples drawn from an unknown distribution Px,y over X' y. Using samples from the dataset D, the objective is to find a predictor f : X + y' among a class of functions F to predict the target y E y of a new sample X. In particular, given loss function É: y x y R. the objective is to minimize the expected risk (or loss) defined, for all / E F, as:
R(f) = B(m Px.s [(f((z),))
(1)
Typically, we choose the mean- -uan-d loss (f(x)) y) = (f(x) y)² for regression problems or the zero-one loss {(f(x) y) f(m)ay for classification problems. We denote the optimal predictor
3

---

by f= arg sen R(f). Since the distribution Px, is unknown in practice, we instead try to minimize an empirical version of the expected risk based on the dataset D == (1)
Remp (f) = T2 Σ e (f(xr))
Distrii-iiiiittinnnn Detection. In machine learning problems, we usually assume that the test data distribution is similar to the training data distribution (the wooedd-oord assumption). As this is not the case in real-world applications, the Out-of- of-Distribution (OOD) detection (Yang et al.) (024) aims to flag inputs that significantly deviate from the training data to prevent unreliable predictions. the following, we denote by Xxy a distribution over X' ) that differs from the training distribution Px,y'. As proposed in Yang et al. (024)) we consider OOD that involves semantic shift and represent concepts or labels not seen during training. A popular class of OOD detection techniques relies on the definition of a scoring function s(- ff = f(),, which uses the probability predictions of the classifier f(.) as scores to flag an instance æ as OOD when the score probability prediction is below a certain threshold X. For example, common approach is to use the Maximum Softmax Probability (MSP) (Hendrycks & Gimpel| (017)) that returns the higher softmax probabilities of the predictor f-)) as scoring function to measure the prediction confidence.
4 Insights on the Double Descent for the Binary Classification in Gaussian Covariate Model
In this section, we introduce the expected OOD risk metric and we present our main theoretical results on binary eqast-squares classifiers applied to Gaussian data. We assume that C Rd and Y' := [o. 1].
Let IR be a mapping, we denote by Fa == { V + d(x w) w Rd] the class of functions considered in this study.
4.1 System Model
Gaussian Covariate Model & Binary Classification. Let w" E Rd and let f
p(x w*)
be an optimal binary classifier. We assume have training dataset D == { v:) j=1 samples drawn from a Gaussian covariate model, i.e., from a distribution Px,y over X x y: where 2; ~ (Oa, Ia) and y/: = x(2;) = ő(x² wt + G)) is the noisy response of æ; with respect to (') in which G (0, o²) is noise capturing approximation errors logits 2; with
of i.i.d
0. In binary classification problems, the objective is find a classifier f() E Fa that fits f-)-) using the training dataset D. Although simple, the Gaussian covariate model is also considered in the double descent
literature to provide theoretical insights (Belkin al. 2020, Mei & Montanari 2022).
Leaat-Squares Binary Classifiers. In this section, we consider least-squares binary classifiers to approximate logits of the optimal binary classifier f*)... Specifically, we define the squares binary classifier f x + "(x" w) in which v is obtained by analytically solving:
i arg min Remp(w) 1
Σ (x w =))² 1 | 2112:
were X x x..... Fn T gaxd is the data matrix containing the n samples x; E Rd and = [ZI,..., ) E R is the target vector of noisy logits. We consider a particular subset of least-squares binary classifier fr E Fa that uses subset T C [d] of features that fits coefficients r E (T)
Φt X+2 and DT 0a-
To measure the ability of binary classifiers f(-) E Fa to provide prediction confidence on samples drawn from both the training distribution Px.y and the OOD distribution pooD,
nattof--Distibbution Risk.
X.y we introduce an expected OOD risk function similar to the expected risk defined 6(x T OOD)
in equation 3 Let fOOD,
E Fa be optimal classifier such that food (x) is

---

close to 0.5 when the sample a is more likely drawn from the X. and close to f*(x) when æ is more likely drawn from Px,y. We define the expected OOD risk ROOD Fa - IR as:
(f) B(a,))xxx.> [(f(z) - foom (x)) (z))²² + (e [ff(x) - foom (x)) (5)
which depicts the expected risk of the binary classifier f()) on the loss function & (Ú. y) + (0 y)² and distributions yx,y and X.y'
Remark 4.1. A low value for RooD(f) indicates two aspects: (i) the classifier f(-) is confident on predictions over the distribution Px,y, which corresponds to low ((..))-xx.)/ [(((a) food (a:)) ] and/or (ai) f(-) is not confident on predictions over the distribution x which is reflected by low (- [(f(z) food (z)) In particular, RooD(f) is minimized when logits are maximally confident on ID samples (only one logit is non-zero) and uniformly distributed on OOD samples.
Remark 4.2. Note that the expected OOD risk defined in equation 5 can be extended to multi-class
classifiers f(.) using the softmax function with
ROOD (f) = P
I + (a.)) (1))))))))))))))))>>)))))))))))) (2=) 2 -
(a) lloc) 2].
where fOoD(x)) (x) is close to 1/C' when the sample is more likely drawn from the x.y C depicts the number of classes, and c| denotes the infinity norm.
In order to use the Random Matrix Theory, we make the following assumptions.
Assumption 4.1. The activation function <(-) is strictly monotonically increasing. Furthermore, its derivative o' (-) is strictly positive and bounded.
Remark 4.3. This assumption holds for many of the activation functions traditionally considered in neural networks, such as sigmoid functions.
Assumption 4.2. Let E, 20DD E Radd defined as
(
( " oan) 2 xxx" and 5OOD = E Bm. )m 2 "0"""" worD
(
(olu" -dl"""" 000) 2 T w-= worD
Σ = )
xx
We assume that
arg min a² Ea agRd a'a
& (T) and
arg min ack
a² 1000 a a'a
g RT)
4.2 Out-Or-Distribution Risk
Leveraging the Random Matrix Theory and extending the Theorem in Belkin (2020), can derive bounds for the expected risk (Appendix A. and for the expected OOD risk (proof in Appendix A.2) of the subset of classifiers defined with equation
Theorem 1. Let (p.q) [d] such that p + q = d, T [d] with IT| p an arbitrary subset of the d first natural integers, and T := [d]
\ T its complement set. Let r e (T) such that wT X E Re and wT: 0,+ E 189 Under Assumptions 4.1 4.2 the expected OOD risk on the predictor fr: + o0x" T ) satisfies
cc(n,)) s Ex Roon S C c(n,p),
where c, C > and
|| + 02) + aad
f
n-p-I
c(n,p) =
400
in
)) 1m900112 -ー (100000 ||2 + 02) + 100
(6)
5

---

Remark 4,4, While our theoretical insights are inspired by Theorem in Belkin et al. (2020), we extend the framework to classification and OOD data. This extension non-trivial, as the original result solely focuses on classical supervised regression. Notably, Theorem derives inequalities for the expected OOD risk unlike the equalities found in [elkin et al. (2019) original formulation.
Remark 4.5. We find Ex [ooD(f)] oo around p = n. which is characteristic of a double descent phenomenon. This result suggests that OOD scoring functions based on the prediction confidence of binary classifiers f(-) exhibit a double descent phenomenon similar to what has been reported for the expected risk (see Appendix (A.I). Like the expected risk in the double descent literature (Mei & Montanari 2022; Louart
2018| Liao et al. 2020, Bach 2024). this result identifies the ratio p/n n as the complexity of a linear model to describe an under- (p/n < 1) and an
over- (p/n > 1) parameterized regimes for the expected OOD risk with a phase transition around p/n = characterized by a peak.
5 Empirical Evidence: Double Descent in Practice
In this section, we provide an empirical evaluation of different OOD detection methods with respect to the model width across multiple neural network architectures.
5.1 Toy Example: : Gaussian Covariate Model & OOD Detection
We investigate the double descent phenomenon in both generalization and OOD detection using the Gaussian covariate model introduced in Section 4.1 The data is generated from a binary Gaussian Mixture Model (GMM), as illustrated in Figure 1 Our model employs Random ReLU Features (RRF): inputs are mapped to a high- nigh-diiennsioal space via random projections followed by a ReLU activation, and then classified using linear binary classifier. The OOD samples are drawn from distinct Gaussian distribution, as described in Section 4.1 By varying the model width, we observe that both the in-distribution Jean-Squared Error (MSE) and the expected OOD risk exhibit double descent behavior, peaking near the interpolation threshold (Figure []. These results show a concrete empirical manifestation of the theoretical Gaussian setup. Full details are in Appendix D.I
5.2 Setup of the real case
General Setup. We perform experiments on multiple DNN architectures: ResNet-18 (He et al. 2016), ResNet-34 (Appendix D.4), a 4-bockk convolutional neural network (CNN), Vision Transform- ers (ViTs) (Dosovitskiy et al. 2021 and Swin Transformers (Liu et al.) 2021b).
Model Setup. To replicate double descent, we follow the experimental setup from Nakkiran et al. (202T), which uses R8sNet-18 as the baseline architecture. We apply a similar setup to the 4-block CNN model, VITs and Swin. We vary the model capacity by altering the hidden dimension (denoted as k) per layer, with values ranging from A = to k = 128. ResNet-18 18, which uses a hidden dimension of 64 channels, operates within the overparameterized regime. The depth of the models is kept constant to isolate the effects of width (effective model complexity). The convolutional models are trained using the cross-entropy loss function, with learning rate of 10 4 and the Adam optimizer for 4 000 epochs. This extended training regime ensures that models converge for all explored model widths. Moreover, each experiment conducted five times (with different random seeds). Further details on the experimental setup for the Transformers are given in the Appendix B.2
Label Noise. To observe the double descent effect, we introduce label noise into the training set by randomly swapping 20% of the labels. This setup simulates real-world scenarios, in which noisy data common. The models are trained on this noisy dataset but evaluated on clean test set. Random data augmentations, including random cropping and horizontal flipping, are applied during training. Experiments on the noiseless case are presented in D.6lannd OOD risk curves are presented in D.5
5.3 Evaluation Metrics
We evaluate both generalization and OOD detection using multiple metrics:
Generalization: We report the test accuracy for in-distribution (ID) classification tasks.

---

We measure OOD detection performance using the area under the receiver operating characteristic curve (AUC), which is threshold freee and widely adopted in OOD detection research. A higher AUC indicates better performance.
OOD Detection:
Neural Collapse: We report NC metrics, which, as noted in Ammar et al. (222)); Haas et al.
) Zhao & Cao (2023) are associated with certain aspects of OOD detection.
5.4 OOD Datasets
For OOD detection, we evaluate each model using six well-established OOD benchmark datasets: Textures (Cimpoi al. 2014). Places365 (Zhou et al.) (017). iNaturalist (Van Hom al. 2018), 100 image subset from (Huang & Li (2021), ImageNet-O (Hendrycks et al. (202] and SUN (Xiao al., 20T0). For experiments where CIFAR-10 (or CIFAR-I00) is the in-distribution dataset, we also include CIFAR-101 100 (or CIFAR-10) as an additional OOD benchmark. CIFAR-10/100 contains 50 000 training images and 10000 test images.
5.5 OOD Detection Methods
In order to have discussion that generalizes across different OOD detection methods, we evaluate several state-of-the-at methods, categorized by the information they rely on:
Logit-based methods: Maximum Softmax Probability (MSP) (Hendrycks & Gimpel 2017). Energy scores (Liu et al. 2020). React (Sun et al. 2021). MaxLogit and KL-
Matching (Hendrycks et al. 2022).
Feature-based methods: Mahalanobis (Lee et al.
2018b and Residual (Wang et al. 2022).
Hybrid methods: ViM (Wang et al. 2022). ASH Djurisic et al. 2023 and NECO
Ammaa
et al. 2024).
Although the double descent effect is observed in all of our experiments, results from only few rep- resentative methods are presented the main paper. Additional results can found in Appendix [
5.6 OOD Detection and Double Descent
Double Descent & OOD Detection. The primary question addressed in this section is whether the double descent phenomenon extends to OOD detection, as suggested by our theoretical insights. We conduct experiments on CIFAR-10 10 and EIFAR-100 ID datasets, and assess OOD detection as a function of model widths. In particular, Figure 2
depicts the evolution of generalization error and OOD detection performance (AUC) for challenging covariate shift scenario between CIFAR-10 10 and CIFAR-100 Refer to Appendix D for more results on multiple OOD datasets. Figure 2 shows double descent phenomenon in all models, with logit-based and hybrid OOD detection methods exhibiting a similar behavior. Experiments evaluating the OOD risk derived from Theorem are provided in Appendix D.5 As shown in the appendix, the OOD risk function also exhibits a double descent phenomenon. This alignment between our theoretical insights and experimental results further reinforces the validity of our analysis.
Feature-Base Based Techniques & Interpolation Threshold. In some cases, no double descent curve observed for feature-based techniques. This result suggests that the double descent depends either on the used architecture or the data, as discussed in Appendix E.3 Furthermore, we observe that the interpolation threshold
always perfectly consistent across OOD datasets or techniques. Those observations are consistent with the Nakkiran et al. (2021)'s 's results on the CIFAR- 10 and CIFAR- 100 datasets. Those results suggest that the effective model complexity (EMC) framework (Nakkiran et al. 202T) defined for the generalization eттоr can be extended to OOD detection.
Smaller Models for OOD Detection. Interestingly, in many cases, smaller models are very good OOD detectors. This is particularly relevant for applications with limited computational resources. While techniques such as pruning (Frankle Carbin 2019) and quantization (Gholami et al. 2022) can also be used to reduce model size, our study focuses specifically on the impact of model complexity. We believe this behavior may be due to smaller models utilizing their parameters more efficiently. This raises the question of when a lower-complexity model might be more ad advantageous
7

---

than deep model for OOD detection. The conditions under which this choice becomes optimal will be discussed in Section 5.77
CHMieourecy 4MR0ccn ridtw Amsitee Mecemattyy
Restle -2 Accumey & G.A220 Withhads Ampllte Uncemanyy
Undon
30
72
Hotakrab
50
Nare endor Meiding
Suin Artuncy JUBOC or Withow Anp Midt Unvorraiity
75
paramettoo Rag
Dri
Accussy
Accuracy
32
Notalanoti
100
200
475
100
700
40
Figure 2: Accuracy and AUC OOD detection metric versus model width. Experiments performed on CNN, ResNet-18, ViT and Swin architectures with CIFARIO and CIFAR as ID and OOD datasets.
Discussion on OOD Methods. OOD detection depends on two key factors: (i) the quality of learned representations, and (ii) the reliability of the confidence score. Logit-based methods rely mainly on model output logits, which are sensitive model size and complexity (Hendrycks & Gimpel. 2017, Liang et al. 2018), making them closely aligned with the double descent behavior. These methods often show smoother double descent trends. In contrast, reature-based methods depend on how well the model separates ID and OOD data in latent space, which may not always be affected by double descent in the same way.
Discussion on Architectures. Architectural biases significantly affect representation quality and OOD performance (Hein et al. 2019 Tripuraneni et al. 2021a). While all models show double descent, their performance differs: ResNet- 18 and Swin improve in the overparameterized regime, CNN performs similarly across regimes, and ViT suffers from poor generalization when overparame- terized. To better understand this, we analyze the latent representations using the Neural Collapse framework (Papyan al. 2020p.
5.7
Overparameterriztton vs. Underparametterizaton through the Lens
of Neural Collapse
From Double Descent to Neural Collapse. We analyze how representation geometry affects the double descent phenomenon in OOD detection. While increased model complexity often improves
performance beyond the interpolation threshold, this trend is not universal (Fig. 2 Some models perform better in the underparameterrize regime. To explain this, use the Neural Collapse (NC)
framework (Papyan et al.) 2020 Ming et al., 22033] Haas et al. 2023 Ammar et al.) 2024). which describes how deep models' final-layer features converge structured configuration where samples collapse around their respective class centroids.
NC1 Metrie for Analyzing Overparanettrizaaion. We use the NC1 metric to measure the C where Σw and Σg are the intra- and
clustering quality of representations: NC1 = Tr Ew inter-class covariances, and C the number of classes. Lower NC1 values indicate better class separation. To quantify the effect of overparameterization we compute the ratio:
NCIu/o NOI,
NCL.
8

---

where NC1, and NC1, are the NCI values in the under- and overparameterized regimes optimums, respectively. Higher NCI..o implies better separation with increased capacity.
Empirical Insights. As shown in Table D NC aligns with OOD performance trends. Overpa- rameterized models with improved N/ > typically detect OOD better the overparamatrized regime. Except the CNN architecture, most architectures seem to show improved performance in the overparamatrized regime. These findings suggest that representation collapse, as captured by NC1, might indicate OOD detection quality in complex models. Further studies on broader datasets are needed generalize this insight.
Models performance in terms of AUC in the underparametrized local minima (AUC.,) and the overparrmettrized maximum width (AUC,), w.F.t NC1,/o value. Best is highlighted in green when AUC, is higher, red when AUC., is higher and blue if both AUC are within standard deviation range. The highest AUC value par-dataset and per arehttehiee is highlighted in bold.
Table 1:
Model
NC1
Method
SUN
Places365
CIFAR-100
AUC,
AUC,
AUC,
AUC,
AUC,, 7
AUC,
Softmax score
76.09- 0.9%
75.08+0.75
75.95-00.6
74.59=0.45
74.33+00.22
72.68=0.31
MaxLogit
72.98-2.22
60.13310.35
73.3342.17
61.25:0.12
73.3810.39
70.37=0.34
Energy
68.08-3.66
59.7310.33
69.0011.36
60.90-00433
70.78.10.33
70.24:00.35
CNN
0.88
Energy+RoAct
4.73
47.49+0.58
60.79+4.5
49.48-0.52
66.00+1.58 58
63.57-0.51
NECO
70.43-2.53
64.224-1.36
71,20++2.99
63.59-0.98
72.40+0095
70.17=0.76
VIM
59.94=2.01
59.77:00.66
61888:188
6093=0040
69.2344078
70.25=:0.35
ASH-P
68.60=3.59
60.36120.37
69.35:13.00
61.4820.43
71.1110.53
70.45=0.41
Softmax
71.18-0.99
75 75.82440.89
71.22-+0.9%
75.52+0.88
71.21+048
75.37-0.42
MaxLogit
70.64=1.53
72.51=+1008 1.33
70.69:1 29
72.64=0.94 94
69, 69.76200.99
73.65=00.5
Energy
69.11=2.49
72.46::1.03 L03
69. 69.1911.08
72.59.0.944
67.58.10.46
73.61:0.39
ResNet
1.96
Energy+ReAks
69.57-2.35
71.83--0.88
69.63++1.95
71.97=0.78
67.25+0.91
72.63-0045
NECO
70.39-2.30
75.60::1.56
70.46+
75.20=1144 1,43
69.9 69.92:0.36
75.28=00.50
VIM
66.51=1199
74.44.0.65
73.12=0.65
61.6110.47
71.51=0.44
ASH-P
69.11=2.49
71.73
29.19:22.08
0..85
67.58
72.89:00355
Softmax score
58.82=2.95
67.91:20.99
29
67 67.66200.99 59
61.89
65.44:0.6
MaxLogit
59.91=3.11
70.75510.45
59.8411340
70.46.00466
61.79
66.95 95..0.55
Energy
60.12=3.68
70.792:0.12
58.77:3.98
70.50:0:44
S5,5212.10 10 66 06.52
1.70
Energy+
60.16-3.79
11.1550.44
48.854403
70.83:10.48
55.5342.
67.27-0052
NECO
64.26-3.20
73.29+0075
6388+3337
72.38-0.66
62.624-2.10 10 68.13-00.76
VIM
60.68=2.61
19.6920.19
58.342260
71.39=0.15
55, 07
68 68.89:0.551
ASH-P
59.49=4.21
70.7410.44
48.1511.42
70.42:10.40
55. 10
66.8
Softmax score
66128-0.19
64.87:00277
66,26+00.6 36
64.61=0.26
65. 18+038 38
62, 12.96=00.35
MasLogit
66.09=1.48
70.301:0.16
66.13:1.50
69.79:20226
61.6010335
66.69-0.39
Energy
64.79-2.81
70.50:10.48
64.8612.65
69.98:10.26
63.0810.4
66.79-1038
VIT
2.32
Energy+ReAc
64.51-2.99
70.51+0.49
64.65+2755
09.77-0.26
62.86+0558
66.78-039
NECO
67.61=1611
75.89:0.47
6..47:1668
74.29=0.29
66. 04.5500554
67.40
VIM
63.14=3.54
72.25=0.37
63.30:3.66
71.41=0.15
64.1+0065
66.34:0030 30
ASH-P
64.79_2.81
70.2710.50
64.8622.65
69.79.0.225
63.08.0.44
66.61:0.366
Empirical Evidence of Depth- Wise Double Descent. We also investigate the influence of model depth and assess OOD detection and accuracy as functions of model depth with fixed width in Appendix F Our results highlight similar double descent patterns in both ID generalization and post-hoe OOD detection. To the best of our knowledge, these are the first results illustrating a depth-wise double descent.
6 Conclusion
In this paper, we presented empirical and theoretical insight that the double descent phenomenon also impacts OOD detection across architectures. Using Random Matrix Theory we showed that both prediction and OOD risks peak at the interpolation threshold for least-squares classifiers on Gaussian data. We have studied the double descent phenomenon across various architectures and OOD detection algorithms. Additionally, we provided insights into why the overparameterized regime can perform better or worse than the underparameterizee one. These observations emphasize the critical role of latent representations in OOD detection under different levels of model complexity. While our theoretical analysis relies on simplified settings (e.g., linear models and Gaussian inputs), future works may investigate more complex systems and include additional factors such as training dynamics, or regularization.
Limitations: Finally, our study is limited to post-hoe oui-o--distribution (OOD) detection; exploring alternative OOD detection paradigms could be a valuable direction for future research.

---

References
Ammar, M. B., Belkhir, N., Popescu, S., Manzanera, A., and Franchi, G. NECO: NEural Col- lapse Based Out-of-distribation detection. In The Twelfth International Conference on Learning Representations, 2024. URL
Bach, F. High-dimensional analysis of double descent for linear regression with random projections.
SIAM Journal on Mathematicc of Data Science, 6(1):26-50, 2024. doi: 10. N0.1137723M15558711.
URL pottpa://doe org/10. 1137/3MM5587799
Bartlett, P. L.., Long, P. M., Lugosi, G., and Tsigler, A. Benign overfitting in linear regression. Proceed- ings of the National Academy of Sciences, 117748):30063-300070 April 2020. ISSN 1091-6490. doi: 11t.33/ma..1003781111111 URL http:://x..oooo oog.:/oo....r.0.00 1073/ pnas. 1907378117
Belkin, M., Ma. S., and Mandal, S. To understand deep learning we need to understand kernel learning. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learming Research, pp. 541-549. PMLR, 10-15 Jul 2018. URLp
mlr press/ ppwss//w0/bellilee www.///eeiinaa..... html
Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learning practice and the classical variancee trade-off. Proceedings of the National Academy of Sciences,
116(32):5849-11588 15854, July 2019. ISSN 1091-6490. doi: 10.1073/pnas.
00073//ma..900370111111 1903070116. URL
http://dx dow. org. 10. 1073/ pnas 1903070116
Belkin, M., Hsu, D., and Xu, J. Two Models of Double Descent for Weak Features. SIAM Journal on Mathematics of Data Science, 2(4): January 2020. ISSN 2577-0187, doi: 10.1137/ 20m1336072. URL http://dx.... doi 01T/100.1137/221133377
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Percira, F., and Vaughan, J. W. A theory of learning from different domains. Machine learning, 79:151-175, 2010. URL https://do/ org/ 10. 000000009999900000 00900009000000 5152-4
Bendale, A. and Boult, T. E. Towards open set deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Bengio, Y. Learning deep architectures for ai. Found. Trends Mach. Leam 2(): January 2009. ISSN 1935-8237. doi: 10.1561/22000000066 / http::ttt///ee //ooi. org/ 10. 1001111122200 1051///0001010/00000000
Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, M. K. Occam's razor. Information
Processing Letters, 24(6):377-380 1987. ISSN 0020-0190. 0020-0190(87)901 14-1. URL, U...ntttps///ws..ceeeeeeeertt ccm////ieeeit//artticl///// 0020019087900147)
doi: https://oi.orr//10.n 1016/
Breiman, L. and Freedman, D. How many variables should be entered in regression equa- tion? Journal of the American Statistical Association, comp:///////////////////// 1080/ 1080/01621459. 10477941. 01621459. 1983 104779411
1983. doi: 10.
Brellmann, D.. Berthier, E.,, Filliat, D.,, and Frehse. G. On double descent in reinforcement learn- ing with LSTD and random features. In The Twelfth International Conference on Learning Representations, 2024. URL htttpp:://penrrerieeeeeeee//aees
Canatar, A., Bordelon, B., and Pehlevan, Spectral bias and task-model alignment explain general- ization in kernel regression and infinitely wide neural networks. Nature communications, 12(1): 2914, 2021. URL httphhtt:::... :/doi.. org. 100 1038/ 0001100381447777 0211://24-777-2122000 021-23103-111111
Cherkassky, V. and Lee, E. H. To understand double descent, we need to understand vo theory. Neural Networks, 169:242-256, 2024. ISSN 0893-6080. doi:
10.04,
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
10

---

Couillet, R. and Liao, Z. Random matrix methods for machine learning. Cambridge University Press, 2022.
D' AAscoli, S., Refinetti, M., Biroli, G., and Krzakala, F. Double trouble in double descent: Bias and variance(s) in the lazy regime. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2280-2290. PMIR, 13-18 Jul 2020. URL https://proceedee html
.ol...mmr.... press/
Deng, Z., Kammoun, A., and Thrampoulidis, C. A model of double descent for high-dimensional binary linear classification. Information and Inference: A Journal of the IMA, 11(2):435 2022.
Dereziński, M.. Liang, F., and Mahoney, M. W. Exact expressions for double descent and implicit regularization via surrogate random design. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829555
DeVries, T. and Taylor, G. W. Learning confidence for -fitt--distriuttinn detection in neural networks, 2018. URI.
Djurisic, A., Bozanic, N., Ashok, A., and Liu, R. Extremely Simple Activation Shaping for Out-of- Distribution Detection. In The Eleventh International Conference on Learning Representations. 2023. URL httpl:tt/ppe:neeeeeeeee
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner. T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL
Drummond, N. and Shearer, R. The open world assumption. In eSI workshop: the closed world of databases meets the open world of the semantic web, volume 15, pp. 1, 2006.
Fang, Z., Li, Y., Liu, F., Han, B., and Lu, J. On the learnability of out-of-distribution detection. Journal of Machine Learning Research, 25(84): 1-83, 2024. URL htt.:tttpjmmm..... .rgg / v25/23- 22257 html
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:///penreevier
net/forum?id-rJ1. b3RcF7
Geman, S., Bienenstock, E., and Doursat, R. Neural Networks and the Bias/Variance Dilemma.
Neural Computation, 4:1-58, 01 1992. doi: 00.1166//ecc.1992...
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and HaIl/CRC, 2022.
Haas, J., Yolland, W., and Rabus, B. T. Linking Neural Collapse and L2 Normalization with Improved nut-of-Distribution Detection in Deep Neural Networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
Hao, Y. and Zhang, T. The Surprising Harmfulness of Benign Overfitting for Adversarial Robustness, 2024. URL https: //atxss///rivvv hatg/abs/24000 12236
Hao, Y., Lin, Y., Zou, D., and Zhang. T. On the Benefits of Oveer-paaamtterizatio for Out-of- Distribution Generalization, 2024. URL
abs//////345 2203. 17592
Hastie, T. The elements of statistical learning: data mining, inference, and prediction, 2009.
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016. doi: 10.1109/CVPR.20169990
11

---

Hein, M., Andriushchenko, M., and Bitterwolf, J. Why relu networks yield high-confidence pre- dictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEF/CVE conference on computer vision and patter recognition, pp. 41-50, 2019.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and -iitti--iitiiittionnn examples in neural networks. In International Conference on Learning Representations, 2017. URL https://openrevian httpiviwwwnweeeeeee net/Eorum?id-Hlk/eee
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural Adversarial Examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19262-152711 June 2021.
Hendrycks, D., Basart, S., Mazeika, M., Mostajabi, M., Steinhardt, J., and Song, D. X. Scaling Duttof-Distriuution Detection for Real-Worle Settings. In International Conference on Machine Learning, 2022. URL
Detpa//aaaaaaaaaaoooooooooooooea 227
Huang, R. and Li, Y. MOS: Towards Scaling Out-of-Distributin Detection for Large Semantic Space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8710-8719, June 2021.
Jacot, A.. Simsek, B., Spadaro, F,, Hongler, C., and Gabriel, F. Implicit regularization of random feature models. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4631-4640. PMLR, 13-18 Jul 2020. URL https: htttttttt///ooeeeeee http://pppppc.n.cceepeppoo mlr
html
Kausik, C., Srivastava, K., and Sonthalia, R. Double descent and overfitting under noisy inputs and distribution shift for linear denoisers. Transactions on Machine Learning Research, 2024. ISSN
2835-8856. URL / httprrrreeeetttttoooooooooiiiooo
Kini, G. R. and Thrampoulidis, C. Analytic study of double descent in binary classification: The impact of loss. In 2020 IEEE International Symposium on Information Theory (ISIT), pp. 2527- 2532. IEEE, 2020.
Krizhevsky, A., Sutskever, I,, and Hinton, G. E. Imagenet classification with deep con- volutional neural networks. In Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, URL http::: http:://////rccedinggss newriie.iippp.. cttrrrisscccceeeeeeee wwcpppperr.../ ffr httll/pppppplpppppplaleppepeeee wtt.f///pppee////////////// pdf
Inc., 2012.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436 2015. URL https://doi. org 10. 1038 1038///tttt1155...
Lee, E. H. and Cherkassky, V. VC Theoretical Explanation of Double Descent, 2022. URL
https://arxxs .rg www./b////000 15549
Lee, K., Lee, K., Lee, H., and Shin. J. A Simple Unified Framework for Detecting -iitt---Ditiibtiion Samples and adversarial attacks. In NeurIPS, 2018a.
Lee, K., Lee, K., Lee, H., and Shin, Simple Unified Framework for Detecting Dut-of-Distributioo Samples and Adversarial Attacks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018b. URL http::ttttt::ceeeiiiiiii http://rocceddinn..eeeeerrrppppp ccpppp..rrcppp..r.ppp
.dff
Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017.
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of-distribation image detection in neural networks. International Conference on Learning Representations, 2018. URL https://openreviae www..neeooooumm---Hii.i...
12

---

Liao, Z., Couillet, R., and Mahoney, M. W. A random matrix analysis of random fourier fea- tures: beyond the gaussian kernel, precise phase transition, and the corresponding double descent. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13939-13950. Curran Associates, Inc., 2020. URL htttps://proceet/// neur.gg.......rin ipstiiiss.... cc/ wwwppapr.... f files/ httpp/ppppp/////////// 400f:308228886111100666777666668 Paper. pdf
Liu, F., Liao, Z.., and Suykens, J. Kernel regression in high dimensions: Refined analysis beyond double descent. In Banerjee, A. and Fukumizu, K. (eds.), Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings Machine Learning Research, pp. 649-657. PMLR, 13-15 Apr 2021a. URL Uttptttss//proeeedaae mlr press/v130/Ls hwt.:///1uulbbhth.m
Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based out-of-distribution detection. In Larochelle, H., Ranzato, M., Hadsell, R.. Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 21464-21475. Curran Associates, Inc., 2020. URL htRt::/tt::///rreee httppe://proceeddigs. ptrpps///pproeeediiggeeeerpppppp wwwpsiiggnneuriirnr.rrc.e 155925500cSSeeaa31447///bb003333
pdf
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hicrar- chical vision transformer using shifted windows. In Proceedings the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Louart, C., Liao, Z., and Couillet, R. random matrix approach to neural networks. The Annals of Applied Probability, // www. jstor. . arg/stabble2654285
2018. ISSN 10505164, 21688737. URL https
Mei, S. and Montanari, A. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathemattccs 75(4):667-766, 2022. doi: https://doi..rr/10.. 1ttp/0.0002///a.220000 URL httph:tt////// http:::///onineeibbraaar wwlly..wwee..................ee. com/
doi /oi /t////bs/110000 10tp://...100////10 10t./ 1ttps//00.0002/pa.22220000 22008
Ming, Y., Sun, Y., Dia, O., and Li, Y. How Exploit Hyperspherical Embeddings for Out-of- Distribution Detection? In The Eleventh International Conference on Learning Representations, 2023. htt.ttttt///peeeeeeeet wttpeeiiweeeeeerruniiee. wtt.frrrmmmddaaaaaao.p.....
Nakkiran, P., Kaplun, G., Bansal, Y.. Yang. T.. Barak, B., and Sutskever, I. Deep double de- scent: where bigger models and more data hurt*. Journal of Statistical Mechanics: The- ory and Experiment, 2021(12);14003, dec 2021.
doi: N0.1088/174--54688aaa
URL
102111188/7212250 Attps://doo org/ 10. 1088 1742- 5474222-466///33774
Olmin, A. and Lindsten, F. Towards understanding epoch wise double descent in two-layer linear neural networks, 2024.
httg//bs/24007 .08445
Papyan, V., Han, X. Y., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, September 2020. ISSN 1091-6490. doi: 10. 00033//aa.20155091111 http //dx www.doi..ogggig/ org/ 10. 1073 ppaa.pas...... 2015509117
Rahimi, A. and Recht, B. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Scheirer, W. J., de Rezende Rocha, A., Sapkota, A., and Boult, T. E. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 1777) 2012.
Stephenson, C. and Lee, T. When and how epochwise double descent happens, 2021. URL https://arxiv.. org/ 2bw./ 12006
Sun, Y., Guo, C., and Li, Y. ReAct: Out-of-distributie Detection With Rectified Activations. In
Beygelzimer, A., Dauphin, Y., Liang. P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https: httt:::://ppmmeriiieeeet//frrrr/
13

---

Sun. Y., Ming, Y., Zhu, X., and Li, Y. ditto--distribution detection with deep nearest neighbors. In Chaudhuri, K., Jegelka, S., Song. L., Szepesvari, C., Niu, G., and Sabato. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 20827-20840. PMLR, 17-23 Jul 2022. URI, htt.httss//prrocediing mlr. www.ppres//ee. http:/vv2////uu sun2d. html
Tripuraneni, N., Adlam, B., and Pennington. J. Overparameeterizationn improves robustness to covariate shift in high dimensions. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 13883-13897 Curran Associates, Inc., 2021a. URL httpps: newrips. cc/ papppppe.... fflles//////s pdf
Tripuraneni, N., Adlam, B., and Pennington Overparametterization
oree imppraaiiiiiiiieeeeeeeeee improves robustness to covariate shift in high dimensions. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.. and Vaughan, W. (eds.), Advances Neural Information Processing Systems, volume 34, pp. 13883-13897, Curran Associates, Inc., 2021b. URL https::/pproceeddnngs nww..nenniipp... cc/ wwwpppppp..../e files/ [ater//5...fii7aaaarrr2220666660
Van Horn, G., Mac Aodha, O., Song. Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. 'he inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Wang, H., Li. Z., Feng, L., and Zhang, W. Vim: Out-of-distributio with virtual-logit matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4921-4999999 192114930, June 2022.
Wang, M. and Deng, W. Deep visual domain adaptation: A survey. Neurocomputiing, 312:135- 153, 2018. ISSN 0925-2312. doi: https://doi.org/10010000 // /www
1016/ newcom.2018.05.00.... URL https http:///c....ooocoeocoo/aaaaa///
Xiao, J. Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. SUN database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp- 3485-3492, 2010. http::/ts:://at semanticschollar org/ CorpusID: 1309931
Yang, J., Zhou, K., Li, Y., and Liu, Z. Generalized dfittr--iittiittiion detection: A survey. Inter- national Journal of Computer Vision, (32(12):5635-566) 2024. URL /ttpp::///// http///ii..or//l/// 10 1007/s11265 024 02117-4
Yang, Z., Yu, Y., You, C., Steinhardt, J., and Ma. Y. Rethinking bias-variance trade-off for generaliza- tion of neural networks. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10767-10777. PMLR, 13-18 Jul 2020. URL (RLLhhtps: httpps://proceedingg mlr press/v119/ yang20j html
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3): February 2021. ISSN 0001- 0782. doi: 10. 10.111 145/344677.. URL https::/dooi org/100 1145/3446776
Zhang, J., Chen, Y., Jin, C., Zhu, L., and Gu, Y. Epa: Neural collapse inspired robust out-of- distribution detector. In ICASSP 2024-2024 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6515-6519, IEEE, 2024.
Zhao, Z. and Cao, L. Dual representation learning for out-of-distributiin detection. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL http://t/////eeeeeeeee..e. net//frrmm id=PHAr3q49h6
Zhou, B., Lapedriza, A., Torralba, A., and Oliva, A. Places: An Image Database for Deep Scene Understanding. Journal of Vision, 17(10):296225) 296, 2017.
14

---

Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the Role of Model Complexity
Table of Contents Appendix
Proof of theorem
I6
A.I Proof of Theorem
Proof of Theorem
B Details about Baselines and Architectures Implementations
22
B. Baselines OOD Methods
22
B.2 Experiments Setup
Details about Neural Collapse
2
D Complementary Results on Double Descent and OOD Detection
24
D.I ReLU Random Feature Model Experimental Setting
24
D.2 CIFAR-10 Additional Results
D.3 CIFAR- 100 Results
25
D.4 ResNet-34 Results
25
D.5 Dut-off-istribution Risk Experiments
26
D.6 Experiments Without Label-Noise Results
26
Model Representations and Neural Collapse Analysis
26
E.1 Structure of the Model Representation
26
E.2 Evolution of NC1
26
Mahalanobis and Residual Joint Performance
wepth-wise Double Descent
F.1 Experimental Setup
28
Results
28
F3 Connection to Width-win wise Experiments,
28
15

---

A Proof of Theorems
While theoretical insights are inspired by Theorem 1 in Belkin al. 2020), extend the framework to classification and OOD data.
A.1 Proof for the Expected Risk
Leveraging the Random Matrix Theory and extending the Theorem in Belkin et al. (2020), we derive bounds in Theorem for the expected risk of the subset of classifiers defined in Fa with equation 4 This section is dedicated to the proof of Theorem and follows the same line of arguments of Theorem I in Belkin al. (2020).
Theorem 2. Let (p,4) E [d]² such that p + q d, T [d] with |T| = an arbitrary subset of the d first natural integers, and T \ its complement set. Let io E (T) such that WT x E R2 and uT. 0g IR9. Under Assumptions 4.1 4.2 the expected risk with respect to the loss function f: (0:y) ~ (ÿ - y)² of the predictor fr a + olx" is) satisfies
c cc(n,p) p) s Ex [n([T]] < dn,p),
where c, C and
p
|| 112 + 02) + 1|
ipp n fn-1sps n (8)
h-p-I
c(n,p)
to0
(1 (1-38 })
w 112 + P-R-I | 11 + +2)) + 1 ifp n+2
Remark A.1. Theorem in Belkin et al. (020) constitutes special case of Theorem for の + æ, which corresponds the case where Σ
2
Remark A.2. From Theorem we have Ex [R(f)] so around p n. The expected risk decreases again as p increases beyond n and highlights double descent phenomenon. This result consistent with the literature of double descent (Mei Montanari] 2022; Louart et al., 2018 Liao et al. 2020 Bach 2024), which identifies the ratio p/n as the complexity of a linear model to describe an under- (p/n < and an over- (p/n > 1) parameterized regimes for the expected risk with phase transition around p/n characterized by peak.
Proof. Let a E X and
A w 2 w
We have
fr( -(x w)) 6xxx w)) + xx A A' (c(x, w w')
From the mean- theorem, there exists
c(x, u. w") E (min(æ² " T w"), max(x ". xt w')
such that
6(x" (w" + Aw)) 6(xx w²)) xx A Dw (c(æ, U, w w))).
We have thus
Ex [R(}T)] EN,x (f(xz) (x)))
Ex,x x
(0(x (1)""w) ه(x ))²
(o A Ó'(2z, w w") )) |
x
A o ('(x,, w. w }}
Tr E -Tr(En,x (c(a, u.  "nn" D A [o
Tr Ex Ex 05 (c(x, w. w Tx A AT
Tr Ex Σ m
Ex TA
16

---

where
Σ Eg [o' (c(x,, ; w )²)) xx
ola: 10) - xx"
Ex
x:
m0
From the min-max theorem, we have
E x
Amin
دل w m S Ex A T MAw!
Ex Amax (>) Am)
From Lemma
there exists C, C' independent of ib (and thus X) such that
c Ex
A t0 Aw]
< Ex
A
SCEX A Aw
For Ex
A T
have w/
Ex
Ex
112
Ex
i - w 11
Ex [00= w
= Ex
पK - w 11
+ 1
From Lemma 1.2
we have
|| + 022)
ifpsn-2
n n-p-1
Ex [1 w
400
if i i
m
|| 1|
p-
| 113 0²)
which concludes the proof.
A.2 Proof for the Expected OOD Risk (Theorem
This section is dedicated to the proof of Theorem
Theorem 1. Let (p.q) [d] such that p q d, T [d] with IT| p arbitrary subset of the d first natural integers, and T := [d] ( its complement Let w (T) such that uh x z E R and urr 0g E IRY. Under Assumptions 4.1 4.2) the expected OOD) risk on the predictor fr: + o(x w satisfies
c cc(n,p) p) s Ex RooD (f)l < C c(n,p),
where c, C and
||w + 02) + 1|
ifp n,
n R-p-I
c(n,p) =
t00
i Sp
11000000011
I100 + 0²)
+ 11 OOD 1
2+2
p-n-1
Proof.
The layout of the proof is similar to the proof of Theorem
2 Let a E X and
w 00D
From the mean- mean-value theorem, there exists c(æ, w, w 00D) E (min(x" ". xx" w OOD), max(x= in x² w OOD))
such that
fr(x) = x6xx w) 0(x" (xo 00D + Aw))
6(x 1 00D) + xTT A Aw (c(æ, w w oOD)).
17

---

We have
Ex [Roo
Pxx,x X (f(x)) 1000 (x0) +B x (f(x) - food (z)))
(0(x w) ó(x² w o)0))) /3|
Perp x.y x (0(z J 2) 0( "(=" w ) And (daz w w )OD)))) ?
Ex x
E x
T (d w w )OD )²]
1000
p0op x
(xt
E x
T Wix w (c(a, u w )0
+ E Re pap x
A A Tan' Aw (c(ar, w w )
-Tr(Ex Ex
ExNPx. (c(x,, w ) 2 xxx T
+ BB Bow / (c(xx, w ) 2 xx | A Aw
= (Tr(Ex
[
500D
= =Ex
T EA
+ Ex
A T E
00D
A
where
Σ Ex [ (c((,, w w ) ) xaa'
O0 (0 -00= 0000, 2 xr 10-00- and w5000 xxx
ExwPxxy
and
g0on
E poon p0ng ( (clar, i w )0D)) )² "aa""
di ól "= ammy a0
E pgoD
xx
From the min-max theorem, we have
Ex (T w|
+ Ex T 0OD
2 Ex (Auiiul () + Amiu (OOD))
A Aw
and
Ex A T TEW
+ Ex T OOD
S Ex (Amaxx (2) + Amax (00)))
A w
From Lemma
I.I there exists c, C > O independent of v (and thus X) such that
Ex w)
< Ex A EA m
+ Ex A
OOD Aw] < C ECE
For Ex
A T
we have
w
Ex
A w = Ex
i w
OOD 112
Ex
OOD lw0
w
[
+ Ex
1
wT
OOD
Ex
1
w
10
oon
+ || W 1o0
10
From Lemma 1.2
we have
p n-p-I
||1 [/3 + 0²)
in-
..000 Ex [1] - wT
to0
if
(1 w
1
2 + -ー (100 + 0²)
ifp2n+2
which concludes the proof.
Lemma 1.1. Let Σ, DOoD E R defined
(0(x 66x0 7 000) (
2
Σ = ExnPx
0000
T...
18

---

and
DOOD
E E pion
[bxe" ui) o(n"" 000)
um
xæ
x w
Under Assumption 4.1
4.2 the matrices and DOOD are positive-defnitt. Furthermore, there exists
C1 C1
and C2. C2
independent of ab such that
Amin (
2
and max Σ C1
and
Amiu E 2OOD and
Amax (OODD I Cz.
Proof. From the mean-value theorem, there exists
c(ar, uØ w 00D) E (min(x" is x." T w O0D), max(xx Too i x. w o00) ))
such that
o' (c(x,, w w )0o))
olar" ub) (am""))-dle dla 0000 " wooD
Using c(æ, w, w )0D), we rewrite the matrix E as
E (e.)) lo' (c(x, w w ))D) xx
The matrix E is positive-deffnite Let ud-) denotes the probability density function of Px...
a" Ee From the min-max theorem, we have a'a
Let u = arg min (0a)
Amin
Since the derivative of d(-) is strictly positive, have
Amin
Σ
""Eu Eu u
t. u
E [o' (c(z, w. w oOD)) xx="
o ('(c(x, i w )0D) (u" x;)² (())
a 3d
J InvITT) o' ((((x, ) ("))))((a)dd
Note that for all æ E (T))), we have
w u xT + wFoxTe w 0f 0d- =
We deduce that c is independent of u since
c(x, w w .0D) E (min(0, xx 20 O0D) max(0, xt w )0D), Vo (e
Furthermore, e since æ + ø' (c(cz, w w OOD)) 2 (u (u) is nonnegative and is not the zero map
on Rdd (T) as u & (T) (Assumption 4.2 We conclude there exists independant of v such that Amin Σ) N c.
Next, we are going to show that there exists > independant of i such that Amax ΣΣ) S C. From Assumption 4.1 know there exists K' > 0 such that for all
have
o' (ar K'
xx T theorem M is semi positive-definiee if for all a E Rd (0z), we have
Let K K' and M (..
E. M symmetric matrix. From the min-max
a" Ma 2 0.
19

---

aet | {0d). we have
a Ma - a²' [K'K"Eee ((, [xx") " x) a
= =² B(a,)) [K2xat xxx' ( [ (c(xx, ú wood) xx
(K2 -K2-0'0 (c(2r, w. w wooD) 2)
(a )
>0
2 0.
We deduce that M 2 0. Therefore, E 3 K K Px xxx"
and we have
Amax
Σ) S K2 Xmax (B(a,))Px..
C is independent of w. With similar proof,
derive result for DOOD
Lemma 1.2. Let (p, q) [df² such that p + q d, T S [d] an arbitrary subset of the d first natural
integers, and T == [d] \ T its complement set. Let vb E Rd such that ut = X E IR and
WT = 0, E R29, We have
P
|1
+ +02)
ify n-
n-p-1
Ex 10 wT
00D
to0
Win-15p S n 1,
1w
0OD
2
2
+
p-
(1000001) +0²)
2
Proof. Let
X x woO0D Since रल. X 2, we have uZ x+ Xy Xy W Therefore,
Ex [|| 10 00D -io = Fxx ||| [Ip - x yw OOD x 11
Ex [|((A - X x ww
OOD
2 12 + | 21(11-x woo x
Since (X² = x X and ( T = x we have
(II, - -x 00D x ((I, x+ x
x OOD x
1W0001 /7 (L, x )T x
1W0OD17 7 'x²n 1w 7 (x xx)
|WOOD T "x= 1WoODD x
O.
(4p - ( COD and x 17 are thus orthogonal. We deduce that
.00 27 = Ex [1 X X 2 Ex wT w.
OOD ||2 +
| m
(9)
Ex [1(L, - X 2
OOD
[1-]
Ex
[|| 12
Leveraging the same arguments used by Belkin et al. descent phenomenon in the regression problem, we distinguish two cases depending on n and p to derive equation 9
2020) to prove the existence of the double
Classical Regime (p < n). Breiman & Freedman (1983) studied this regime for the regression problem and found:
Ex
[ - w
O0D
2
p
(1wm
0OD + 02)
n
20

---

Interpolating Regime (p 2 n). The interpolating regime has been considered in Belkin et al. (200)) for the regression problem. We can observe that:
OOD WT (Tp 15-x772 Xy JwyT OOD
uT
and
X x X x OOD (II, xxXx) 7
(I, x w
) D 7 (x
[w0 1 I (X7XT -
OOD
Since (I, -1,,x xixx)uu0f JODD and X X w are orthogonal, have
| 2
2 || x X w
OOD
X
1
2
(Pythagorean theorem)
and thus
|| - x X
w
O0D
W
0OD
x X
00D
Putting the equation above into equation
9 we obtain
Ex 10- 10 [|] Ex
Ex
x OOD
10D11
[|]
Ex
x (10)
x
Note that II, X xx X is the orthogonal projection matrix for the row space of Xy- We thus write X 1 linear combination of rows of Xy. Then, using the fact that the æ; in X are i.i.d. and drawn from standard normal distribution and by rotational symmetry of the standard normal distribution, it follows:
Ex
w 1 =
0OD
Putting equation above in equation
10
into, we obtain
Ex
|
[
1
p Ex
||
X m
We are going to evaluate Ex
X 1
First, we observe that
- 00D
z -
XTw OOD
WT + X 00D
OOD
OOD
(11)
xx.wooo OOD tc,
where =
Because X Yowoo and X wooD are both uncorrelated, we have
x /| [1]] Tr Ex X x Ex (m1
Ex
As p n, we have X
1 and thus
Ex X X Ex (X XF-11
X follows a Wishart distribution: X ~ Wr, (p. In), and (Xx follows an inverse Wishart distribution:
(X+x ~ Wr-t(p. In). Its expectation is given by:
[E((X x - I
We have
Ex X |
Ex (n mm
p-n-1
From equation III we have
Ex [n'n]]
Ex [( X Xy 0OD
(z XowDODD
Ex
(Xy= OOD + +
(Xy=w 200 +
= -lo T [x XxF.Xy0
woo
+ Ex
T Ex
X
w
00D
+
[w
00D
Ex X
Ex
+Ex
[Te)) c).
=0
21

---

Xyo follows a Wishart distribution, i.e., X x X ~ (n Iq).
We obtain thus
Ex [x[n"n) n] = n/ 1z w7: no²
(12)
= n0 _/2 + no²
From equation
12
we deduce that
Ex
x ||x = p-n-1 (110g0
OOD 1 + +0²)
and
Ex
10 - w0 u O00
= 10 11 #) + p-n- |100 00D 112 + 0²)
B Details about Baselines and Architecture Implementations
B.1 Baseline OOD Methods
In this section, we present an overview of the baseline methods used in our experiments. We describe the principles behind these baselines, and the chosen hyperparameters. is worth noting that extensive hyperparameter search for each method were not performed to maintain stability. Hence, once the final model selected, hybrid methods like ViM, ASH and NECO performance may increase if such task is performed.
Softmax Score. scoring function
This score uses the maximum softmax probability (MSP) of the model OOD) (Hendrycks & Gimpel (017).
Energy. Liu et al. (2020) proposes using the energy score for OOD detection, where the energy function maps the logit outputs to a scalar. To maintain the convention that lower scores correspond to in-distribution (ID) data, (Liu al. 2020) uses the negative energy as the OOD score.
propose clipping extreme-valued activations. The original paper found that clipping activations at the 90th percentile of ID data was optimal. Moreover, the authors propose, we employ the ReAct+Energy configuration.
ReAct. Sun et al. 2021
KL.-Matching & MaxL.git.. KI.-Matching computes the class-wise average probability using the entire training dataset. Consistent with the approach outlined in (Hendrycks et al.) (022), this calculation is based on the predicted class rather than the ground-truth labels. MaxLogit employs the maximum logit value of the model as an OOD scoring function.
Mahalanobis. This score leverages the feature vector from the layer preceding the final classifica- tion layer (Lee et al. 2018a). To estimate the precision matrix and the class-wise mean vector, we used the entire training dataset. It's important to note that we incorporated ground-truth labels during this computation process.
VIM & Residual. Wang al. (2022) decomposes the latent space into a principal space and null space P The VIM score is calculated by projecting the features onto the null space to create a virtual logit, which is then combined with the logits using the norm of this projection. To enhance performance, they calibrate this norm with a constant which is determined by dividing the sum of the maximum logits by the sum of the norms of the null space projections, both measured on the training set. The Residual score is derived by computing the norm of the latent vector's projection onto the null space. We followed the author's suggestions for the null space, by setting it to half the size of the full feature vector, adapted to each model width.
ASH. Djurisic et al. (2023) employs activation pruning at the penultimate layer, just before the application of the DNN classifier. This pruning threshold is determined on per-sample basis, eliminating the need for pre-computation of data statistics. The original paper presents three different post-hoc scoring functions, with the only distinction among them being the imputation
22

---

method applied after pruning. We employ ASH-P in our experiments as it performed the best, in which the clipped values are replaced with zeros. As specified in the original paper, we fix the pruning threshold value to 90%.
2024) leverages the geometric properties of Neural Collapse, measuring the relative norm of sample within the subspace defined by the ETF to identify OOD samples. NC typically involves a collapse in the variability of class representations, leading to a more structured and simplified feature space. It is hypothesized that this collapse also impacts OOD detection, particularly through the emerging orthogonality between ID and OOD data. NECO utilizes this orthogonality to effectively distinguish between ID and OOD data by measuring the relative norm of each data point within the approximated ETF space scaled by the maximum logit value as the OOD score. We use dimension to approximate the ETF sub-space for all architectures, with e being the number of classes.
NECO. Ammar et al.
B.2 Experiments Setup
VIT Experimental Setup. For all experiments, we trained a set of ViT models with hidden dimensions [4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 60, 80, 100, 120, 160, 200, 240, 280, 320, 360, 400, 480, 520, 600, 680, 760, 800]. This hidden dimension serve as the dimension of the output layer after the linear transformation (the class-token size). The dimension of the FeedForward layer is the width multiplied by 4. The input size is set to 32 and the patch size to 8, no dropout is used and we use 4 heads with Transformer blocks. The ViT models are first randomly initialized and then trained on CIFAR- 10 using stochastic gradient descent with CE loss. The weights are fine-tuned for 000 steps, with no warm-up steps, (24 batch size, 0.9 momentum, and a learning rate of 0.03.
Swin Experimental Setup. We used standard 4 block Swin architecture. with a downscaling factor of 12,2,2,1) ) for each block respectively. The width ranges from to 100, with a window size of an input size of 32, and a filter size of 4. The model is randomly initialized and then optimized using an Adam optimizer with CE loss for I 000 epoch using a batch size of 1 024. The initial learning rate 0.0001.
CNN Experimental Setup. Similar to Nakkiran al. (2021), we define a standard family CNN models formed by 4 convolutional stages of controlled base width [k: 2k, 4k; 8k), for ls in the range of [1, 128], with a fully connected layer as classifier. The MaxPool set to [2, 2, 2, 4] for the four blocks respectively. For all the convolution layers, the kernel size is set to 3, stride and padding to 1.
C Details about Neural Collapse
For overparametrised model trained through the terminal phase of training (TPT), Neural Collapse (NC) phenomenon emerges, particularly in the penultimate layer and in the linear classifier of DNNs
(aapynn et al. 2020 Ammar et al. 2024). NC is characterized by five main properties:
1. Variability Collapse (NC1): the within-class variation in activations becomes negligible as each activation collapses toward its respective class mean.
2. Convergence to Simplex ETF (NC2): the class-mean vectors converge to having equal lengths, as well as having equal-sized angles between any pair of class means. This configuration corresponds to Simplex Equiangular Tight Frame (ETF).
3. Convergence to Self-Duality (NC3): in the limit of an ideal classifier, the class means and linear classifiers of a neural network converge to each other up rescaling, implying that the decision regions become geometrically similar and that the class means lie at the centers of their respective regions.
4. Simplification to Nearest Class-Center (NC4): The network classifier progressively tends to select the class with the nearest class mean for given activation, typically based on standard Euclidean distance.
5. ID/OOD Orthogonality (NC5): As the training procedure advances, OOD and ID data tend to become increasingly more orthogonal to each other. In other words, the clusters of
23

---

OOD data become more perpendicular to the configuration adopted by ID data (i.e., the Simplex ETF).
These NC properties provide valuable insights into DNNs learned representation structure and properties, which allows for a considerable simplification. Additionally, the convergence of NC can be linked to OOD detection Ammar et al. (2024); Haas et al. (2023); Zhao & Cao (2023); Zhang et al. (2024). For further details refer to (Papynn et al. 20200 Ammar et al. 2024)
D
Complementary Results on Double Descent and OOD Detection
D.1 ReLU Random Feature Model Experimental Setting
ReLU Random Feature (RRF) models (Rahimi & Recht 2007) offer a class of simple yet expressive models. These models approximate two-layer neural networks by fixing the first-layer weights randomly and learning linear output weights. Formally, the model is defined as:
112
RRF(x)
Σ a, dlw; x x+ b(), where <(2) max(0, z),
(13)
with Wi Dw, b, Dj drawn iid. from fixed distributions, and the output weights a, E R learned via linear regression on the transformed data. At their core, these models perform a randomized nonlinear projection followed by linear mapping, offering computationally tractable means of studying nonlinear learning behaviors.
We stack sigmoid activation function on the model outputs, which aligns with the theoretical setting underlying Theorem Jand Theorem
fx)= = d' (RRF(x)), where x'(2) Ite
This allows to closely mirror the binary classification setting.
We vary the model width (i.e., the number of random features) from 2 to 10 000 and train each model for 600 epochs. For training and evaluation, we generate two datasets of 000 samples each by selecting two CIFAR-I0 classes (class 0 and class 1) and fitting Gaussian Mixture Models (GMMs) to this data. These GMMs then serve as generative sources for the training and test samples. Out-of- distribution (OOD) data is generated by sampling 1 000 inputs from a distinct Gaussian distribution with the same shape CIFAR- 10 images.
To evaluate OOD detection, employ the following score, which is tailored to the binary classifica- tion context:
g(x) 9 0.5]
(14)
0.5
where ú denotes the model's prediction. This score reflects our OOD detection risk proxy defined in 5 capturing the model's confidence by penalizing under-confident predictions (a sigmoid value close to 0.5). Interestingly, the AUC drops around 50% at the interpolation threshold, indicating behavior no better than random guessing. To better understand how the OOD score evolves across the double descent curve, Figure 3 shows the distribution of g(x) from equation [4 across different model complexities- pepplexxties-spccfffieie -specfically y in underparameterized, overparameterized, and interpolation-threstt
regimes. At the interpolation threshold, the distribution is notably heavy- tailed, revealing that the model is producing extreme and unjustified confident outputs, often close to 0 for all inputs. This overconfidence, applied indiscriminately both ID and OOD samples, explains the poor AUC for the defined OOD score in equation [4 In contrast, models outside the interpolation region exhibit more informative score distributions: OOD samples tend to receive low scores, while ID scores are more uniformly spread, enabling better separation. However, as model complexity approaches the interpolation threshold from either side (width of value 500 and 5 000), score separability start diminishing, primarily because OOD samples begin receiving increasingly higher confidence scores.
D.2 CIFAR-10 Additional Results
To further show the consistency of double descent for OOD detection, Figures 6 7 and show the OOD detection metrics performance on six more semantic-shift OOD datasets. To illustrate the
24

---

performance of other OOD-methods while maintaining visibility, we show two different methods at each dataset alongside the petter-performin and most stable three: MSP, NECO, and ASH.
D.3 CIFAR-100 Results
In this section, we present results for the CIFAR-I00 dataset as ID for ResNet-18 model. Figure illustrates the OOD detection metrics performance, Figure I0 shows the accuracy and eigenvalues distribution (see section E.1 for discussion about eigenvalues). We can observe similar behaviors between the ResNet-18 trained on CIFAR- 10, and this current configuration on a harder dataset (CIFAR-100).
Table 2 illustrates the evolution of AUC between the underparametrrized and overparametrized regime and its correlation with the NC1 w/o for the remaining OOD datasets. As the CNN's NC1, a/o its performance stagnates or deteriorates with overparametrization, while the other models improve. Additionally, we can see how the hybrid-based methods improve considerably and become competitive with logit-methods when NC'l/o 1.
Table 2: Models performance in terms of AUC in the underparametrized local minima (AUC.,) and the overparametrized maximum width (AUC,), w.r.t NClafo value. Best is highlighted in green when AUC,, is higher, red when AUC., is higher and blue if both AUC are within standard deviation range. The highest AUC value per-dataset and per-architecture highlighted in bold.
ImageNet-OO
Textures
iNaturalist
Model
NClujo
Method
AUC,
AUC, f
AUC, 1
AUC, f
AUC..
AUC, 1
Softmax score
71.60+0.37
0.57
70.76++0.99
0.78
66.19:1.27
72.07
MaxLogit
68.91=0.96
65.94=0.51
62.01
1..5
63.38=1.89
61.58+2.07
Energy
65.06+1.557
65.76 +0.51
53.62+5.87
45.07+1.04
60.11+2.32
61.32+-2.05
CNN
0.88
Energy Energy+ReAct
59.06++2.60
58.08+0.80
42.80+7.54
32.67+00.8
51.17+2.22
49.41+2447
NECO
67.45=1.114 68.89-0.35
56,5443.83
55.7543.63
59.79+1.96
68.5243..0
ViM
67.13+1.61
65.76+0.51
49.84+3.59
45.09=#0.79
51.58+2.82
1.93
ASH-P
66.20:11.48
66.26110.5
56.07:15.44
46.23:1107
60.73-2.83
61.5242.11
Softmax score
70.90==0.52
75.91=0.49
68.03+0.75
72.78 +1114
65.79=2.16
1.39
MaxLogit
69.45-0.88
72.50+0.88
65.06+180
64.43++2.04
63888+3.47
72.82+2.32
Energy
6..3611.22
72.44-0.89
62.2842.55
64.3511.84
61.731497
72.77+22.3
ResNet
1.96
Encrgy+ReAct
68.02=1.441
72.00=0.81
64.15+2.68
66.13=1.200
61.68=6.83
71.85+22.60
NECO
70.42.10.83
76.11111422
67.5611.84
73.18:13.20
64.5013.51
75.12.1..21
ViM
70.94=1.54
75.03-10.62
77.8811666
81.02:1.42
62,5624.91
67.12+1244
ASH-P
67.36=1.26
71.57-20.44
62.28+2.55
62.96-2.111
61.73-4977
71.7242.28
Softmax score
56.62-1..15 64.78.11.48
49.7113.38
63.51-10.33
49.1016.81
60.26:12.08
MaxLogit
55,86=3.995
64.70:12.07
49.4714.18
60.29-10.38
49.22-5.07
58.91+1881
Energy
49.58.15.48
64.56.12.05
49.4916.73
59.46
$1.6117.96
58.73.1..75
Swin
1.70
Energy ReAAt
49.81=5.51
65.431:2.99
50.66116.07
62.38.10.24
518717.51
19.3911183
NECO
57.63=1.114
68.1942.71
56.50=3.64
68.06:10.66
49.09=5.14
62.5811.21
ViM
65.18.11.83
13.45
84.4
78.67.11.65
67.8612.44
63.83.12.63
ASH-P
49.46.15.66
64.48.12.11
48.96.1.885
59,90.10.48
51.33
58.6911.99
Softmax score
64.17-0.64
63.64:0.80
67.73+1.82
70.27-10.36
52.79-0.77
58.2340.51
MaxLogit
63.15.10.88
68.87.10.52
67.22.12.25
79.25
51,90
61.28.10.94
Energy
61,2411.08
69.10-1..00
65.5912.78
79.68.10338
51.1113.04
61.40-1..97
VIT
2.32
+ReAt
11.30++1221
69.09+0..99
65.64++2.66
79.68
51.63+4.61
61.39+0.98
NECO
65.83.11.35
69.95.10.37
69.4112.16
77.32.10.49
52.5012.05
62.86
VIM
68.76.11.52
67.82-0.55
65.5812.69
74.13-10.38
56.4113.81
60.42-10.90
ASH-P
61,24+1.08
68.96+00.48
65.59+2.78
79,39+0.42
51.11+3.04
61.28+0..99
D.4 ResNet-34 Results
In this section, we present the results for the ResNet-34 architecture, deeper version of the ResNet family. Results include both CIFAR-10 and CIFAR-IOM datasets as ID. Figure shows the accuracy curve and Figure depicts the OOD detection metrics performance. We observe a similar curve to that of ResNet-18 for both datasets and for all OOD methods, with slightly higher performances. We highlight that the interpolation threshold occurs at smaller width for ResNet-34 (k=8), compared to
25

---

ResNet- 18 (k=10) for the CIFAR-10 case. This higher complexity also contributes to its lower NC1 values in Figure 22
D.5
Out-of-Distribution Risk Experiments
In Section 4.1 introduced RooD (f) a metric for OOD detection. Figures 13
14
15
16 and 17
show the expected OOD risk as a function of the model width and highlight a similan ar double descent phenomenon across all OOD datasets.
D.6 Experiments Without Label-Noise Results
Like experiments depicted in Nakkiran et al. (2021), our experimental results in Section S consider noisy labels. In this section, we consider a noiseless setting with Figure that depicts the model ac- curacy as function of the model width in a noiseless setup. Instead of a double descent phenomenon, a plateau or stagnation appears near the interpolation threshold. Similarly, Figure 19 features the OOI) detection performance with respect to the model width under this setup. The curves exhibit similar behavior than for the model accuracy. Moreover, we observe that removing label noise makes the learning task easier with better performance, e.g., ResNet- 18 max accuracy rising from 83. 83.40% to 94.48% with a similar rise observed for AUC.
E
Model representations and Neural Collapse Analysis
E.1 Structure of the Model Representation
Convergence towards Neural Collapse can be indicator of improved model representation, as defined by the ETF structure. As such, the model eigenvalues distribution can be seen as describing how much the model representation aligns with this structured manifold. The properties of the ETF implies that the top c eigenvalues will be equally prominent, as they represent the top e eigenvectors that constructs the ETF subspace (Papyan et al. (220)) [Ammar al.] (222)). The remaining eigenvalues should be less influential, as the ID the data lies essentially in the c-dimensional ETF subspace. Figure 20 shows the distribution of each model eigenvalues at the overparametrized regime. It is worth noting that it seems that for all architectures, all overparametrized model widths show similar curves. From Figure 20 we can see that ResNet-18 18 and Swin models follow the expected NC pattern, with with steep drop in importance at the th cigenvalue indicating the limit of the ETF. However, Figure 20 also shows that VIT and CNN exhibit a slowly decaying curve, which indicates a lack of clear separation in the model representation between highly important ID information and noisy features. This lack of global structure in their representation results in both models failing to reliably outperform their underparametrized minima.
This highlights the importance of the ETF structure and NC convergence in enhancing representation stability, which can be useful for improving ID classification and OOD detection tasks.
E.2 Evolution of NC1
In this section, we further analyze the evolution of NC1 and the model's learned representation with overparametrization, further show their correlation. Our analysis will focus primarily on the ResNet and CNN models, due to their similarities. To compare the NC convergence across architectures, we include the overparametrized and underparametrized NCI value of each architecture in Table 3 We will not address the transformer- based models whose performance, especially for generalization, were lower than those of ResNet or CNN. This is because transformers typically require extensive pre-training, particularly for small datasets, and this was not the case for our experiments.
In order to visualize the variability collapse predicted by NCI, Figure 21 shows the last-layer activations for both models at their optimal underparameterizee and overparameterized widths. In ResNet, transitioning to overparameterized models leads to significant improvements in the compactness and separation of ID clusters, as well as enhanced orthogonality with OOD points. In contrast, the CNN model does not show clear improvements in ID compactness or OOD separation, making it difficult to determine which representation is better.
26

---

Figure 22 shows NC1 metric against the model widths for both ResNets and CNN. While both models exhibit a double descent pattern, CNN barely matches its underparameterized metric value, whereas ResNet continuously improves with added complexity. This discrepancy in NC convergence with overparametrization explains why only ResNet benefits from increased complexity, suggesting that without improvement in NC, increasing model complexity provides no benefits for the learned representations.
Table 3: Comparison of NC1 values for different architectures in under- and over wer-parameterized regimes.
Architecture
NCI (Underrparammetrriiee NCI (OOeerpaaarre)))))r)e)
ResNet-18 18
1.35 + 0.04
0.687
ResNet-34
1.29 + 0.02
0.64 = 0.02
CNN Swin
2.37 = 0.08
2.71 = 0.03
35.08 = 3.53
20.65 + 1.88
VIT
29.52 = 2.17
12.71 + 0.74
ResNet (Depth)
1.53 + 0.02
1.57 + 0.09
E.3
Mahalanobis and Residual Joint Performance
We noticed that for all architectures, and on all OOD datasets, Mahalanobis and Residual follow the evolution curve, with usually slightly higher AUC in favor of Mahalanobis. This behavior intriguing, due to the fact that each method relies on different types of information. While Mahalanobis models the ID distribution, i.e., the principal space, Residual relies computing the null space norm, which is orthogonal to the principal space.
We associate this behavior with the noise isolation in each architecture, which is specific to the double descent training paradigm. Indeed, in order for models to be able to perfectly interpolate all the training data and achieve (almost) zero training error, noisy samples must be represented closer to their assigned (noisy) label, rather than to their true label. This will cause the train class clusters (using the true labels) to be less compact and separable, making their high-likelihood region to span almost the entire principal space, in which the ID data is represented. Hence, to separate ID from OOD, learning the Mahalanobis GMM (fitted on the train data) becomes equivalent separating the principal and null space, which the same reasoning behind the Residual score.
This overfitting occurs at the interpolation threshold, which causes the learned distributions by Mahalanobis to be sparse and not robust to OOD data, impeding its improvement as we transition towards overparametrization It is important to note also that both of these methods are usually below, or struggle to surpass the random choice threshold of 0.5 AUC in the overparametrized regime (with the exception of texture dataset on ResNet- 18 case).
Interestingly, both of these methods suffer much less from this behavior under the Transformer based architecture, and even exhibit a double descent curve on most datasets. This can be explained by the fact that even the most overparametrized Transformer variant have a training error higher than 4%. In comparison, convolutional models consistently achieve a training error lower than 0.01% %. Hence, Transformers suffer less from this effect because they have not interpolated the noise the training data perfectly. It is worth noting that interpolating the noise is desirable, as it is necessary for Generalization in this setup (Bartlett et al. 2020). Transformer-based architectures require extensive pre- pre-training to generalize well, especially for small scale dataset, which was not performed in our experiments. This inability of transformers to perfectly interpolate the training data contributes to their lower performance in terms of Generalization in the overparametrized regime, especially in the ViT case.
F Depth-Wise Double Descent
In this section, we investigate how the model depth affects ID generalization and post-hoc OOD detection performance. Most works on the double descent phenomenon focus on varying the network width rather than depth (Belkin et al. 2020: Couillet & Liao| 2022 Bach 2024 Nakkiran et al. 202T). A primary reason stems from challenges modifying network depth within both empirical and theoretical frameworks. From an empirical perspective, increasing depth often introduces
27

---

training instabilities and requires hyperparameter adjustments (e.g., learning rates) that complicate comparative analyses. From a theoretical perspective, studies typically rely on frameworks like Random Matrix Theory (RMT) or statistical physics, which often model networks as linear systems with random features. Increasing the depth also increases nonlinear interactions and hierarchical dependencies, which may limit the applicability of these mathematical tools.
F1 Experimental Setup
We conduct experiments on ResNet-like architectures composed of four residual blocks, with CIFAR- 10 as the ID dataset. The width of the network is fixed (at k 10), while the model complexity is controlled by modifying the model depth. In particular, the model depth defined as the total number of residual layers across all four blocks. Starting with shallow configurations, we incrementally increase the number of layers per block, focusing on deepening of the blocks with the fewest layers first. For instance, the sequence of architectures progresses follows:
1,1,111 + 1,1,1,1 + 2, + 2,2,2,2 + 2, - + 10, 10, 10, 10,
where each tuple denotes the number of residual layers in each block, respectively. In Figure 23] the r-axis depicts the total depth, i.c. the of the four block values. To obtain architectures with depth less than 4, we replace residual blocks with MaxPooling layers to maintain a constant overall width across all models. The training protocol follows the same setup the width-wise experiments (see Section 5.2). Furthermore, the same set of metrics is considered: test accuracy, OOD detection AUC, Neural Collapse, and the OOD risk defined in equation
F.2 Results
Figure 23 depicts similar double descent phenomenon for model depth as observed for model width. Furthermore, Table 4 shows correlation between the NClufo metric and OOD detection performance, which may suggest an alignement between the NC1ufo and OOD performance.
Table 4: Models performance in terms of AUC in the underparametrized local minima (AUC.,) and the overparametrized maximum width (AUC,), w.r.t NC1,/o value. Best is highlighted in green when AUC, is higher, red when AUC., is higher and blue if both AUC are within standard deviation range. The highest AUC value per-dataset and per-architecture is highlighted in bold.
SUN
Places365
CIFAR-100
Model
NClujo
Method
AUC, f
AUC, f
AUC,
AUC,, 1
AUC..
AUC,, 1
Softmax
72.45-+00.9
69.23+22.66
72.47-0.86
69.01+t0.97
71.62+00.65
68.84=1.39
Maxl.gii
72.57++0.72
71.00+2.86
72.57+1.15
70.76+1.93
70.66+0.74
2.
Energy
71.81:10.70
70.98.12.87
71.83.11.24
70.74:1..2
69.43-10.80
70.81
Depth
0.97
ReAct
71.83-11.58
71.56.12.99
71.89.1.83
71.3012.04
70.13-11.32
70.71-2.28
NECO
72.83-10.89
71.244
1.074:1.03
70.77+1.99
70.8511.766
69.54_1.07
VIM
71.86:22.11
70.32+4.91
71.07=1.70
69.46+3.92
71.14=1.311
67.29=1.70
ASH-P
71.78:10.66
70,50.1..29
71.79.1.27
70.1912.40
69.40.10.87
70.34-2.45
F33
Connection to Width-Wiss Experiments.
Although the depth-wise experiments also exhibit double descent phenomenon, it differs in important ways from the width-wise case observed in ResNet- 18 and ResNet-34. Notably, the best models in both the underparameterized and overparameterized regimes achieve comparable performance. This distinct behavior is reflected in the ratio NC1,fo = 0.97 < 1, indicating that feature collapse did not improve with increased depth.
These findings emphasize that out-of-distribution (OOD) performance is influenced not only by architectural biases but also by the specific training dynamics and representational geometry induced by scaling along different dimensions. is important to note that at very large depths, additional complexities such as optimization instabilities and slower convergence emerge.
28

---

000
000
(a) Underparameterized models
2400
210
(b) Interpolation threshold model
0C0
000
Overparameterized models
Distribution of the values of the OOD scoring function g(x) defined in equation [4] evaluated on Random ReLU Feature (RRF) models with varying widths. (a) Score distributions for underparameterizee models with widths, from left to right, of 100 and 500 respectively, (b) Score distribution at the interpolation threshold (width = 000), where performance degrades sharply. (c) Score distributions for overparameterized models with widths, from left to right, of 000 and 10000 respectively.
Figure 3:
Average CNN AUROC Widths with Amplified Uncertainty
Average ResNet AUROC Widths with Amplified Uncertainty parametijoed
80-
75-
Under rog m
Over regime
6.9gimp
paramáziia
regime
70-
AUSOI PAL
60
55-
45
Mahelanobia
0_Wattcing
40
40
60
80
100
120
40
40
08
130
120
Average AUROC Amplified Uncertainty
Average
Uncertainty
60
75 Indwe
paramicriizdd
Duea
petamatiized
Utdee
regitna
saremettionl
rag
70.
70
AINDE (N)
63-
59
40
120
200
300
400
500
800
700
800
40
60
100
Widto
Widths
Figure 4: OOD detection (AUC) metric versus model width. Experiments performed on CNN,
ResNet-18. 18, VIT, and Swin with CIFARIO as ID dataset and ImageNet-O as OOD dataset.
29

---

Average
Uncertaintly
Average ResNet
Amplifidd Uncertainty
parametriond
Oven regime
paramctiized
Bdiinwi
pa paromearadd
Dvee-
regi me
75-
60-
AIROC CM
ASH
Bhi
30-
tmwrgy + Reack
20
40
80
100
1.00
20
40
80
120
Widtte
Widhs
Average VIT AUROC vs Widths with Amplified Uncertainty
Average Swin AUROC vs Widths with Amplifted Uncertainty
Does
soramerizzdd
saramptrizzd
a5
regine
reg megime
60-
AUROC (44
AUROCC Ph
60
50
30
30-
150
700
800
20
69
100
Widit
Widtis
Figure 5: OOD detection (AUC) metric versus model width. Experiments performed CNN, ResNet-18, 18, VIT and Swin with CIFAR1O as ID dataset and Texture as OOD dataset.
Average
Uncertainty
Average (hare
Certaiint
Undar sarametribed rioed
Doers parameMized
parametaued
par paramazion aod
70
AUROC:
NECO ASH
50
Maxi.gg K
Energy
40
20
40
80
100
1.0
40
80
100
120
Witthe
Widhis
Average AUROC vs Widths with Amplified Uncertainty
Average Swin AUROC vs Widths with Ampliiied Uncertainty
para
70
Dues paramenttred
paratwitiied
70tander
Oven par weremaliizzd
60.
3.0
50
48
45
40
40
1.00
200
300
400
500
700
800
35-
100
Widthe
Widths
Figure 6: OOD detection (AUC) metric versus model width. Experiments performed CNN,
ResNet-18 18, ViT, and Swin with CIFARIO as ID dataset and iNaturalist as OOD dataset.
30

---

Amplitied Uncertainty
ResNet
Uncertainty
Over- regime
parametrioed
parsmetrized
nd 1eam
reg me
regima
73
AJRO2
AURIC
AS1i-P
VIN
K1-4Matching mg
40
20
40
60
80
1.00
120
60
40
62
80
100
120
Vidit
Average
Widths Amplified Uncertainty
Average
Amplified Uncertainty
Aarametrizg
paromentized
rogime
r20 mc
70
AiROc 14)
65
60
50
50-
45-
40
150
800
40-
100
Whiths
Widhs
Figure 7: OOD detection (AUC) metric versus model width. Experiments performed on CNN, ResNet-18 18, ViT, and Swin with CIFARIO as ID dataset and SUN as OOD dataset.
Average CNN AUROC vs Widths with Amplified Uncertainty
Average ResNet 18 AUROC Widths with Amplified Uncertainty
80
parametrize rued rogin
paramatiiad rogima
pgpametized
Pegam
paramazz ragima
Ra
AUROC IN
AUROC
NECO
MSP
Mahatonobls
33-
50-
The my
20
20
40
80
100
120
20
40
90
100
120
Average AUROC Ampliiied Uncertainty
Average
Uncertainty
golode: regime
Does regimee
poramerippd
80Ulndee
paramitized
Duee-
70-
AIROCC (h1
60
AUROC 1h
30-
20
120
200
300
405
310
700
800
20
60
100
Widtho
Widhi
Figure 8: OOD detection (AUC) metric versus model width. Experiments performed on CNN, ResNet-18, VIT, and Swin with CIFARIO as ID dataset and places365 as OOD dataset.
31

---

Average CIFAR 10 AUROC vs Widths with Amplified Uncertainty
Averege ImageNet -0 AUROC vs Widths with Ampliiied Uncertainty
parametrued
penumazized rog
parametr
Cvicr par parameza
70-
mgmm
regime
70
BIRGE (50
63
555
NECO
60
NECO
Misi
MSP
50
Mahalanob
120
Textures Widths Amplttee Uncertainty
Inaturalist Wilthss Amplitied Uncertainty
Inden.
#4r3matrass aed
peromezizzd
Over- regime
parametries
regme
MARGE (N)
55-
Bne
Energy
The may- Reacct
Mexi
35
20
40
60
80
100
120
50
20
40
10
150
120
Widts
Average
vs Amplitied Uncertainty
Average Places365 vs Widths Amplitied Uncortainty
parathsr- aed
perumezizze regime
#ypmmer ted
pa ramemrrred
65-
migim
Tegmn
regima
85
AUBOCC (NN
KUNGC
55-
55--
50
NECO ASK
45-
VM
Maholanoble
40
VI
40-
les duel
10, MaMacching
Energy React
35
40
80
100
1.2
35
40
18
120
Widis
Widhss
Figure 9: OOD detection (AUC) metric versus model width. Experiments performed on ResNet-18 with CIFAR 100 as dataset and CIFAR-10, ImageNet-O, Texture, iNaturalist, SUN and places365 OOD dataset.
Average Reshltt-18 Accurary vs Wielte with Amaliited Unterr paramirraad
poromeziizdd mogime
Average explared variance
mgmmm
Tap-c compaeetts (ETF)
0.020
0.035
0.030
20
0.005
0.000
Figervalues (lagscak))
Figure 10: Accuracy versus model width (left) of ResNet-18 model on CIFAR-100. Explained variance versus eigenvalues (right) in the overparametrized regime for ResNet-18 (width 64) on CIFAR-108 100. The black line depicts the 1h eigenvalue.
32

---

Unieerityy
p- righte
RBC0
RECO
fedda
Widias Ampl Sarteerty
Rea AUTOC is Widits Amplifed
Tighte
rights
Roco
Ro02
A04
Fatarraai
Patuaraal
RosPett
56 AUHOC va Widi's in Angd Sorte
ResRit
34 AJROC is Willis at Anglifed cottart
patunatous
rigita
quei
Medogt
Medopt
ReaPett 34 AUROC va WMdt with Anglledd certaatty
Reafect -34 AUROC vs Widias to Anpllied Ureettartt
53
YM
Auc dull
Rasieet -24 AUROC yc Witt with Anpdited
Rastit -24 AIROC ye Wittin th Amplieetireeet
persigrosd
Idlo
Ab-aacingg
KL-uccing
Rest
.64 AUROC Wild11 i Amy Fied tincetaaitt
RosRet 334 AURDC is Will15 with Ampl uicetartty
artamerrosd
Ovep
AECO
AECO
Cherp
Figure 11: OOD detection (AUC) metric versus model width. Experiments performed on ResNet-34 with CIFARIO (left) and CIFAR-100 (right) as ID dataset and CIFAR-100 (right), CIFAR-10 (left), ImageNet-O, O. iNaturalist, places365, SUN and Textures as OOD dataset (from top bottom).
33

---

Average Resilit- 34 Acourccy is Wiiths with Ampliied Unceraa nay soromiiiizzo "Meimm
Averoge Reshett -34 Accuracy us Winths wit Amsiified Uncoriaamn my
po poromezizzd
te remmerttted roginne
paramerriee
regiiee
regime
Eiee
24-
Geniae ACuroy
vistane
Wetts
Figure 12: Accuracy model width. Experiments performed CNN, ResNer-34 with CIFARIO (left) and CIFARI00 (right) as ID datasets.
Average ResNet-
OOD Widths with Amplified Uncertainty
Average ResNet- ODD Widths Amplified Uncertainty
Inde
Overr poromerized
3.50)dde parametrzed
Ovier pa regime
paramdroed
2.85
0.05
0.85
0.84
0.62
0.62
8,000
R,00D
0.78
120
Whtth
Widhs
Figure 13: OOD risk metric versus model width. Experiments performed on ResNet-18 with CIFAR10 as ID dataset and CIFAR100 (left) and ImageNet-O (right) OOD datasets.
Average CNN R,0 vs Widthe with Amplified Uncertaintly
Average CNN R, vs Widths with Amplified Uncertainty
0.92
0.52
Under parmettroed
Over paramettazd
Unde: Aarometiize
Over paramettized
0.00
0.88
V we/
0.84
//
0.82
Figure 14: OOD risk metric versus model width. Experiments performed on CNN with CIFAR10 10 as ID dataset and iNaturalist (left) and Textures (right) as OOD datasets.
34

---

Uncertainty
Average
Amplified Uncertainty
0.97
Under
Oeti- rugime
Unde-
Ovar- pa ragis
parame
650ggm
0.8
0.85
0.64
0.82
R. R_000
R_0OD
0.0
100
0.75
40
120
Widte
Widtss
Figure 15: OOD risk metric versus model width. Experiments performed on ResNet-34 with CIFARIO as ID dataset and iNaturalist (left) and ImageNet-O (right) as OOD datasets.
Avrrage VIT 0,000 Widths with Ampliiied Uncertainty
0.6
VIT R,000 Wi didths Amplified Uncertaintly
Under aafealme
Oues
Under
Ovlen
regime
regime
regi me
0.86
0.85
0.85
0.85
0.82
150
200
300
400
500
500
700
800
0.82
100
200
300
400
500
600
700
600
Figure 16: OOD risk metric versus model width. Experiments performed ViT with CIFAR10 10 as ID dataset and Places365 (left) and ImageNet-O (right) as OOD datasets.
Average Swin OOD vs Widths with Amplitied Uncertainty
Average Swin R. R,OOD vs Widths with Amplitied Uncertainty
R. 8,030
R_pOD
0.69tendr rerizedd
Oyyen par macrized
Dipers (ig) me
Kalime
0.85
0.6
Widih
Widhs
Figure 17: OOD risk metric versus model width. Experiments performed on Swin with CIFAR 10 as ID dataset and SUN (left) and ImageNet-O (right) as OOD datasets.
35

---

CNN
Under ragime
Over regime
Under Segimic
Over
regime
Ver
Vertance
23
120
22
122
Weths
Wethy
Average WT Accuracy
Average Swin Accuracy Wittha
vokiidor Peggini
Oueor
paramentired
784
Regima
60
210
50
20
-20
202
3.0
4013
26
100
Welts
Figure 18: Accuracy versus model width. Experiments performed on CNN (top left), ResNet-18 18 (top right), ViT (bottom left) and Swin (bottom right) with CIFARIO as ID dataset in the noiseless setting.
36

---

Average CNN AUROC Vs Widths
Average CNN AUROC Widtis
Under
OveT
Undoy reginee
Over
90
reg th
reg ITR
Fag
80
20
AUROE [he
AUROC 060
70
70
NECO
NECO
MEP
MSP
VH
50
ASH
Energy
50-
KLaWatch ng
Energy stecc
Ki Ki-Merriing
VIM
40
50
120
40
20
50
60
100
120
Wiaths
Withhs
Average ResNet
AUROC
Average
500 aadmmtaoao
parsmit-as
40
regmm
90
00
AUNOC (b)
60
AUROC (K))
70
60
60
50-
Widhss
Wathe
Average ViT NUROC us Widths
Average VIT AUROC Widths
90
Over-
paramenteed
30
AL 731
70
2,80C 134
70
100
50
40
100
202
305
420
505
6.0
703
B30
40
100
200
300
400
530
500
730
100
AUROC Widths
Average AUROC Widths
rormmtriied legime
Over reginec
metrited
parametriped
Segima
regima
80
80
AuRoc 180
AUROC 190
70
60
50
40
100
Widhs
Witth
Figure 19: OOD detection (AUC) metric versus model width. Experiments performed on CNN, ResNet- 18. VIT and Swin with CIFAR10 as ID dataset and CIFAR-100 (left) and ImageNet-O as (right) OOD datasets in the noiseless setting.
37

---

0.150
Tpc comporents (277
Average accla vatferce
Tip-c comporents 12971
Average acela varfence
Dowered wnieate
9.04
9.320
9.03
9.75
0.00
0.82
0.255
0.00
301
Egstralles log Scake)
C getsalue dog Scako)
0.10-
00.15
0.54
0.23
0.50
0.5
gertvalues Sug Scakd)
Bogg Scale)
Figure 20: Explained variance versus eigenvalues in the overparametrized region for CNN (width 64), ResNet-18 (width 64), VIT (width 400) and Swin (width 50) with CIFAR-10 as ID dataset. Black line depicts the 10th eigenvalue.
CNN withh (trnih/TTuuuerrrrer [undeerparammttriide local minima)
ReaNel18 width 1 [underparametrisse local minima)
000
000
CNN width 128 (Overoarametrtsed optimal model)
BesNet18 with 128 DOverparametrieed optimal model
000
OOD
300
X
-60
Visualization of the last-layer activations on the test set for ResNet and CNN in the underparametrized local minima and the overparametrized width 128 model, with c0far10 as ID and c0far100 OOD datasets. ID points are shown in colors and OOD in black. ResNet underparamterized (top right), ResNet overparametrized (bottom right), CNN underparametrized (top left) CNN overparametrized (bottom left).
Figure 21:
38

---

Double Descent: Average NC1 vs Model Widths with Amplified Uncertainty
BosNet) 34 Intorpotation threshold
ResNet-18 interpolation threshald
Interpolation threshold
201
Avreage MC) Resbat
Vaturne
Aecrage MC2 ONN
Virlance
103
40
100
Witths
Figure 22: NC1 metrics evolution (Log scale) versus model width. ResNet-18 is shown in blue, ResNet-34 in green CNN in red. Dashed lines represent the interpolation thresholds for each model with matching color.
ResHet AL desthe
ResHett AUROC destle
ResWet AUROC vs destle
Oata
agor
AMOCC
PSP
AE00
50
Panuerabl
Panseraba
3510
Dephre
Deptr
Figure 23: OOD detection (AUC) metric versus model depth (total number of residual layers across four blocks), with width fixed 10. Experiments performed ResNet type models, with CIFARIO as ID dataset and CIFARI00 (left), ImageNet-o (middle), and SUN (right) as OOD dataset.
Avitagd Rechetii 030 Depite
Aviigi Anchet OBSTR
Avarage Reshvt & 0:0 Deadit
3.84
380
Figure 24: OOD risk metric versus model depth. Experiments performed ResNet with CIFAR10 as ID dataset and 100AR100 (left), ImageNet-o (middle), and SUN (right) as OOD datasets.
39

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The empirical investigation of double descent occurrence in Post-hoc OOD detection is detailed in Section 5 Theoretical insights explaining this occurrence provided in Section[4] Section 5.7 details the suggested method to identify whether simpler models are better for OOD detection.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
40

---

Answer: [Yes]
Justification: We provide the correct and complete proofs for each theoretical result, which are illustrated in detail in the Appendix A.1 and A.2 The full set of assumptions provided in Section
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Datasets, models, and hyperparameters used are all described in details. See Section 5 and Appendix B.2 Appendix D.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
41

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We will provide the code to reproduce the experimental results in the paper upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
mublic//guides/Codeeuuiiiis wwt.aaammmiioooooooooooooooeeiie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //ips.cc/puut
guides/ wwtp/oeeeiiiiiiinnnooooooooo Pollccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All necessary hyperparameters to understand our results are detailed in Sec- tion and Appendix B.2 Appendix D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We plot the statistical variance across many seeds in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
42

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Due to the nature of our madyyeexplorng double descent and out-of- distribution (OOD) detection across wide range of model architectures, dataset sizes, and OOD detection methods-we trained and evaluated large number of models under varying conditions. Because this variability and the exploratory scope, we did not report exact compute details (e.g.. runtime, memory, or hardware) for each configuration. However, all experiments were conducted using standard GPU-equipped machines, and we aim to release code and configurations support reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research is conducted with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
43

---

Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All assets used in the paper properly eredited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
44

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.paperssttt paperssii thcode. com con/dataset package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
45

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
This paper made no such use of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
46


Reviews:
[{"review_id": "i1hqkl7yRQ", "reviewer": "Reviewer_RJrF", "summary": "This paper presents both a theoretical study of and empirical results for the double descent phenomenon in OOD detection.  Using simplifying assumptions such as a linear classifier, monotonic activations and Gaussian embedding, the authors show that as network caapacity increases, a double descent in loss occurs.  They demonstrate the phenomenon on mutiple architectures with many different OOD detection methods and show that the second descent is sometimes beneficial and sometimes overfits leading to reduced performance.", "questions": "How is one to know what to do in a particular OOD application given this result?  How is the theorem of value or use to practicioners?", "limitations": "It is not clear why the double descent phenomenon is important for OOD detection.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "A strong theoretical underpinning for a useful observation about OOD training.  Very detailed results, and proofs seem to be well developed.\n\nDifficult to follow the proof steps and to understand the value of the main result as it's use is not really discussed clearly. Given the mixed bag of results, some networks benefit from increased capacity and training to the second descent, others don't.", "was_revised": true, "final_justification": "A strong theoretical paper, that allow not very accessible, makes interesting points about double descent for OOD.  Worthwhile.", "rebuttal": "Dear reviewer RJrF,\n\nWe would like to thank you for your interest and feedback regarding our paper.  \n\nWe would like to remind that **overparametrization** is known to benefit generalization (through the double descent phenomenon), but its **effects on OOD detection still remain unclear**.\nIn this work, we investigate how the model complexity affects the effectiveness of *post-hoc* OOD detection methods. *Post-hoc* OOD detection methods operate on top of a pretrained classifier after the model’s training process is finished and leverage internal model properties (probabilities, logits, or features) to define a scoring function measuring the prediction confidence. The objective in this paper is to investigate and better understand the **relationship** between **model complexity** and ***post-hoc* OOD detection methods**.\n\nWe first focus experimentally to highlight that ***post-hoc* OOD detection methods** are **sensitive** to the **model complexity**. In particular, we observed that **logit-based** and **hybrid methods describe a double descent phenomenon**, while **feature-based methods may not** as depicted in Figure 2 and Table 1. Those results suggest that both logit-based and hybrid OOD methods performance improve as the model complexity increases in the overparametrized regime, while we may observe an opposite behavior for feature-based methods. Furthermore, we observed that some *post-hoc* OOD methods may demonstrate stronger performance in the under-parameterized regime than in the overparameterized regime for certain architectures, e.g. as shown by the CNN in Figure 2. Interestingly, we also found that Neural Collapse can be used to provide insights into whether an overparameterized or underparameterized model is more suitable for OOD detection. \n\nIn the **theoretical insights**, we show that the expected OOD risk for least-squares binary classifiers on Gaussian covariate models under some assumptions exhibits an infinite peak when the number of parameters is close to the number of data, which we associate with the double descent phenomenon. This result may suggest that confidence-based *post-hoc* OOD detection methods may exhibit a double descent phenomenon.\n\n**Overall, our study provides practical guidelines for selecting *post-hoc* OOD methods and choosing the appropriate model complexity for effective OOD detection.**"}, {"review_id": "797KZ85JSD", "reviewer": "Reviewer_UoHZ", "summary": "This work shows that the double descent phenomenon reported by Belkin et al. (2020) for regression also arises in out-of-distribution detection. They confirmed this behavior experimentally for multiple model architectures, supervised OOD detection techniques, and datasets.", "questions": "See Weaknesses.", "limitations": "Yes, but the Gaussian constraint of the data $X$ should be mentioned.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The authors validate their theory not only on toy data but also across multiple standard model architectures, benchmark datasets, and detection techniques. \n- Their formulation of Out-of-Distribution Risk is nice.\n\n**Weaknesses**\n\n1. *Gaussian constraint*\n\nThe analysis assumes that $X$ is Gaussian. Is this assumption needed for the expectation identities in Lines 961 - 963 in Appendix?\nFor image data, it implies uncorrelated pixel noise, which differs greatly from realistic images. How might this assumption be relaxed?\n\n2. *Concerns about the distribution of OOD*\n\nThis work extends Belkin’s theory for regression to out-of-distribution detection. Unlike regression, OOD detection cannot ignore the manifold hypothesis [1][2]: in-distribution samples $X \\in R^D$ lie on a manifold of intrinsic dimension $d << D$, and OOD samples are viewed as lying off that manifold. \nHowever, the paper assumes  $X$ is Gaussian in $R^D$,  implying that the in-distribution supports the full ambient space with no manifold structure. In this setting, how is the OOD distribution defined? Can the proposed theory be reconciled with manifold-based arguments? \nMoreover, if $X$ truly lies on a lower-dimensional manifold, then $\\Sigma$ and $\\Sigma^{OOD}$ would be rank-deficient; does that affect the interpretation of Theorem 1?\n\n- [1] Ross, Brendan Leigh, et al. \"A Geometric Framework for Understanding Memorization in Generative Models.\" ICLR 2025.\n- [2] Kamkari, Hamidreza, et al. \"A geometric explanation of the likelihood OOD detection paradox.\" ICML 2024.\n\n3. *The degree of the distribution divergence*\n\nIn OOD detection, the divergence between in-distribution and OOD data distribution is as critical to task difficulty as the training sample size $n$ and model parameter count $p$, yet the proposed model neither defines the OOD distribution nor provides a parameter to control this divergence.\n\n4. *CIFAR-10/100 experimental setup is not clearly described*\n\n (my apologies if they are elsewhere) The paper calls the method post-hoc; does this imply the detector’s classifier is trained solely on in-distribution data? Were OOD samples ever used in training? If so, which ones and in what quantity relative to in-distribution samples? Since the in-distribution to OOD sample ratio likely impacts detection results, is this captured in the proposed theoretical framework?\n\n----\nThe above Points 2 and 3 arise specifically because the theory is applied to OOD detection. \nThe reviewer remains unclear why this work selected OOD detection rather than classification as Belkin’s extension; clarifying this rationale would strengthen the paper.\nThe reviewer rated this submission as a borderline reject, but there is potential to improve its score.", "was_revised": true, "final_justification": "While the Gaussian data-distribution assumption remains a concern, I have no objection to the claim ”comprehensive empirical evaluation across a wide range of architectures”. And A.2 addressed my concern.", "rebuttal": "Dear reviewer UoHZ,\n\nWe would like to thank you for your interest and for appreciating both the novelty and the experimental findings of our study.  \n\nAs highlighted in the review, our paper provides a **comprehensive empirical evaluation** across a **wide range of architectures** (CNNs, ResNets, ViTs, Swin) and ***post-hoc* OOD detection methods**. We want to highlight that the **major contribution** of our study lies in the **empirical observations of a double descent phenomenon in *post-hoc* OOD detection**. To the best of our knowledge, the double descent phenomenon has not been observed before in the OOD context. Our experiments depict how model complexity affects *post-hoc* OOD detection methods and exhibit, for the first time, a double descent phenomenon in OOD detection. Furthermore, we observe that *post-hoc* OOD detection methods may perform less effectively on overparameterized models compared to underparameterized ones. The **theoretical section** in our paper should be seen as **insights** with an illustrative example of a toy distribution, and not as an exhaustive analysis. However, this part depicts a **first attempt to mathematically study** the **double descent phenomenon in *post-hoc* OOD detection** methods based on confidence through a simplified model.\n\n### A.1 Gaussian Constraints & Assumptions on OOD Distribution (Q1+Q2+Q3):\n\nOur theoretical insights show that the expected OOD risk for least-squares binary classifiers on Gaussian covariate models under some assumptions exhibits an **infinite peak** when the **number of parameters is close to the number of data**, which we associate with the double descent phenomenon. \n\nIn this work, we adopt the conventional OOD framework, where we consider **two distinct distributions**: an IN distribution $P_{\\mathcal{X}, \\mathcal{Y}}$ from which training samples are drawn and a separate OOD distribution $P_{\\mathcal{X}, \\mathcal{Y}}^\\text{OOD}$. In Theorem 1, the **influence of the OOD distribution** is encapsulated within the **matrix $\\Sigma^\\text{OOD}$**. Assumptions made in the paper ensure that matrices $\\Sigma^\\text{OOD}$ and $\\Sigma$ are positive-definite, for which we have $\\lambda_{\\min}(\\Sigma)>0$ and $\\lambda_{\\min}(\\Sigma^\\text{OOD})>0$. While the choice of the IN and OOD distributions may induce a manifold structure, **we do not need to leverage the manifold structure** to show the non-singularities of $\\Sigma^\\text{OOD}$ or $\\Sigma$. Indeed, the matrix $\\Sigma^\\text{OOD}$ is positive-definite if we have $u^T\\Sigma^\\text{OOD}u>0$ for all non-zeros $u \\in \\mathbb{R}^d$. Let $u \\in \\mathbb{R}^d$. From the mean-value theorem and since $\\phi(\\cdot)$ is differentiable and positive, we have\n\n$$u^T \\Sigma^\\text{OOD} u=u^T E_{(x, \\cdot) \\sim P_{\\mathcal{X}, \\mathcal{Y}}^\\text{OOD}} \\biggl[\\Bigl(\\tfrac{\\phi(x^T \\hat w)-\\phi(x^T w^\\text{OOD})}{x^T \\hat w-x^T w^\\text{OOD}}\\Bigr)^2 x x^T \\biggr] u = \\int_{\\mathbb{R}^d}\\Bigl(\\tfrac{\\phi(x^T \\hat w)-\\phi(x^T w^\\text{OOD})}{x^T \\hat w-x^T w^\\text{OOD}}\\Bigr)^2 (x^T u)^2 \\mu^\\text{OOD}(x)  dx >0.$$\n\nThe non-singularities of these matrices ensure that the expected OOD risk is lower bounded by $c(n, p, \\sigma)$. Since the expected OOD risk is also upper bounded by $c(n, p, \\sigma)$ and that $c(n, p, \\sigma)=\\infty$ when $n-1 \\leq p \\leq n+1$, we can deduce from the squeeze theorem that $E_X[R_\\text{OOD}(\\hat f)]=\\infty$ when $n-1 \\leq p \\leq n+1$. Note that **$c(n, p, \\sigma)$** depicts the behavior of $E_X[|| \\hat w - w^\\text{OOD}||^2]$ and **does not depend on the OOD distribution**. As noted in the review, we derive $c(n, p, \\sigma)=\\infty$ by using the **Gaussian assumption on $X$** and Wishart distribution properties. Note that the Gaussian constraint on $X$ is often mentioned in the paper and is explicitly defined in l.141-147.\n\nFor $p < n-1$ or $p>n+1$, the degree of distributional shift between in- and out-of-distribution scenarios affects how tightly the bounds approximate the expected OOD risk. However, our primary objective was not to provide accurate proxies for the expected OOD risk, but rather to **highlight the existence of an infinite peak when $n$ is close to $p$** that we associate with the double descent phenomenon. **This explains why we do not consider assumptions on divergences between IN/OOD distributions**. We hope those clarifications have addressed the concerns raised in the review.\n\n### A.2 Ambiguity in CIFAR-10/100 Experiments (Q4):\n\nAs stated in the paper (l.40-43), our study considers ***post-hoc*  OOD detection methods**.  *Post-hoc* OOD detection methods operate on top of a pretrained classifier after the model’s training process is finished and leverage internal model properties (probabilities, logits, or features) to define a scoring function measuring the prediction confidence. All classifiers are **trained exclusively on in-distribution** (ID) data, i.e., **without using OOD samples during training**. The OOD samples are only used to measure the performance of *post-hoc* OOD detection methods with the AUC and FPR@95."}, {"review_id": "0zMHVXhoAV", "reviewer": "Reviewer_ekWE", "summary": "This paper explores the relationship between model complexity and post-hoc out-of-distribution (OOD) detection, with a focus on the double descent phenomenon. The authors provide theoretical analysis using a simplified Gaussian covariate model and support their findings with broad empirical results across multiple architectures. They demonstrate that OOD detection performance exhibits a double descent pattern with respect to model width, and highlight scenarios where smaller models outperform larger ones. While the theoretical setup is somewhat limited in scope, the paper presents novel observations and thorough experimentation, offering useful insights into model selection for OOD detection.", "questions": "* Could you elaborate on the assumptions made about the OOD distribution, particularly in the theoretical analysis? \n\n* Have you investigated whether the double descent phenomenon appears when varying model depth instead of width? \n\n* To what extent do your theoretical insights generalize beyond the Gaussian covariate model and binary classification with least-squares loss?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper offers a novel perspective by connecting the double descent effect with post-hoc out-of-distribution (OOD) detection, an area that has been largely unexplored.\n\nThe paper provides a comprehensive empirical evaluation across a wide range of architectures (CNNs, ResNets, ViTs, Swin) and OOD detection methods, enhancing the robustness and generality of its findings.\n\nWeaknesses:\nThe theoretical framework relies on simplified assumptions—specifically Gaussian input distributions and binary classification using least-squares loss—which may limit its applicability to more realistic or complex settings.\n\nTheoretical novelty appears limited, as Theorem 1 closely mirrors results from Belkin et al. (2020).\n\n[1] Belkin et al., Two models of double descent for weak features, 2020.", "was_revised": true, "final_justification": "Thank you to the authors for the rebuttal and for addressing my questions and comments. I am keeping my score based on the response.\n\nBest,\nReviewer", "rebuttal": "Dear reviewer ekWE,\n\nWe would like to thank you for your interest and for appreciating the novelty of our study.  \n\nAs highlighted in the review, our paper provides a **comprehensive empirical evaluation** across a **wide range of architectures** (CNNs, ResNets, ViTs, Swin) and ***post-hoc* OOD detection methods**. We want to highlight that the **major contribution** of our study lies in the **empirical observations of a double descent phenomenon in *post-hoc* OOD detection** that had never been observed. Our experiments depict how model complexity affects *post-hoc* OOD detection methods and exhibit, **for the first time**, a **double descent phenomenon in OOD detection**. We would like to emphasize the challenges involved in conducting these experiments with real-world deep networks. Furthermore, we observe that *post-hoc* OOD detection methods may perform less effectively on overparameterized models compared to underparameterized ones. While the paper also includes a **theoretical part**, this part  provides **insights** rather than an exhaustive analysis. In particular, this part depicts a **first attempt to mathematically study** the **double descent phenomenon in *post-hoc* OOD detection** methods based on confidence through a simplified model.\n\n### A.1 Assumptions on OOD distribution (Q1):\n\nOur theoretical insights show that the expected OOD risk for least-squares binary classifiers on Gaussian covariate models under some assumptions exhibits an **infinite peak** when the **number of parameters is close to the number of data**, which we associate with the double descent phenomenon. \n\nIn Theorem 1, the **influence of the OOD distribution** is encapsulated within the **matrix $\\Sigma^\\text{OOD}$**. Assumptions made in the paper ensure that matrices $\\Sigma^\\text{OOD}$ and $\\Sigma$ are positive-definite, for which we have $\\lambda_{\\min}(\\Sigma)>0$ and $\\lambda_{\\min}(\\Sigma^\\text{OOD})>0$. The non-singularities of these matrices ensure that the expected OOD risk is lower bounded by $c(n, p, \\sigma)$. Since the expected OOD risk is also upper bounded by $c(n, p, \\sigma)$ and that $c(n, p, \\sigma)=\\infty$ when $n-1 \\leq p \\leq n+1$, we can deduce from the squeeze theorem that $E_X[R_\\text{OOD}(\\hat f)]=\\infty$ when $n-1 \\leq p \\leq n+1$. Note that **$c(n, p, \\sigma)$** depicts the behavior of $E_X[|| \\hat{w} - w^\\text{OOD}||^2]$ and **does not depend on the OOD distribution**.\n\nFor $p < n-1$ or $p>n+1$, the degree of distributional shift between in- and out-of-distribution scenarios affects how tightly the bounds approximate the expected OOD risk. However, our primary objective was not to provide accurate proxies for the expected OOD risk, but rather to **highlight the existence of an infinite peak when $n$ is close to $p$** that we associate with the double descent phenomenon.\n\nIn the experimental part, we focused on **semantic shifts** and considered both **near and far OOD distributions**, i.e., shifts that are close to or distant from the in-distribution (ID). Our results show that both types of semantic shift are similarly affected by the double descent phenomenon, following the same curve (see Figure 2 and Appendix Figures 4-9). Additionally, to the best of our knowledge, we think that quantifying the amount of shift between two shifted distributions is more relevant to covariate shift scenarios linked to domain adaptation as in [5]. In our setting, divergence-based assumptions are less applicable.\n\n### A.2 Depth versus Width (Q2):  \n\nMost works on the double descent phenomenon focus on **varying the network width rather than depth** [1-4]. A primary reason stems from challenges in modifying network depth across empirical and theoretical frameworks. From an empirical perspective, increasing depth often introduces training instabilities and necessitates hyperparameter adjustments (e.g., learning rates) that complicate comparative analyses. From a theoretical perspective, studies typically rely on frameworks like Random Matrix Theory (RMT) or statistical physics, which often model networks as linear systems with random features. Increasing the depth also increases nonlinear interactions and hierarchical dependencies, which may limit the applicability of these mathematical tools.\n\nNonetheless, **we have recently investigated the influence of the model depth on the double descent phenomenon**. In these experiments, we used ResNet-like models with fixed width and increased the model complexity by varying their depth. We used CIFAR10 as the ID dataset and reused the same OOD datasets from the width experiments. Note that we kept the same training hyperparameters for the study of the depth. Our results depict comparable double descent patterns in both ID generalization and *post-hoc* OOD detection performance (measured via AUC).  We are open to including these figures in the paper. To the best of our knowledge, they represent the **first empirical illustration of depth-wise double descent in realistic models and datasets**. Since we can not add additional figures during the rebuttal, we include a table similar to Table 1 to describe our results. In particular, we report the accuracy results for the underparametrized optimum (depth = 4), the interpolation threshold (depth =  8),  and overparametrized optimum (depth = 120). These results are averaged across five seeds. \n\n**Accuracy (Under / Interpolation / Over):** 77.31% / 66.98% /  76.21%  \n**NC1 Ratio (Under / Over):**  $\\frac{1.53}{1.57} = 0.97$\n\n| Method      | CIFAR100 (U/O) | SUN (U/O) | iNaturalist (U/O) | ImageNet-O (U/O) | Textures (U/O) | Places365 (U/O) |\n|-------------|----------------|-----------|--------------------|-------------------|----------------|------------------|\n| MSP           | 71.62 / 68.84          |  72.45   / 69.23    |  64.80   /   66.53        |    69.78 /       69.13    |    64.03 /  67.40      |   72.47  / 69.00         |\n| MaxLogit      |  70.66   /    70.82    |   72.57  /   71.00  | 65.18    /  68.73         |    67.37 / 71.42          |   52.79  / 68.56       |    72.56 /    70.76      |\n| Energy        |  69.43   /    70.81    |   71.82  /  70.98   |   64.68  /   68.73        |   65.69  /      71.41     | 53.37    / 68.92       |  71.83   /  70.74        |\n| ReAct         |   70.13 /    70.71    |   71.83  /  71.56   |  64.31   /   66.93        |    66.89 /       71.32    |  55.73   / 69.61       |  71.89   / 71.30        |\n| NECO          | 70.85 / 69.55          |   72.83  /  71.24   |  65.62   /      68.47     |    67.98 / 72.32          |   61.15  / 72.72       |    72.74 /   70.77       |\n| ViM           |   71.14  /   67.29     |    71.86 /    71.32|   65.10  /     64.35      |   73.67  / 71.06          |   79.24  /    75.22    |   71.07  /   69.46       |\n| ASH-P         |   69.40  /  70.34      |   71.79  /   70.50  |  64.99   /      69.35     |   65.71  / 71.05          |    53.37 / 69.19       |   71.79  /   70.19       |\n\n\n\n\n### A.3 Generalization of the Theoretical Results (Q3):  \n\nWe emphasize that this work focuses primarily on **empirical investigation** and practical analysis rather than theoretical derivation. Its **core contribution** lies in empirically analyzing the **relationship between model complexity and *post-hoc* OOD detection methods**. The major contribution stems from the empirical observation that ***post-hoc* OOD detection methods** also exhibit a **double descent phenomenon**. While the paper includes theoretical elements, they serve as illustrative tools rather than exhaustive analysis. They depict a first attempt to mathematically study the double descent phenomenon in ***post-hoc* OOD detection** through a simplified model. Although the theoretical framework is constrained to a specific model architecture, the derivation of Theorem 1 is a non-trivial extension of Theorem 1 in [1]. Future works could extend this analysis to more complex scenarios, such as linear models with random features, gradient descent-based algorithms, or sub-Gaussian distributions. We believe these theoretical extensions fall outside our current scope, and we reserve these explorations for future research.\n\n### References \n\n[1] Belkin, Mikhail, Daniel Hsu, and Ji Xu. \"Two models of double descent for weak features.\" SIAM Journal on Mathematics of Data Science 2.4 (2020): 1167-1180.\n\n[2] Couillet, Romain, and Zhenyu Liao. Random matrix methods for machine learning. Cambridge University Press, 2022.\n\n[3] Bach, F. (2024). High-dimensional analysis of double descent for linear regression with random projections. SIAM Journal on Mathematics of Data Science, 6(1), 26-50.\n\n[4] Nakkiran, Preetum, et al. \"Deep double descent: Where bigger models and more data hurt.\" Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021): 124003.\n\n[5] Ben-David, S., Blitzer, J., Crammer, K., & Pereira, F. (2007). \"Analysis of Representations for Domain Adaptation.\" NeurIPS 2006."}, {"review_id": "2pQnGaBpVi", "reviewer": "Reviewer_7SLJ", "summary": "This paper mainly investigates the relationship between model capacity and the post-hoc out-of-distribution (OOD) detection, with a particular focus on whether the \"double descent\" phenomenon exists. The paper empirically demonstrates the double descent phenomenon in OOD detection in CNN and Transformer architectures, and provides an analysis based on random matrix theory.", "questions": "Same as the weakness mentioned above. \n\nIf the authors could provide a clearer explanation or point out if I have any misunderstandings about the OOD risk, I would be willing to increase my score.\n\nThere are additional questions:\n1. In Table 1, could the authors report the NC1 metric for both the underparameterized and overparameterized regimes for each architecture, rather than just their ratio? This would make it easier to compare neural collapse across different architectures.\n2. One issue regarding overparameterization theory is whether there is any theoretical understanding of the impact of the number of parameters across different architectures. It is generally believed that transformers have more parameters than ResNets, but they do not necessarily achieve better OOD detection performance, even in the overparameterized regime. Is there any theoretical explanation for this?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The authors conducted extensive experiments, making comprehensive comparisons of several post-hoc OOD detection algorithms across four different network architectures: CNN, ResNet, ViT, and Swin Transformer.\n2. The authors’ analysis of the experiments is well-organized and easy to follow. In particular, they discovered different behaviors between logit-based and feature-based OOD detection algorithms in the overparameterized regime, which offers some insights for future research.\n3. The overparameterized ResNet is effective in detecting OOD samples, providing a strong backbone for further OOD research.\n4. The authors effectively explain why CNNs tend to perform worse in the overparameterized regime, while other architectures can achieve better results, by leveraging Neural Collapse theory.\n\n**Weaknesses:**\n\nThe main issue lies in the authors’ definition of OOD risk. The OOD risk, as defined by the authors, is very similar to the risk associated with samples that lie between two classes in a binary classification problem. However, OOD samples are not limited to this type. For example, in a binary Gaussian mixture model, samples that are far away from the decision boundary can also be considered OOD. Therefore, I believe the authors’ definition of OOD risk is quite limited.", "was_revised": true, "final_justification": "The rebuttal addressed most of my concerns, particularly clarifying the NC1 analysis. The only remaining point is that the current definition of OOD risk may be somewhat simplified, though it still reasonably captures key intuitions. I will keep my score and recommend acceptance of the paper.", "rebuttal": "Dear reviewer 7SLJ,\n\nWe would like to thank you for your interest and for appreciating both the novelty and experimental findings of our study.  \n\n### A.1 Expected OOD Risk (Q1):  \n\nWe acknowledge that out-of-distribution (OOD) samples arise in many forms, including those that are far from the in-distribution (ID) support or decision boundary. Our definition of the expected OOD risk draws inspiration from the OpenOOD benchmark [1], which categorizes semantic shifts by assigning a binary score (1 for ID data, 0 for OOD data) through an unsupervised approach. Drawing inspiration from both this framework and traditional expected risk, we propose the **expected OOD risk to evaluate whether confidence predictions from a binary classifier $f(\\cdot)$ can be used to define an OOD scoring function**. In particular, the expected OOD risk measures how far the predictions are from an idealized confidence function $f^\\text{OOD}(\\cdot)$, which satisfies:\n$f^{\\text{OOD}} (x) \\approx 0$ or $1$ and is aligned with the probabilities of a perfect classifier when $x$ is ID;\n$f^{\\text{OOD}}(x) \\approx 0.5$ when $x$ is OOD and depicts maximal uncertainty. \n\nThe greater the deviation from this ideal classifier, the higher the expected OOD risk. **We emphasize that the expected OOD risk is not a direct OOD detection metric but evaluates whether a classifier's confidence predictions can be effectively used in *post-hoc* OOD detection methods**, such as Maximum Softmax Probability (MSP) or MaxLogit approaches [2].\n\n### A.2 NC1 metric in Under- Over-parameterized Regimes (Q2):  \n\nWe agree that reporting the NC1 metrics alongside their corresponding ratios would enhance clarity. While we plan to incorporate these values into the final version, we can already share their values in the table below. Note that we have also added values for the depth-wise experiments as recommended by Reviewer ekWE.\n\n| Architecture | NC1 (Under-parameterized) | NC1 (Over-parameterized) |\n|--------------|---------------------------|---------------------------|\n| ResNet-18    | 1.35                      | 0.687                     |\n| ResNet-34    |          1.29           |         0.64           |\n| CNN          | 2.37                      | 2.71                      |\n| Swin         | 35.08                     | 20.65                     |\n| ViT          | 29.52                     | 12.71                     |\n| ResNet (Depth)         |         1.53             |    1.57                  |\n\n### A.3 Overparameterization across Architectures (Q3):  \n\nTransformer architectures may indeed have more parameters than conventional neural networks due to their attention mechanisms. However, increasing attention parameters doesn't automatically lead to superior performance, and the effects of overparameterization in transformers remain poorly understood. For instance, recent studies have shown that these architectures may suffer from attention rank collapse, i.e., a phenomenon where all tokens converge to a single representation with wider or deeper architectures [3-6]. Other works observed that the norm of the residual stream grows exponentially along the layers over the forward pass of multiple transformer blocks [7-9]. [10] shows that such behavior may hurt *post-hoc* OOD detection. \n\nIn our study, we have decided to increase the model complexity of transformers by increasing the width of the MLP in the transformer block, following a similar strategy to that described in [11]. Note also that in our experiments, the number of parameters is not consistent and varies significantly across architectures. Additionally, the training procedure differs for each architecture, which may also impact OOD detection [12].\n\n### References\n\n[1] Yang, Jingkang, et al. \"Openood: Benchmarking generalized out-of-distribution detection.\" Advances in Neural Information Processing Systems 35 (2022): 32598-32611.\n\n[2] Hendrycks, Dan, and Kevin Gimpel. \"A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks.\" International Conference on Learning Representations. 2017.\n\n[3] Dong, Yihe, Jean-Baptiste Cordonnier, and Andreas Loukas. \"Attention is not all you need: Pure attention loses rank doubly exponentially with depth.\" International conference on machine learning. PMLR, 2021.\n\n[4] Noci, Lorenzo, et al. \"Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.\" Advances in Neural Information Processing Systems 35 (2022): 27198-27211.\n\n[5] Saada, Thiziri Nait, Alireza Naderi, and Jared Tanner. \"Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers.\" arXiv preprint arXiv:2410.07799 (2024).\n\n[6] Geshkovski, Borjan, et al. \"A mathematical perspective on transformers.\" Bulletin of the American Mathematical Society 62.3 (2025): 427-479.\n\n[7] Wang, Haoqi, Tong Zhang, and Mathieu Salzmann. \"Demystifying Singular Defects in Large Language Models.\" Forty-second International Conference on Machine Learning.\n\n[8] Heimersheim, Stefan, and Alex Turner. \"Residual stream norms grow exponentially over the forward pass.\" AI Alignment Forum. 2023.\n\n[9] Merrill, William, et al. \"Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n[10] Mueller, J., Zhu, Y., & Liang, P. Mahalanobis++: Improving OOD Detection via Feature Normalization. Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.\n\n[11] Nakkiran, Preetum, et al. \"Deep double descent: Where bigger models and more data hurt.\" Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021): 124003.\n\n[12] Kim, C., Singh, S., & Kolter, J. Z. Investigation of Out-of-Distribution Detection Across Various Models and Training Methodologies. Neural Networks, Volume 172, 2024, Pages 1–12."}, {"review_id": "A7j6Ycid2M", "reviewer": "Reviewer_rex2", "summary": "The authors analyze the **Double Descent** phenomenon in post-hoc out-of-distribution (OOD) detection. \n\nThey define a proxy risk that measures:\n1) the performance of the predictor on in-distribution (ID) data, and\n2) how close to ambiguous prediction the model is on OOD data. \n\nThey theoretically analyze this risk in a simplified learning setup and derive bounds that suggest the existence of the double descent phenomenon. The authors then observe the phenomenon on a toy example. \n\nWhen noise is added to labels the authors also observe the phenomenon on real world data.", "questions": "1) line 148: You refer to the vector $\\mathbb{y}$ of targets as \"vector of target probabilities\". As this vector is generated by a random process with Gaussian noise, the values of $y_i$ are not bounded and may not lie in [0,1]. How do you deal with this?\n\n\n2) Eq. 4: The provided coefficients $\\hat w_\\mathcal{T}$ would minimize the risk for the case of $\\phi(x) = x$, but not otherwise. Even though the authors later assume that $\\phi$ is monotonic, and thus has an inversion $\\phi^{-1}$, this does not allow us to minimize the risk in such a simple manner (a natural idea would be to compute $\\phi^{-1}(y)$, but this generally leads to a different optimization result for $w$). Could you please clarify the formulation? In the current form is seems incorrect to claim that Eq. 4 minimizes the empirical risk.\n\n\n3) lines 321-333: You propose that the neural collapse might indicate quality of OOD detection. Could the same results not be observed if the OOD detection capability was just strongly correlated with ID accuracy, without the need for neural collapse? Could you, please, comment on this?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Notes\n- Exceptionally well written and easy to understand introduction, related work and preliminaries.\n- lines 140-155: \n        - line 148: You refer to the vector $\\mathbb{y}$ of targets as \"vector of target probabilities\". As this vector is generated by a random process with Gaussian noise, the values of $y_i$ are not bounded and may not lie in [0,1]\n        - line 152: You write \"... To analytically solve equation 3, we assume ...\". Eq. 3 does not define a problem, so technically, there is nothing to solve there; however, I understand the intent to minimize the risk w.r.t. the weights.\n        - Eq. 4: The provided coefficients $\\hat w_\\mathcal{T}$ would minimize the risk for the case of $\\phi(x) = x$, but not otherwise. Even though the authors later assume that $\\phi$ is monotonic, and thus has an inversion $\\phi^{-1}$, this does not allow us to minimize the risk in such a simple manner (a natural idea would be to compute $\\phi^{-1}(y)$, but this generally leads to a different optimization result for $w$). \n                - I see this as the biggest detriment to the theoretical analysis. The theoretical bounds derivations seem correct and are a non-trivial modification of Belkin et al. (2020). However, they stand upon the assumption that the weights are set as in Eq. 4, which is in my opinion incorrect.\n- lines 159-167: The definition of a proxy response $z(x)$ seems unintuitive to me. The proposed OOD risk can be (up to a constant multiplier) be instead written with the term $(\\hat f(x) - f^{OOD}(x) - \\frac{\\epsilon}{2})^2$ inside the expectations. What is the motivation for defining $z(x)$?\n- Figure 2: Just a minor point, but the X-axis is not labeled. It is however described in the caption\n- Table 1: \"... underparametrized local minima ...\"\n        - There are often two local minima, at width = 0 and width ~= the interpolation threshold. Is this a typo that should say maxima? If not, which minimum is meant here?\n- lines 321-333: You propose that the neural collapse might indicate quality of OOD detection. Could the same results not be observed if the OOD detection capability was just strongly correlated with ID accuracy, without the need for neural collapse? Could you, please, comment on this?\n\n# Strengths:\n- Novelty\n- Very well written\n- Convincing empirical results\n- Theoretical analysis of a simplified setup; though based on previous work, the modification is non-trivial\n\n# Weaknesses:\n- (Resolved in the rebuttal - typo in original manuscript) The definition of the binary classifier in Section 4.1 is not clear, specifically, the weights $w$ do not seem to minimize the defined empirical risk\n- The definition of the noisy response z(x) seems unnecessary\n- (Resolved in the rebuttal - typo in original manuscript)The derivations of the OOD risk bounds require the specific setting of the weights $w$ by the pseudo inverse\n\n# Comment on the rating:\nI am torn on this paper. It is very well written, the idea to analyze double descent in context of OOD detection is novel, the topic is interesting, and the empirical results are nice. However, the derivation of the bounds is based on Eq. 4, with which I have issues (All of which were resolved in the rebuttal). It is not clear to me why the weights would be set in such a manner. My issues with the paper are limited to Section 4.1. If the authors can address my concerns with Eq. 4 I am willing to increase my rating. In its current form, I believe that the paper would be stronger without the theoretical analysis and simply as an empirical study.", "was_revised": true, "final_justification": "The authors have successfully addressed my main concerns in the rebuttal, particularly the critical issue regarding Section 4.1. The explanation of the typo in Equation 4 resolves my concern and makes the theoretical analysis logically sound.\n\nAll the strong points that were brought up in my initial review still stand.", "rebuttal": "Dear reviewer rex2,\n\nWe thank you for the thorough review and your encouraging comments. \n\nAs highlighted in the review, our paper provides a **comprehensive empirical evaluation** across a **wide range of architectures** (CNNs, ResNets, ViTs, Swin) and ***post-hoc* OOD detection methods**. We want to highlight that the **major contribution** of our study lies in the **empirical observations of a double descent phenomenon in *post-hoc* OOD detection**. To the best of our knowledge, the double descent phenomenon has not been observed before in the OOD context. Our experiments depict how model complexity affects *post-hoc* OOD detection methods and exhibit, for the first time, a double descent phenomenon in OOD detection. Furthermore, we observe that *post-hoc* OOD detection methods may perform less effectively on overparameterized models compared to underparameterized ones. While the paper also includes a **theoretical part**, this part provides **insights** rather than an exhaustive analysis. In particular, this part depicts a **first attempt to mathematically study** the **double descent phenomenon in *post-hoc* OOD detection** methods based on confidence through a simplified model.\n\n### A.1 & A.2 Least-Squares Binary Classifier Definition (Q1 & Q2):  \n\nOur theoretical insights show that the expected OOD risk for least-squares binary classifiers on Gaussian covariate models under some assumptions exhibits an **infinite peak** when the **number of parameters is close to the number of data**, which we associate with the double descent phenomenon. To establish this result, we derive a lower bound $c(n,p,\\sigma)$ for the expected OOD risk, and we show that $c(n,p,\\sigma)=\\infty$ when $n-1 \\leq p \\leq n+1$. $c(n, p, \\sigma)$ describes $E_X[|| \\hat w - w^\\text{OOD}||^2]$.\n\nThe confusion surrounding questions Q1 and Q2 arises from a typo in the definition of the least-squares binary classifier in equation 4. As correctly noted in the review, the least-squares binary classifier minimizes the logit-based expected risk rather than the probability-based expected risk. Therefore, in the definition of $\\hat w_\\mathcal{T}$ in equation 4, $y$ should be replaced with $u=X w^\\text{OOD}$, which is the vector of logits of the optimal classifier. Note that we considered the logit-based definition of $\\hat w_\\mathcal{T}$ in the proofs, e.g., equation 12 (l.139). While the least-binary classifier minimizes the logits-based expected risk, **Theorem 1 focuses on the probability-based expected OOD risk**. The objective of this theorem is to show that probability-based *post-hoc* OOD detection methods may exhibit a double descent behavior. Without loss of generality, note also that the Gaussian noise could be added to the logit vector $u$. Accordingly, we will update definitions 4 and 5. We sincerely thank you for bringing this element to our attention. Corrections will be made promptly to eliminate further misinterpretation.\n\nWe emphasize that this work focuses primarily on **empirical investigation** and practical analysis rather than theoretical derivation. Its core contribution lies in empirically analyzing the relationship between model complexity and *post-hoc* OOD detection methods. The major contribution stems from the empirical observation that *post-hoc* OOD detection methods also exhibit a double descent phenomenon. While the paper includes theoretical elements, these serve as **insights** with an illustrative example rather than exhaustive analysis. They depict a **first attempt** to mathematically study the double descent phenomenon in *post-hoc* OOD detection method through a **simplified model**. As mentioned in the review, although the theoretical framework is constrained to a particular model, the derivation of Theorem 1 is a non-trivial extension of Theorem 1 in [1]. We hope that future works could extend this analysis to more complex scenarios, such as linear models with random features, gradient descent-based algorithms, or sub-gaussian distributions. We believe these theoretical extensions fall outside our current scope, and we reserve these explorations for future research.\n\n### A.3 Correlations between ID Accuracy and OOD Detection (Q3):\n\nWe would like to clarify that we did not claim a direct link between NC and OOD quality. Rather, we explained lines 318-321 that we use NC as a tool to describe the structure of the final-layer features. This representation helped us gain a better understanding of the quality of OOD detection, as we showed empirically, lines 327-333. Also, while OOD detection performance and ID accuracy may be positively correlated, prior works show that this correlation is not universally consistent [1, 2]. For instance, Kim et al. [1] showed that ResNet-50 v1 outperforms its v2 counterpart on OOD benchmarks despite having lower in-distribution accuracy (76% vs. 80%). [2] and [3] observed a similar trend. These results align with our empirical findings. For example, ViT demonstrates lower ID accuracy in overparameterized regimes but achieves superior OOD detection performance using NECO and ViM methods in Figure 2. Furthermore, variations between ID accuracy and OOD detection performance may not show a consistent correlation. For instance, feature-based OOD methods do not exhibit a double descent phenomenon consistently, while the ID accuracy describes a double descent curve. To better illustrate this, the Table below compares OOD detection performance (for logit- and a hybrid-based methods) vs accuracy/NC ratios like in Table 1. The accuracy ratio is defined as the local maximum accuracy in the over-parameterized regime divided by the local maximum in the under-parameterized regime. It follows that value greater than 1 indicate improved accuracy with over-parametrization. We highlight in bold the ratio if its value correlates with the OOD detection performance improvement or degradation. As illustrated, the accuracy ratio fails to provide information on OOD detection performance in the under- and over-parameterized regimes.\n\n|Model|Method|Acc ratio/NC ratio|SUN(U/O)|Places365(U/O)|CIFAR100(U/O)|\n|-|-|-|-|-|-|\n|CNN|MaxLogit|0.99/**0.88**|72.98/60.13|73.33/61.25|73.38/70.37|\n|CNN|ASH-P||68.60/60.36|69.35/61.48|71.11/70.45|\n|ResNet-18|MaxLogit|**1.08**/**1.96**|70.64/72.51|70.69/72.64|69.76/73.65|\n|ResNet-18|ASH-P||69.11/71.73|69.19/71.85|67.58/72.89|\n|Swin|MaxLogit|**1.07**/**1.70**|59.91/70.75|59.84/70.46|61.79/66.95|\n|Swin|ASH-P||59.49/70.74|58.15/70.42|55.10/66.89|\n|ViT|MaxLogit|0.89/**2.32**|66.09/70.30|66.13/69.79|64.60/66.69|\n|ViT|ASH-P||64.79/70.27|64.86/69.79|63.0866.61|\n|ResNet-Depth|MaxLogit|**0.98**/**0.97**|72.57/71.00|72.56/70.76|70.66/70.82|\n|ResNet-Depth|ASH-P||71.79/70.50|71.79/70.19|69.40/70.34|\n\n### A.4 Typos  \n\nRegarding other feedback:\n- We will add the x-axis label in Figure 2.\n- We will change the minima to maxima in Table 1.\n- $z(x)$ depicts predictions rescaled within the range [-1, 1]. We believe this formulation helps in better understanding the expected OOD risk definition, particularly concerning Remark 4.1 and Remark 4.2.\n\nWe would like to thank you again for your thoughtful review.\n\n### References\n\n[1] Kim, C., Singh, S., & Kolter, J. Z. Investigation of Out-of-Distribution Detection Across Various Models and Training Methodologies. Neural Networks, Volume 172, 2024, Pages 1–12.\n\n[2] Mueller, J., Zhu, Y., & Liang, P. Mahalanobis++: Improving OOD Detection via Feature Normalization. Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.\n\n[3] Yang, Jingkang, et al. \"Openood: Benchmarking generalized out-of-distribution detection.\" Advances in Neural Information Processing Systems 35 (2022): 32598-32611.\n\n[4] Ammar, M. B., Belkhir, N., Popescu, S., Manzanera, A., and Franchi, G. NECO: NEural Collapse Based Out-of-distribution detection. In The Twelfth International Conference on Learning\nRepresentations, 2024.\n\n[5] Ming, Y., Sun, Y., Dia, O., and Li, Y. How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection. International Conference on Learning Representations,2023.\n\n[6] Haas, J., Yolland, W., and Rabus, B. T. Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856."}]

Author rebuttal:

Meta-review: The paper extends the double descent phenomenon (previously demonstrated in regression by Belkin et al. 2020) to out-of-distribution (OOD) detection. The authors show both theoretically and empirically that OOD detection performance exhibits a characteristic double descent pattern as model capacity increases.

Strengths: 

- extensive experimental validation across multiple standard model architectures, benchmark datasets, and detection techniques and wide range of architectures.

- novelty of applying double descent analysis to OOD detection, which has been largely unexplored

Weaknesses:

- reliance on simplified assumptions, e.g. Gaussian data-distribution assumption 

- OOD distribution and OOD risk definition potentially limited

- limited practical guidance for practitioners
Decision: Accept (poster)