Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis of the role of model complexity

## Abstract

Abstract While overparameterization is known to benefit generalization, its impact on Out-Of-Distribution (OOD) detection is less understood.
This paper investigates the influence of model complexity in OOD detection.
We propose an expected OOD risk metric to evaluate classifiers confidence on both training and OOD samples.
Leveraging Random Matrix Theory, we derive bounds for the expected OOD risk of binary least-squares classifiers applied to Gaussian data.
We show that the OOD risk depicts an infinite peak, when the number of parameters is equal to the number of samples, which we associate with the double descent phenomenon.
Our experimental study on different OOD detection methods across multiple neural architectures extends our theoretical insights and highlights a double descent curve.
Our observations suggest that overparameterization does not necessarily lead to better OOD detection.
Using the Neural Collapse framework, we provide insights to better understand this behavior.
To facilitate reproducibility, our code will be made publicly available upon publication.

## 1 Introduction

In recent years, large neural networks have seen increased use in Machine Learning due to their impressive generalization properties (Brown, ; Dubey et al., ).
While empirical evidence suggests that rich machine learning systems obtain near-optimal generalization results when trained to interpolate training data (Zhang et al., ), the classical bias-variance trade-off theory (Geman et al., ) suggests that such models overfit and generalize poorly.
Indeed, the classical literature describes the generalization error with respect to the model complexity as a U-shaped curve and suggests finding a model between underfitting and overfitting, i.e., a model rich enough to express underlying structure in the data and simple enough to avoid fitting spurious patterns.
To bridge the gap between the classical theory and the modern practice, Belkin et al. () introduced the concept of “double descent” within a unified generalization error curve.
In this setting, for “small” model complexities, the generalization error curve exhibits the U-shaped curve described by the bias-variance trade-off.
However, when the model complexity is higher than the interpolation threshold, i.e., when the model is rich enough to fit the training data, increasing the model complexity leads to a second decrease in the generalization error.
A popular intuitive explanation of this phenomenon is that by considering large model complexities that contain more candidate predictors compatible with the training data, we are also able to
find interpolating functions that are “simpler” and are smoother to follow a form of Occam’s razor (Belkin et al., ).

Although the double descent phenomenon provides valuable insights to understand generalization of rich models on unseen data, its understanding on Out-Of-Distribution (OOD) detection has received less attention.
OOD detection addresses a distinct challenge in deep neural networks (DNNs): their tendency to make high-confidence predictions, even for inputs that differ significantly from the training data.
While generalization focuses on the model’s ability to classify data that has shifted, OOD detection emphasizes the model’s capacity to recognize when a shift is too large and refrain from confident predictions.
In real-world applications, reliable OOD detection is crucial to ensuring the safety and reliability of AI systems. This includes fields such as healthcare (Schlegl et al., ), industrial inspection (Paul Bergmann & Stege, ), and autonomous driving (Kitt et al., ).
Therefore, understanding the influence of the model complexity on OOD detection is crucial for model selection in critical tasks.

While overparameterization is known to benefit generalization, its effects on OOD detection still remain limited.
In this work, we investigate the role of the model complexity in OOD detection.
In particular, we highlight a double descent phenomenon similar to the one observed for generalization error.
To the best of our knowledge, the double descent phenomenon has never been observed in OOD detection.
The results of our theoretical and empirical analyses are outlined below.

#### Contributions.

We make the following contributions, taking a step towards a better theoretical and empirical understanding of the influence of model complexity on OOD detection:

- 1.
We propose an expected OOD risk metric to evaluate classifiers confidence on both training and OOD samples.
- 2.
Using Random Matrix Theory, we derive bounds for both the expected risk and OOD risk of binary least-squares classifiers applied to Gaussian data with respect to the model complexity.
We show that both risks exhibit an infinite peak, when the number of parameters is equal to the number of samples, which we associate with the double descent phenomenon.
- 3.
We empirically observe a double descent phenomenon curve in various OOD detection methods and across multiple neural architectures, including transformer-based (ViT, Swin) and convolutional-based models (ResNet, CNN).
- 4.
We also observe that OOD detection in the overparametrized regime is not guaranteed to be better than in the underparametrized regime.
Using the Neural Collapse framework Papyan et al. (), we propose to better explain this architecture-dependant improvement on OOD detection with overparametrization.

To facilitate reproducibility, our code will be made publicly available upon publication.

## 2 Related work

OOD Detection. The OOD detection research focuses on two primary directions: supervised and unsupervised approaches. We will focus on latter ones, also called post-hoc methods. These can be categorized based on the essential feature used for the scoring function.
First, the logit- or confidence-based methods leverage network logits to derive a confidence measure used as an OOD scoring metric (Hendrycks & Gimpel, ; DeVries & Taylor, ; Liu et al., ; Huang & Li, ; Hendrycks et al., ). A common baseline for this methods is the softmax score (Hendrycks & Gimpel, ), which simply uses the model softmax prediction as the OOD score.
Then, the Energy (Liu et al., ) elaborates on it by computing the the LogSumExp on the logits, thus offering empirical and theoretical advantages over the Softmax confidence score.
Second, feature-based and hybrid methods (Lee et al., ; Wang et al., ; Sun et al., ; Ming et al., ; Djurisic et al., ; Ammar et al., ) exploit the model’s final representation to derive the scoring function.
Mahalanobis (Lee et al., ) estimates density on ID training samples using a mixture of class-conditional Gaussians based on the feature distributions.
NECO (Ammar et al., ), on the contrary, leverages the geometric properties of Neural Collapse to construct a scoring function based on the relative norm of a sample within the subspace defined by the Simplex Equiangular Tight Frame (ETF) formed by the ID data.
Hybrid methods are characterized by the fact that they can be augmented by using the logits as weighting factors on the scoring metrics defined on the features.

Double Descent.
The double descent risk curve was introduced by Belkin et al. () to explain the good performance observed in practice by overparameterized models (Zhang et al., ; Belkin et al., ; Nakkiran et al., ) and to bridge the gap between the classical bias-variance trade-off theory and modern practices.
Theoretical investigation into this phenomenon mainly focuses on various linear models in both regression and classification problems through the Random Matrix Theory (Louart et al., ; Liao et al., ; Jacot et al., ; Derezinski et al., ; Kini & Thrampoulidis, ; Mei & Montanari, ; Deng et al., ; Bach, ; Brellmann et al., ), techniques from statistical mechanics (d’Ascoli et al., ; Canatar et al., ), the VC theory (Lee & Cherkassky, ; Cherkassky & Lee, ), or novel bias-variance decomposition in deep neural networks (Yang et al., ).
The double descent phenomenon has also been observed in experiments with popular neural network architectures (Belkin et al., ; Nakkiran et al., ).
In addition to depending on the model complexity, the double descent phenomenon also depends on other dimensions such as the level of regularization (Liao et al., ; Mei & Montanari, ), the number of epochs (Nakkiran et al., ; Stephenson & Lee, ; Olmin & Lindsten, ), or the data eigen-profile (Liu et al., ).
Finally, the theoretical background on double descent and benign overparametrization developed by Bartlett et al. () inspired subsequent works that focused on generalization under dataset shifts (Tripuraneni et al., ; Hao et al., ; Kausik et al., ; Hao & Zhang, ).
It is important to note that these dataset shifts concern scenarios where the model can generalize, e.g., the class labels are the same as in the training. This studies do not address the issue of presenting the model with samples that differ significantly beyond the point of generalization, i.e., OOD detection.

## 3 Preliminaries

#### Notations.

For a real vector ${\bm{v}}$, we denote by $\lVert{\bm{v}}\rVert_{2}$ the euclidean norm of ${\bm{v}}$.
When the matrix ${\bm{A}}$ is full rank, we denote by ${\bm{A}}^{+}$ the Moore-Penrose inverse of ${\bm{A}}$.
We depict by $[d]:=\{1,\ldots,d\}$ the set of the $d$ first natural integers.
For a subset $\mathcal{T}\subseteq[d]$, we denote by $\mathcal{T}^{c}:=[d]\,\backslash\,\mathcal{T}$ its complement set.
For a subset $\mathcal{T}\subseteq[d]$, a $d$-dimensional vector ${\bm{v}}\in\mathbb{R}^{d}$ and a $n\times d$ matrix ${\bm{A}}=\bigl{[}{\bm{a}}^{(1)}|\ldots|{\bm{a}}^{(n)}\bigr{]}^{T}\in\mathbb{R}
^{n\times d}$, we use ${\bm{v}}_{\mathcal{T}}=[{\bm{v}}_{j}:j\in\mathcal{T}]$ to denote its $|\mathcal{T}|$-dimensional subvector of entries from $\mathcal{T}$ and ${\bm{A}}_{\mathcal{T}}=\bigl{[}{\bm{a}}^{(1)}_{\mathcal{T}}|\ldots|{\bm{a}}^{(
n)}_{\mathcal{T}}\bigr{]}^{T}$ to denote the $n\times|\mathcal{T}|$ design matrix with variables from $\mathcal{T}$.
We use $\lambda_{\min}({\bm{A}})$ and $\lambda_{\max}({\bm{A}})$ to depict the minimum and maximum eigenvalues of ${\bm{A}}$, respectively.
$\mathcal{N}(0,{\bm{I}}_{d})$ denotes the standard multivariate Gaussian distribution of $d$ random variables.

#### Supervised Learning Problems.

In supervised learning, we employ training dataset $\mathcal{D}:=\bigl{\{}({\bm{x}}_{1},y_{1}),\cdots,({\bm{x}}_{n},y_{n})\bigr{\}}$ of $n$ independent and identically distributed (i.i.d.) samples drawn from an unknown distribution $P_{\mathcal{X},\mathcal{Y}}$ over $\mathcal{X}\times\mathcal{Y}$.
Using samples from the dataset $\mathcal{D}$, the objective is to find a predictor $\hat{f}:\mathcal{X}\rightarrow\mathcal{Y}$ among a class of functions $\mathcal{F}$ to predict the target $y\in\mathcal{Y}$ of a new sample ${\bm{x}}\in\mathcal{X}$.
In particular, given a loss function $\ell:\,\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}$, the objective is to minimize the expected risk (or loss) defined, for all $\hat{f}\in\mathcal{F}$, as:

$$ $R(\hat{f})=\mathbb{E}_{({\bm{x}},y)\sim P_{\mathcal{X},\mathcal{Y}}}\bigl{[} \ell\bigl{(}\hat{f}({\bm{x}}),y\bigr{)}\bigr{]}=\int_{\mathcal{X}\times \mathcal{Y}}\ell\bigl{(}\hat{f}({\bm{x}}),y\bigr{)}dP_{\mathcal{X},\mathcal{Y} }({\bm{x}},y).$ (1) $$

Typically, we choose the mean-squared loss $\ell(\hat{f}({\bm{x}}),y)=\bigl{(}\hat{f}({\bm{x}})-y\bigr{)}^{2}$ for regression problems or the zero-one loss $\ell(\hat{f}({\bm{x}}),y)=\mathbf{1}_{\hat{f}({\bm{x}})\neq y}$ for classification problems.
We denote the optimal predictor by $f^{*}\vcentcolon=\operatorname*{arg\,min}_{f\in\mathcal{F}}R(f)$.
Since the distribution $P_{\mathcal{X},\mathcal{Y}}$ is unknown in practice, we instead try to minimize an empirical version of the expected risk based on the dataset $\mathcal{D}:=\bigl{\{}({\bm{x}}_{i},y_{i})\bigr{\}}_{i=1}^{n}$ and defined as

$$ $R_{emp}(\hat{f})=\tfrac{1}{n}\sum_{i=1}^{n}\ell\bigl{(}\hat{f}({\bm{x}}_{i}),y _{i}\bigr{)}.$ (2) $$

#### Out-of-Distribution Detection.

In machine learning problems, we usually assume that the test data distribution is similar to the training data distribution (the closed-world assumption).
As this is not the case in real-world applications, the Out-of-Distribution (OOD) detection aims to flag inputs that significantly deviate from the training data to prevent unreliable predictions.
In the following, we denote by $P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}$ a distribution over $\mathcal{X}\times\mathcal{Y}$ that differs from the training distribution $P_{\mathcal{X},\mathcal{Y}}$.
OOD data typically involve a semantic shift and represent concepts or labels not seen during training.
A popular class of OOD detection techniques relies on the definition of a scoring function $s(\cdot\leavevmode\nobreak\ ;\hat{f})$, which uses the probability predictions of the classifier $\hat{f}(\cdot)$ as scores to flag an instance ${\bm{x}}$ as OOD when the score $s({\bm{x}};\hat{f})$ is below a certain threshold $\lambda$.
A common approach is to use the Maximum Softmax Probability that returns the higher softmax probabilities of the predictor $\hat{f}(\cdot)$ as a scoring function to measure the prediction confidence.

## 4 Double Descent for the Binary Classification in Gaussian Covariate Model

In this section, we introduce the expected OOD risk metric and we present our main theoretical results on binary least-squares classifiers applied to Gaussian data.
We assume that $\mathcal{X}\subseteq\mathbb{R}^{d}$ and $\mathcal{Y}:=[0,1]$.
Let $\phi:\mathbb{R}\to\mathcal{Y}$ be a mapping, we denote by $\mathcal{F}_{d}:=\bigl{\{}f:\mathcal{X}\to\mathcal{Y},{\bm{x}}\mapsto\phi({\bm
{x}}^{T}{\bm{w}})\leavevmode\nobreak\ |\leavevmode\nobreak\ {\bm{w}}\in\mathbb
{R}^{d}\bigr{\}}$ the class of functions considered in this study.

### 4.1 System Model

#### Gaussian Covariate Model & Binary Classification.

We assume we have a training dataset $\mathcal{D}:=\bigl{\{}({\bm{x}}_{i},{\bm{y}}_{i})\bigr{\}}_{i=1}^{n}$ of $n$ i.i.d samples drawn from a Gaussian covariate model, i.e., from a distribution $P_{\mathcal{X},\mathcal{Y}}$ over $\mathcal{X}\times\mathcal{Y}$; where ${\bm{x}}_{i}\sim\mathcal{N}(0,{\bm{I}}_{d})$ and ${\bm{y}}_{i}$ is a noisy response of ${\bm{x}}_{i}$ with respect to the function $f^{*}:{\bm{x}}\mapsto\phi({\bm{x}}^{T}{\bm{w}}^{*})$ that is defined as

$$ ${\bm{y}}_{i}=f^{*}({\bm{x}}_{i})+{\bm{\epsilon}}_{i}=\phi({\bm{x}}_{i}^{T}{\bm {w}}^{*})+{\bm{\epsilon}}_{i},$ $$

with ${\bm{\epsilon}}_{i}\sim\mathcal{N}(0,\sigma^{2})$ and $\sigma>0$.
The objective is to find a classifier $\hat{f}(\cdot)\in\mathcal{F}_{d}$ that fits $f^{*}(\cdot)$.
Without loss of generality, this problem can be interpreted as a binary classification problem where $f^{*}(\cdot)$ returns a probability.
Let ${\bm{X}}=[{\bm{x}}_{1},\ldots,{\bm{x}}_{n}]^{T}\in\mathbb{R}^{n\times d}$ be the data matrix containing the $n$ samples ${\bm{x}}_{i}\in\mathbb{R}^{d}$ as column vectors and ${\bm{y}}=[{\bm{y}}_{1},\ldots,{\bm{y}}_{n}]^{T}\in\mathbb{R}^{n}$ be the target vector of probabilities.
Given the loss function $\ell:(\hat{y},y)\mapsto(\hat{y}-y)^{2}$, in order to find ${\bm{w}}^{*}$ (and thus $f^{*}(\cdot)$), we want to minimize the empirical risk $R_{emp}:\mathcal{F}_{d}\to\mathbb{R}$ defined in equation 2 as

$$ $\displaystyle R_{emp}(\hat{f})=\tfrac{1}{n}\sum_{i=1}^{n}\ell\bigl{(}\hat{f}({ \bm{x}}_{i}),{\bm{y}}_{i}\bigr{)}=\tfrac{1}{n}\sum_{i=1}^{n}\bigl{(}\phi({\bm{ x}}_{i}^{T}{\bm{w}})-{\bm{y}}_{i}\bigr{)}^{2}=\tfrac{1}{n}\|\phi({\bm{X}}{\bm{ w}})-{\bm{y}}\|_{2}^{2}.$ (3) $$

#### Least-Squares Binary Classifiers.

To analytically solve equation 3, we assume that $n\ll d$ and that the data matrix ${\bm{X}}$ is full row rank.
We consider a particular selection of classifiers $\hat{\mathcal{F}}_{d}:=\bigl{\{}\hat{f}_{\mathcal{T}}:{\bm{x}}\mapsto\phi({\bm
{x}}^{T}\hat{\bm{w}})\in\mathcal{F}_{d}\leavevmode\nobreak\ |\leavevmode
\nobreak\ \mathcal{T}\subseteq[d]\bigr{\}}$, in which $\hat{f}_{\mathcal{T}}\in\hat{\mathcal{F}}_{d}$ uses a subset $\mathcal{T}\subseteq[d]$ of features that fits coefficients $\hat{\bm{w}}\in\mathbb{R}^{d}$ as

$$ $\hat{{\bm{w}}}_{\mathcal{T}}={\bm{X}}_{\mathcal{T}}^{+}{\bm{y}}\quad\text{and} \quad\hat{{\bm{w}}}_{\mathcal{T}^{c}}=\mathbf{0}.$ (4) $$

#### Out-of-Distribution Risk.

To measure the ability of binary classifiers $\hat{f}(\cdot)\in\hat{\mathcal{F}}_{d}$ to provide prediction confidence on samples drawn from both the training distribution $P_{\mathcal{X},\mathcal{Y}}$ and the OOD distribution $P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}$, we introduce an OOD risk function similar to the expected risk defined in equation 1.
Let $f_{\text{OOD}}^{*}:\mathcal{X}\to\mathcal{Y}$ be the mapping chosen from $\mathcal{F}_{d}$ such that $f_{\text{OOD}}^{*}({\bm{x}})$ is close to $0.5$ when the sample ${\bm{x}}$ is more likely drawn from the $P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}$ and close to $f^{*}({\bm{x}})$ when ${\bm{x}}$ is more likely drawn from $P_{\mathcal{X},\mathcal{Y}}$.
We define the noisy response $z$ to a given sample ${\bm{x}}\in\mathcal{X}$ for the mapping $f^{*}_{\text{OOD}}(\cdot)$ as:

$$ $z=2f^{*}_{\text{OOD}}({\bm{x}})-1+\epsilon^{\prime}=2\phi({\bm{x}}^{T}{\bm{w}} ^{*}_{\text{OOD}})-1+\epsilon^{\prime},$ $$

where $\epsilon^{\prime}\sim\mathcal{N}(0,\sigma^{\prime})$ and $\sigma^{\prime}>0$.
To measure whether prediction confidences of a binary classifier $\hat{f}(\cdot)$ can be used for defining an OOD scoring function, we define the Out-of-Distribution Risk $R_{\text{OOD}}:\mathcal{F}_{d}\to\mathbb{R}$ as:

$$ $R_{\text{OOD}}(\hat{f})=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X}, \mathcal{Y}}}\bigl{[}(2\hat{f}({\bm{x}})-1-z)^{2}\bigr{]}+\mathbb{E}_{({\bm{x} },\cdot)\sim P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}}\bigl{[}(2\hat{f}({\bm{x }})-1-z)^{2}\bigr{]},$ (5) $$

which depicts the expected risk of the predictor $2\hat{f}(\cdot)-1$ on the loss function $\ell:(\hat{y},y)\mapsto(\hat{y}-y)^{2}$ and distributions $P_{\mathcal{X},\mathcal{Y}}$ and $P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}$.

###### Remark 4.1 .

A low value for $R_{\text{OOD}}(\hat{f})$ indicates two aspects: $(i)$ the classifier $\hat{f}(\cdot)$ is confident on predictions over the training distribution $P_{\mathcal{X},\mathcal{Y}}$, which corresponds to a low $\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}}\bigl{[}(2\hat{f}
({\bm{x}})-1-z)^{2}\bigr{]}$; and $(ii)$ the classifier $\hat{f}(\cdot)$ is not confident on predictions over the distribution $P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}$, which is reflected by a low $\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}}
\bigl{[}(2\hat{f}({\bm{x}})-1-z)^{2}\bigr{]}$.

###### Remark 4.2 .

Note that the OOD risk function $R_{\text{OOD}}(\cdot)$ defined in equation 5 can be extended to multi-class classifiers $\hat{f}(\cdot)$ using the softmax function as

$$ $R_{\text{OOD}}(\hat{f})=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X}, \mathcal{Y}}}\bigl{[}(2\lVert\hat{f}({\bm{x}})\rVert_{\infty}-1-z)^{2}\bigr{]} +\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}} \bigl{[}(2\lVert\hat{f}({\bm{x}})\rVert_{\infty}-1-z)^{2}\bigr{]},$ $$

where $\lVert\cdot\rVert_{\infty}$ denotes the infinity norm.

In order to use the Random Matrix Theory, we make the following assumption on the activation function $\phi:\mathbb{R}\to\mathcal{Y}$.

###### Assumption 4.1 .

The positive-definite matrix

$$ ${\bm{\Sigma}}=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}} \bigl{[}{\bm{x}}{\bm{x}}^{T}\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*})^{2}\bigr{] }=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}}\Bigl{[}\tfrac{ \partial\phi({\bm{x}}^{T}vw^{*})}{\partial{\bm{w}}^{*}}^{T}\tfrac{\partial\phi ({\bm{x}}^{T}{\bm{w}}^{*})}{\partial{\bm{w}}^{*}}\Bigr{]}\in\mathbb{R}^{d \times d}$ $$

is nonsingular.

###### Remark 4.3 .

Note that Assumption 4.1 may be satisfied even if the matrix ${\bm{x}}{\bm{x}}^{T}\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*})^{2}$ is a rank one matrix.
E.g, for the activation function $\phi:{\bm{x}}\mapsto{\bm{x}}$, we have ${\bm{\Sigma}}=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}}
\bigl{[}{\bm{x}}{\bm{x}}^{T}\bigr{]}={\bm{I}}_{d}$.

###### Remark 4.1 .

###### Remark 4.2 .

###### Assumption 4.1 .

###### Remark 4.3 .

### 4.2 Prediction Risk

Leveraging the Random Matrix Theory and following the same line of arguments of Theorem 1 in Belkin et al. (), we derive bounds for the risk of the subset of classifiers defined in $\hat{\mathcal{F}}_{d}$ with equation 4 (see proof in Appendix A.1).

###### Theorem 1 .

Let $(p,q)\in\llbracket 1,d\rrbracket^{2}$ such that $p+q=d$, $\mathcal{T}\subseteq[d]$, be an arbitrary subset of the $d$ first natural integers, and $\mathcal{T}^{c}:=[d]\,\backslash\,\mathcal{T}$ its complement set.
Let $\hat{{\bm{w}}}\in\mathbb{R}^{d}$ such that $\hat{{\bm{w}}}_{\mathcal{T}}={\bm{X}}_{\mathcal{T}}^{+}{\bm{y}}\in\mathbb{R}^{p}$ and $\hat{{\bm{w}}}_{\mathcal{T}^{c}}=\mathbf{0}\in\mathbb{R}^{q}$.
Then the expected risk with respect to the loss function $\ell:(\hat{y},y)\mapsto(\hat{y}-y)^{2}$ of the predictor $\hat{f}_{\mathcal{T}}:{\bm{x}}\mapsto\phi({\bm{x}}^{T}\hat{\bm{w}})\in\hat{
\mathcal{F}}_{d}$ satisfies

$$ $\lambda_{\min}({\bm{\Sigma}})c(n,p,\sigma)+\sigma^{2}\leq\mathbb{E}_{\bm{X}} \bigl{[}R(\hat{f}_{\mathcal{T}})\bigr{]}\leq\lambda_{\max}({\bm{\Sigma}})c(n,p ,\sigma)+\sigma^{2},$ $$

where

$$ $\displaystyle c(n,p,\sigma)$ $\displaystyle=\begin{cases}\tfrac{n}{n-p-1}\Bigl{(}\|{\bm{w}}_{\mathcal{T}^{c} }^{*}\|_{2}^{2}+\sigma^{2}\Bigr{)}+\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}& \text{if }p\leq n-2,\\ +\infty&\text{if }n-1\leq p\leq n+1,\\ \bigl{(}1-\tfrac{n}{p}\bigr{)}\bigl{\|}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2} ^{2}+\tfrac{n}{p-n-1}\Bigl{(}\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}+\sigma ^{2}\Bigr{)}+\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}&\text{if }p\geq n+2. \end{cases}$ (6) $$

###### Remark 4.4 .

In the case in which ${\bm{\Sigma}}=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}}
\bigl{[}{\bm{x}}{\bm{x}}^{T}\bigr{]}=\lambda{\bm{I}}_{d}$ for some $\lambda>0$, we have $\mathbb{E}_{\bm{X}}\bigl{[}R(\hat{f}_{\mathcal{T}})\bigr{]}=\lambda c(n,p,
\sigma)+\sigma^{2}$.
Note that Theorem 1 in Belkin et al. () constitutes a special case of Theorem 1 for $\phi:{\bm{x}}\mapsto{\bm{x}}$, which corresponds to the case where ${\bm{\Sigma}}={\bm{I}}_{d}$.

###### Remark 4.5 .

From Theorem 1, we have $\mathbb{E}_{\bm{X}}\bigl{[}R(\hat{f}_{\mathcal{T}})\bigr{]}=\infty$ around $p=n$.
The expected risk decreases again as $p$ increases beyond $n$ and highlights a double descent phenomenon.
This result is consistent with the literature of double descent (Mei & Montanari, ; Louart et al., ; Liao et al., ; Bach, ), which identifies the ratio $p/n$ as the model complexity of a linear model to describe an under-($p/n<1$) and an over-($p/n>1$) parameterized regimes for the expected risk with a phase transition around $p/n=1$ characterized by a peak.

###### Theorem 1 .

###### Remark 4.4 .

###### Remark 4.5 .

### 4.3 Out-Of-Distribution Risk

Using a similar approach to that Theorem 1, we obtain the following result on the subset of classifiers defined in $\hat{\mathcal{F}}_{d}$ with equation 4 (see proof in Appendix A.2).

###### Theorem 2 .

Let $(p,q)\in\llbracket 1,d\rrbracket^{2}$ such that $p+q=d$, $\mathcal{T}\subseteq[d]$, be an arbitrary subset of the $d$ first natural integers, and $\mathcal{T}^{c}:=[d]\,\backslash\,\mathcal{T}$ its complement set.
Let $\hat{{\bm{w}}}\in\mathbb{R}^{d}$ such that $\hat{{\bm{w}}}_{\mathcal{T}}={\bm{X}}_{\mathcal{T}}^{+}{\bm{y}}\in\mathbb{R}^{p}$ and $\hat{{\bm{w}}}_{\mathcal{T}^{c}}=\mathbf{0}\in\mathbb{R}^{q}$.
If $({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}$, then the expected OOD risk of the predictor $\hat{f}_{\mathcal{T}}:{\bm{x}}\mapsto\phi({\bm{x}}^{T}\hat{\bm{w}})\in\hat{
\mathcal{F}}_{d}$ satisfies

$$ $\mathbb{E}_{{\bm{X}}}\bigl{[}R_{\text{OOD}}(\hat{f})\bigr{]}\geq\bigl{(} \lambda_{\min}({\bm{\Sigma}})+\lambda_{\min}({\bm{\Sigma}}^{\text{OOD}})\bigr{ )}c(n,p,\sigma^{\prime})+2\sigma^{\prime 2}$ $$

and

$$ $\mathbb{E}_{{\bm{X}}}\bigl{[}R_{\text{OOD}}(\hat{f})\bigr{]}\leq\bigl{(} \lambda_{\max}({\bm{\Sigma}})+\lambda_{\max}({\bm{\Sigma}}^{\text{OOD}})\bigr{ )}c(n,p,\sigma^{\prime})+\sigma^{\prime 2},$ $$

where $c(n,p,\sigma^{\prime})$ is defined in equation 6
and ${\bm{\Sigma}}^{\text{OOD}}\in\mathbb{R}^{d\times d}$ is the positive-definite matrix defined as

$$ ${\bm{\Sigma}}^{\text{OOD}}=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X}, \mathcal{Y}}^{\text{OOD}}}\bigl{[}{\bm{x}}{\bm{x}}^{T}\phi^{\prime}({\bm{x}}^{ T}{\bm{w}}^{*})^{2}\bigr{]}=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X}, \mathcal{Y}}^{\text{OOD}}}\Bigl{[}\tfrac{\partial\phi({\bm{x}}^{T}{\bm{w}}^{*} )}{\partial{\bm{w}}^{*}}^{T}\tfrac{\partial\phi({\bm{x}}^{T}{\bm{w}}^{*})}{ \partial{\bm{w}}^{*}}\Bigr{]}.$ $$

###### Remark 4.6 .

Like the expected risk in Theorem 1, we find $\mathbb{E}_{{\bm{X}}}\bigl{[}R_{\text{OOD}}(\hat{f})\bigr{]}=\infty$ around $p=n$, which is characteristic of a double descent phenomenon.
This results suggests that OOD scoring functions based on the prediction confidence of binary classifiers $\hat{f}(\cdot)\in\hat{\mathcal{F}}_{d}$ exhibit a double descent phenomenon similar to what has been reported for the expected risk.

###### Theorem 2 .

###### Remark 4.6 .

## 5 Experiments

In this section, we provide an empirical evaluation of different OOD detection methods with respect to the model width across multiple neural network architectures.

### 5.1 Setup

#### General Setup.

We aim to investigate whether the double descent phenomenon, widely observed in model generalization setup, also extends to OOD detection. To explore this, we perform experiments on multiple DNN architectures: ResNet-18 (He et al., ), a 4-block convolutional neural network (CNN), Vision Transformers (ViTs) (Dosovitskiy et al., ) and Swin Transformers (Liu et al., ).

#### Model Setup.

To replicate double descent, we follow the experimental setup from Nakkiran et al. (), which uses ResNet-18 as the baseline architecture. We apply a similar setup to the 4-block CNN model, ViTs and Swin. We vary the model capacity by altering the number of filters (denoted as $k$) per layer, with values ranging from $k=1$ to $k=128$. ResNet-18, which uses 64 filters, operates within the overparameterized regime. The depth of the models is kept constant to isolate the effects of width (effective model complexity). The convolutional models are trained using the cross-entropy loss function, with a learning rate of 0.0001 and the Adam optimizer for 4 000 epochs.
This extended training regime ensures that models converge for all explored model widths.
Moreover, each experiment is conducted five times (with different random seeds).
Finally, further details on the experimental setup for the Transformers are given in the Appendix B.2.

#### Label Noise.

To induce the double descent effect, we introduce label noise into the training set by randomly swapping 20% of the labels. This setup simulates real-world scenarios, where noisy data is common. The models are trained on this noisy dataset but evaluated on a clean test set. Random data augmentations, including random cropping and horizontal flipping, are applied during training.

### 5.2 Evaluation Metrics

We evaluate both generalization and OOD detection using multiple metrics:

- •
Generalization: We report the test accuracy for in-distribution (ID) classification tasks.
- •
OOD Detection: We measure OOD detection performance using the area under the receiver operating characteristic curve (AUC), which is threshold-free and widely adopted in OOD detection research. A higher AUC indicates better performance.
- •
Neural Collapse: We report NC metrics, where lower values imply better performance in terms of both generalization and OOD detection.

### 5.3 OOD Datasets

For OOD detection, we evaluate each model using six well-established OOD benchmark datasets: Textures (Cimpoi et al., ), Places365 (Zhou et al., ), iNaturalist (Van Horn et al., ), a 10 000 image subset from (Huang & Li, ), ImageNet-O (Hendrycks et al., ) and SUN (Xiao et al., ).
For experiments where CIFAR-10 (or CIFAR-100) is the in-distribution dataset, we also include CIFAR-100 (or CIFAR-10) as an additional OOD benchmark. CIFAR-10/100 contains 50 000 training images and 10 000 test images.

### 5.4 OOD Detection Methods

In order to have a discussion that generalises across different OOD Detection methods, we evaluate several
state-of-the-art methods, categorized by the information they rely on:

- •
Logit-based methods:
Maximum Softmax Probability (MSP) (Hendrycks & Gimpel, ),
Energy scores (Liu et al., ),
React (Sun et al., ),
MaxLogit and KL-Matching (Hendrycks et al., ),
- •
Feature-based methods:
Mahalanobis distance (Lee et al., )
and Residual score (Wang et al., ).
- •
Hybrid methods:
ViM (Wang et al., ),
ASH (Djurisic et al., )
and NECO (Ammar et al., ).

Although the double descent effect is observed in all of our experiments,
results from only a few representative methods will be presented in the main paper. Additional results can be found in Appendix D.

### 5.5 OOD Detection and Double Descent

#### Double Descent & OOD Detection.

The primary question addressed in this section is whether the double descent phenomenon extends to OOD detection, as suggested by our theoretical framework. The results focus on the relative performance across underparameterized and overparameterized regimes.
We conduct experiments using CIFAR-10 and CIFAR-100 as ID datasets, and assess OOD detection across increasing model widths. Figure 1 presents the evolution of generalization error and OOD detection performance (AUC) for a challenging covariate shift scenario between CIFAR-10 and CIFAR-100. Refer to
Appendix D for more results on multiple OOD datasets.
This figure illustrates a generalization double descent phenomenon in all models, with logit-based and hybrid OOD detection methods exhibiting a similar curve. This demonstrates that this phenomenon is not exclusive to generalization, but
it extends to OOD detection as well.
Moreover, the figure displays the average result (from the five runs) as well as the associated variance. These can be seen to be very narrow, which confirms the prevalence of the phenomena.

#### Feature-Based Techniques & Interpolation Threshold.

In some cases, no double descent curve is observed for feature-based techniques.
This result suggests that the double descent depends either on the used architecture or the data, as discussed in Appendix E.3.
Furthermore, we observe that the interpolation threshold is not always perfectly consistent across OOD datasets or techniques.
Those observations are consistent with the Nakkiran et al. ()’s results on the CIFAR-10 and CIFAR-100 datasets.
Those results suggest that the effective model complexity (EMC) framework (Nakkiran et al., ) defined for the generalization error can be extended to OOD detection.

#### Smaller Models for OOD Detection.

Interestingly, in many cases, smaller models are very good OOD detectors.
This suggests that in applications where model pruning or DNN simplification is important, using smaller models may offer advantages for detecting OOD samples.
Similarly, resource-constrained environments may benefit from lighter models as a viable option for robust OOD detection.
The conditions under which this choice becomes optimal will be discussed in Section 5.6.

Figure: Figure 1: Generalisation (single curve) and OOD detection (multiple curves) evolution plots w.r.t model’s width (x-axis), in terms of accuracy and AUC respectively. With CIFAR10 as ID and CIFAR100 as OOD, and for (from top-left to bottom-right) CNN, ResNet-18, ViT and Swin.
Refer to caption: x1.png

#### Discussion on OOD Methods.

It is important to note that effective OOD detection depends on two main factors: the quality of the learned representations (which shape the feature space) and a reliable confidence score. Different OOD detection methods emphasize one of these factors over the other. Logit-based methods rely primarily on the confidence score, which is determined by the model’s output logits. These logits are typically sensitive to the model size and complexity, making them closely tied to the double descent phenomenon. As a result, logit-based methods tend to exhibit smoother double descent curves, with fewer drastic shifts at the interpolation threshold.
In contrast, feature space-based techniques rely more heavily on the model’s ability to learn high-quality representations that can effectively separate ID from OOD data.
However, there is no guarantee that the quality of the latent space discriminative power will be impacted by the double descent phenomenon in the same manner as the output logits.

#### Discussion on Different Architectures.

The learned representation is highly complex, with its properties and structure determined by the intrinsic biases of the architecture. This complexity goes beyond what can be directly inferred from performance metrics alone. As a result, even though all architectures display a double descent pattern, their performance variability remains significant. Notably, some architectures, such as ResNet-18 and Swin, consistently achieve higher performance in the overparameterized regime for both of our metrics of interest. In contrast, the CNN model performs comparably in both regimes, while the ViT struggles with generalization when overparameterized.
This performance variability is expected due to the fundamental differences between architectures. To gain deeper insight into the causes of this variability, we study the model’s learned representation, which captures its intrinsic biases. In particular, we will use the Neural Collapse framework in our analysis, examining its influence in the double descent setting, and how it can inform us on the possible improvement in OOD detection performance across various architectures.

### 5.6 Representation Analysis and Neural Collapse Role

#### From Double Descent to Neural Collapse.

One of the primary arguments for the interest in double descent in DNNs is that increasing model complexity beyond the interpolation threshold can lead to improved models, compared to those found in underparameterized local minima. However, this improvement does not occur uniformly across all architectures. Although the OOD double descent curve is consistently observed, each architecture exhibits a unique behavior in both generalization and OOD detection. These differences can be attributed to the complexities of the learned representations. We analyze the learned representations using the NC framework to understand this behavior. Previous works (Papyan et al., ; Ming et al., ; Haas et al., ; Ammar et al., ) have shown that NC positively influences both generalization and OOD detection by ensuring stability and strong performance as models converge.

#### Background on Neural Collapse.

Neural Collapse (NC)
describes the convergence of model representations
during the late phases of training
towards a low-dimensional and highly structured configuration known as the Equiangular-Tight Frame Simplex (ETF). This structure is characterized by data clustering within each category, with low intra-class covariance, high inter-class separation, and equiangular and equinorm relationships between class representations. Appendix C provides further details on the Neural Collapse phenomenom.

#### NC1-based Metric for Overparameterization Analysis.

We will analyze the data clustering and separation properties by leveraging the NC1 metric on the clean test set. NC1 measures the signal-to-noise ratio, where lower values indicate more compact intra-class clustering and greater inter-class separation. The NC1 metric is computed as follows:

$$ $\operatorname{NC1}=\operatorname{Tr}\left[\frac{{\bm{\Sigma}}_{W}{\bm{\Sigma}} _{B}^{+}}{C}\right];$ $$

where ${\bm{\Sigma}}_{W}$ is the intra-class covariance matrix of the penultimate layer of the DNN that depicts noise, ${\bm{\Sigma}}_{B}$ is the inter-class covariance matrix of the penultimate layer of the DNN that represents the signal, and $C$ is the number of classes.
As the NC1 value converges towards a lower value, the activations of samples collapse toward their respective class means (see Appendix C for more details).
To quantify the influence of overparameterization on the NC1 property, we compute the following ratio:

$$ $NC1_{u/o}=\frac{NC1_{u}}{NC1_{o}},$ $$

where $NC1_{u}$ represents the NC1 value at the underparameterized local minimum, and $NC1_{o}$ is the NC1 value for the most overparameterized model.
Values of $NC1_{u/o}>1$ indicate improved data separation with increased model complexity.

#### Analysis of the Results

Table 1 shows that the $NC1_{u/o}$ ratio strongly correlates with improvements in OOD detection. Models that achieve better overparameterized NC1 values tend to improve as their complexity increases. In contrast, the CNN model either stagnates or performs worse with overparameterization, as its $NC1_{u/o}$ metric degrades.
We also observe that logit-based methods are well correlated with NC1, with the exception of the MSP method on the ViT model, due to the degradation in generalization.
Since NC1 reflects class variability collapse and improved clustering, our results suggest that the separation and clustering effects of the latent space, as measured by NC, can indicate OOD detection performance in the overparameterized regime. The ViT model is an exception, as its accuracy suffers due to the lack of pretraining.
Besides the improved clustering and data separation provided by lower NC1, NC convergence implies that the model representation has aligned with an ETF structure. This global representation structure brings more stability and can provide more guarantees to performance improvement.
In
Appendix E.1, we will study this structure through the eigenvalues to further explain the performance variability.

**Table 1: Models performance in terms of AUC in the underparametrized local minima ($\texttt{AUC}_{u}$) and the overparametrized maximum width ($\texttt{AUC}_{o}$), w.r.t $NC1_{u/o}$ value. Best is highlighted in green when $\texttt{AUC}_{o}$ is higher, red when $\texttt{AUC}_{u}$ is higher and blue if both AUC are within standard deviation range. The highest AUC value per-dataset and per-architecture is highlighted in bold.**
| Model | $NC1_{u/o}$ | Method | SUN | Places365 | CIFAR-100 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\texttt{AUC}_{u}$ $\uparrow$ | $\texttt{AUC}_{o}$ $\uparrow$ | $\texttt{AUC}_{u}$ $\uparrow$ | $\texttt{AUC}_{o}$ $\uparrow$ | $\texttt{AUC}_{u}$ $\uparrow$ | $\texttt{AUC}_{o}$ $\uparrow$ |  |  |  |
| CNN | 0.88 | Softmax score | 76.09$\pm$0.96 | 75.08$\pm$0.75 | 75.95$\pm$0.76 | 74.59$\pm$0.45 | 74.33$\pm$0.32 | 72.68$\pm$0.31 |
| MaxLogit | 72.98$\pm$2.22 | 60.13$\pm$0.35 | 73.33$\pm$2.17 | 61.25$\pm$0.42 | 73.38$\pm$0.39 | 70.37$\pm$0.34 |  |  |
| Energy | 68.08$\pm$3.66 | 59.73$\pm$0.36 | 69.00$\pm$3.56 | 60.90$\pm$0.43 | 70.78$\pm$0.73 | 70.24$\pm$0.35 |  |  |
| Energy+ReAct | 59.12$\pm$4.73 | 47.49$\pm$0.58 | 60.79$\pm$4.53 | 49.45$\pm$0.52 | 66.00$\pm$1.58 | 63.57$\pm$0.51 |  |  |
| NECO | 70.43$\pm$2.53 | 64.22$\pm$1.36 | 71.20$\pm$2.59 | 63.59$\pm$0.98 | 72.40$\pm$0.95 | 70.17$\pm$0.76 |  |  |
| ViM | 59.94$\pm$2.01 | 59.77$\pm$0.66 | 61.88$\pm$1.89 | 60.93$\pm$0.40 | 69.23$\pm$0.78 | 70.25$\pm$0.35 |  |  |
| ASH-P | 68.60$\pm$3.59 | 60.36$\pm$0.37 | 69.35$\pm$3.50 | 61.48$\pm$0.43 | 71.11$\pm$0.53 | 70.45$\pm$0.41 |  |  |
| ResNet | 1.96 | Softmax score | 71.18$\pm$0.93 | 75.82$\pm$0.89 | 71.22$\pm$0.93 | 75.52$\pm$0.88 | 71.21$\pm$0.48 | 75.37$\pm$0.42 |
| MaxLogit | 70.64$\pm$1.53 | 72.51$\pm$1.03 | 70.69$\pm$1.29 | 72.64$\pm$0.94 | 69.76$\pm$0.39 | 73.65$\pm$0.38 |  |  |
| Energy | 69.11$\pm$2.49 | 72.46$\pm$1.03 | 69.19$\pm$2.08 | 72.59$\pm$0.94 | 67.58$\pm$0.46 | 73.61$\pm$0.39 |  |  |
| Energy+ReAct | 69.57$\pm$2.35 | 71.83$\pm$0.88 | 69.63$\pm$1.93 | 71.97$\pm$0.78 | 67.25$\pm$0.91 | 72.63$\pm$0.45 |  |  |
| NECO | 70.39$\pm$2.30 | 75.60$\pm$1.56 | 70.46$\pm$1.85 | 75.20$\pm$1.42 | 69.92$\pm$0.36 | 75.28$\pm$0.50 |  |  |
| ViM | 66.54$\pm$1.99 | 74.44$\pm$0.65 | 65.38$\pm$1.99 | 73.42$\pm$0.65 | 64.61$\pm$0.47 | 71.54$\pm$0.44 |  |  |
| ASH-P | 69.11$\pm$2.49 | 71.73 $\pm$1.09 | 69.19$\pm$2.08 | 71.85$\pm$0.98 | 67.58$\pm$0.46 | 72.89$\pm$0.35 |  |  |
| Swin | 1.70 | Softmax score | 58.82$\pm$2.98 | 67.91$\pm$0.69 | 59.01$\pm$2.90 | 67.66$\pm$0.59 | 61.89$\pm$1.42 | 65.44$\pm$0.62 |
| MaxLogit | 59.91$\pm$3.11 | 70.75$\pm$0.45 | 59.84$\pm$3.40 | 70.46$\pm$0.46 | 61.79$\pm$1.73 | 66.95$\pm$0.53 |  |  |
| Energy | 60.12$\pm$3.68 | 70.79$\pm$0.42 | 58.77$\pm$3.98 | 70.50$\pm$0.44 | 55.52$\pm$2.10 | 66.92$\pm$0.51 |  |  |
| Energy+ReAct | 60.16$\pm$3.79 | 71.15$\pm$0.44 | 58.85$\pm$4.03 | 70.83$\pm$0.48 | 55.53$\pm$2.10 | 67.27$\pm$0.52 |  |  |
| NECO | 64.26$\pm$3.20 | 73.29$\pm$0.75 | 63.88$\pm$3.37 | 72.38$\pm$0.66 | 62.62$\pm$2.10 | 68.13$\pm$0.76 |  |  |
| ViM | 60.68$\pm$2.61 | 71.69$\pm$0.19 | 58.34$\pm$2.60 | 71.39$\pm$0.15 | 55.95$\pm$0.77 | 68.89$\pm$0.51 |  |  |
| ASH-P | 59.49$\pm$4.21 | 70.74$\pm$0.44 | 58.15$\pm$4.42 | 70.42$\pm$0.40 | 55.10$\pm$2.28 | 66.89$\pm$0.55 |  |  |
| ViT | 2.32 | Softmax score | 66.28$\pm$0.19 | 64.87$\pm$0.27 | 66.26$\pm$0.36 | 64.61$\pm$0.26 | 65.18$\pm$0.38 | 62.96$\pm$0.33 |
| MaxLogit | 66.09$\pm$1.48 | 70.30$\pm$0.46 | 66.13$\pm$1.50 | 69.79$\pm$0.26 | 64.60$\pm$0.35 | 66.69$\pm$0.39 |  |  |
| Energy | 64.79$\pm$2.81 | 70.50$\pm$0.48 | 64.86$\pm$2.65 | 69.98$\pm$0.26 | 63.08$\pm$0.44 | 66.79$\pm$038 |  |  |
| Energy+ReAct | 64.51$\pm$2.93 | 70.51$\pm$0.49 | 64.65$\pm$2.75 | 69.97$\pm$0.26 | 62.86$\pm$0.58 | 66.78$\pm$039 |  |  |
| NECO | 67.61$\pm$1.61 | 75.89$\pm$0.47 | 67.47$\pm$1.68 | 74.29$\pm$0.29 | 66.28$\pm$0.54 | 67.40$\pm$0.27 |  |  |
| ViM | 63.14$\pm$3.54 | 72.25$\pm$0.37 | 63.30$\pm$3.36 | 71.41$\pm$0.15 | 64.81$\pm$0.65 | 66.34$\pm$0.30 |  |  |
| ASH-P | 64.79$\pm$2.81 | 70.27$\pm$0.50 | 64.86$\pm$2.65 | 69.79$\pm$0.25 | 63.08$\pm$0.44 | 66.61$\pm$0.36 |  |  |

## 6 Conclusion

In this work, we conducted a theoretical and empirical study on the double descent phenomenon in both classification and OOD detection.
Our findings indicate that the double descent phenomenon also occurs in OOD detection, with significant implications for model performance.
We introduced the expected OOD risk to evaluate classifiers’ confidence on both training and OOD samples.
Using Random Matrix Theory, we demonstrated that both the expected risk and OOD risk of least-squares binary classifiers applied to Gaussian models exhibit an infinite peak, when the number of parameters is equal to the number of samples, which we associate with the double descent phenomenon.
Our experimental study on different OOD detection methods revealed a similar double descent phenomenon across multiple neural architectures.
However, we observed significant variability in performance among different models, with some showing no advantages from overparameterization.
Using the Neural Collapse (NC) framework, we revealed that OOD detection improves with overparameterization only when it enhances NC convergence, boosting the performance of OOD detection methods.
This emphasizes the crucial role of learned representations in the performance of overparameterized models and their significance in model selection.

We hope our insights and extensive experiments will benefit practitioners in OOD detection and inspire further theoretical research into this aspect of DNNs. Ultimately,
although this paper introduces a novel theoretical framework for understanding the double descent phenomenon in OOD detection, its theoretical scope has some limitations including a focus on binary classification, the choice of loss function, and specific model architectures. Hence solving these limitations would be a valuable direction for future work.

## Table of Contents - Supplementary Material

## Appendix A Proof of theorems

### A.1 Proof of Theorem 1

This section is dedicated to the proof of Theorem 1 and follows the same line of arguments of Theorem 1 in Belkin et al. ().

###### Theorem 3 .

Let $(p,q)\in\llbracket 1,d\rrbracket^{2}$ such that $p+q=d$, $\mathcal{T}\subseteq[d]$ an arbitrary subset of the $d$ first natural integers, and $\mathcal{T}^{c}:=[d]\,\backslash\,\mathcal{T}$ its complement set.
Let $\hat{{\bm{w}}}\in\mathbb{R}^{d}$ such that $\hat{{\bm{w}}}_{\mathcal{T}}={\bm{X}}_{\mathcal{T}}^{+}{\bm{y}}\in\mathbb{R}^{p}$ and $\hat{{\bm{w}}}_{\mathcal{T}^{c}}=\mathbf{0}\in\mathbb{R}^{q}$.
Then the expected risk with respect to the loss function $\ell:(\hat{y},y)\mapsto(\hat{y}-y)^{2}$ of the predictor $\hat{f}_{\mathcal{T}}:{\bm{x}}\mapsto\phi({\bm{x}}^{T}\hat{\bm{w}})\in\hat{
\mathcal{F}}_{d}$ satisfies

$$ $\lambda_{\min}({\bm{\Sigma}})c(n,p)+\sigma^{2}\leq\mathbb{E}_{\bm{X}}\bigl{[}R (\hat{f}_{\mathcal{T}})\bigr{]}\leq\lambda_{\max}({\bm{\Sigma}})c(n,p)+\sigma^ {2},$ $$

where

$$ $\displaystyle c(n,p)$ $\displaystyle=\begin{cases}\tfrac{n}{n-p-1}\Bigl{(}\|{\bm{w}}_{\mathcal{T}^{c} }^{*}\|_{2}^{2}+\sigma^{2}\Bigr{)}+\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}& \text{if }p\leq n-2,\\ +\infty&\text{if }n-1\leq p\leq n+1,\\ \bigl{(}1-\tfrac{n}{p}\bigr{)}\bigl{\|}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2} ^{2}+\tfrac{n}{p-n-1}\Bigl{(}\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}+\sigma ^{2}\Bigr{)}+\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}&\text{if }p\geq n+2. \end{cases}$ $$

###### Proof.

Let ${\bm{x}}\in\mathcal{X}$ and

$$ ${\bm{\Delta}}_{\bm{w}}=\hat{\bm{w}}-{\bm{w}}^{*}.$ $$

Using the Taylor’s theorem, the linear approximation of the predictor $\hat{f}_{\mathcal{T}}(\cdot)$ near ${\bm{x}}^{T}{\bm{w}}^{*}$ is equal to

$$ $\displaystyle\hat{f}_{\mathcal{T}}({\bm{x}})=\phi({\bm{x}}^{T}\hat{{\bm{w}}})$ $\displaystyle=\phi\bigl{(}{\bm{x}}^{T}({\bm{w}}^{*}+{\bm{\Delta}}_{\bm{w}}) \bigr{)}$ $\displaystyle=\phi({\bm{x}}^{T}{\bm{w}}^{*})+{\bm{x}}^{T}{\bm{\Delta}}_{\bm{w} }\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*})+o({\bm{x}}^{T}{\bm{\Delta}}_{\bm{w}})$ $\displaystyle\approx\phi({\bm{x}}^{T}{\bm{w}}^{*})+{\bm{x}}^{T}{\bm{\Delta}}_{ \bm{w}}\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*}),$ $$

where $o(\cdot)$ denotes the little-o notation.
We have

$$ $\displaystyle\mathbb{E}_{\bm{X}}\bigl{[}R(\hat{f}_{\mathcal{T}})\bigr{]}= \mathbb{E}_{{\bm{x}},{\bm{X}}}\Bigl{[}\bigl{(}\phi({\bm{x}}^{T}\hat{{\bm{w}}}) -y\bigr{)}^{2}\Bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{x}},{\bm{X}}}\Bigl{[}\bigl{(}\phi({\bm{x}}^{T} \hat{{\bm{w}}})-\phi({\bm{x}}^{T}{\bm{w}}^{*})-\epsilon\bigr{)}^{2}\Bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{x}},{\bm{X}}}\Bigl{[}\bigl{(}{\bm{x}}^{T}{\bm{ \Delta}}_{\bm{w}}\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*})\bigr{)}^{2}\Bigr{]}+ \sigma^{2}$ $\displaystyle=\mathbb{E}_{{\bm{x}},{\bm{X}}}\Bigl{[}{\bm{\Delta}}_{\bm{w}}^{T} {\bm{x}}{\bm{x}}^{T}{\bm{\Delta}}_{\bm{w}}\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{ *})^{2}\Bigr{]}+\sigma^{2}$ $\displaystyle=\operatorname{Tr}\Bigl{(}\mathbb{E}_{{\bm{x}},{\bm{X}}}\bigl{[} \phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*})^{2}{\bm{x}}{\bm{x}}^{T}{\bm{\Delta}}_{ \bm{w}}{\bm{\Delta}}_{\bm{w}}^{T}\bigr{]}\Bigr{)}+\sigma^{2}$ $\displaystyle=\operatorname{Tr}\Bigl{(}\mathbb{E}_{{\bm{x}}}\bigl{[}\phi^{ \prime}({\bm{x}}^{T}{\bm{w}}^{*})^{2}{\bm{x}}{\bm{x}}^{T}\bigr{]}\mathbb{E}_{{ \bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}{\bm{\Delta}}_{\bm{w}}^{T}\bigr{]}\Bigr{ )}+\sigma^{2}$ $\displaystyle=\operatorname{Tr}\Bigl{(}{\bm{\Sigma}}\mathbb{E}_{{\bm{X}}}\bigl {[}{\bm{\Delta}}_{\bm{w}}{\bm{\Delta}}_{\bm{w}}^{T}\bigr{]}\Bigr{)}+\sigma^{2}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{ \Sigma}}{\bm{\Delta}}_{\bm{w}}\bigr{]}+\sigma^{2}.$ $$

From the min-max theorem, we have

$$ $\lambda_{\min}({\bm{\Sigma}})\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w }}^{T}{\bm{\Delta}}_{\bm{w}}\bigr{]}+\sigma^{2}\leq\mathbb{E}_{{\bm{X}}}\bigl{ [}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{\Sigma}}{\bm{\Delta}}_{\bm{w}}\bigr{]}+\sigma ^{2}\leq\lambda_{\max}({\bm{\Sigma}})\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta} }_{\bm{w}}^{T}{\bm{\Delta}}_{\bm{w}}\bigr{]}+\sigma^{2}.$ $$

Note that for $\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{\Delta}}_{\bm{w}}
\bigr{]}$, we have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{ \Delta}}_{\bm{w}}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\|\hat{{\bm{w}}}-{\bm{w}}^{*}\|_{2} ^{2}\Bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\|\hat{{\bm{w}}}_{\mathcal{T}}-{\bm {w}}_{\mathcal{T}}^{*}\|_{2}^{2}\Bigr{]}+\mathbb{E}_{{\bm{X}}}\Bigl{[}\|\hat{{ \bm{w}}}_{\mathcal{T}^{c}}-{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}\Bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\|\hat{{\bm{w}}}_{\mathcal{T}}-{\bm {w}}_{\mathcal{T}}^{*}\|_{2}^{2}\Bigr{]}+\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2 }^{2}.$ $$

as $\hat{{\bm{w}}}_{\mathcal{T}^{c}}=\mathbf{0}$.
In the following, we provide a decomposition of $\mathbb{E}_{{\bm{X}}}\bigl{[}\|\hat{{\bm{w}}}_{\mathcal{T}}-{\bm{w}}_{\mathcal
{T}}^{*}\|_{2}^{2}\bigr{]}$.
Let ${\bm{\eta}}={\bm{y}}-{\bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}$.
Since $\hat{{\bm{w}}}_{\mathcal{T}}={\bm{X}}_{\mathcal{T}}^{+}{\bm{y}}$, we have $\hat{{\bm{w}}}_{\mathcal{T}}={\bm{X}}_{\mathcal{T}}^{+}({\bm{\eta}}+{\bm{X}}_{
\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*})$.
Therefore,

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}\|{\bm{w}}_{\mathcal{T}}^{*}-\hat{{ \bm{w}}}_{\mathcal{T}}\|_{2}^{2}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}({\bm{I}}_{p}-{\bm{X}}_{ \mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*}-{\bm{X}}_{ \mathcal{T}}^{+}{\bm{\eta}}\bigr{\|}_{2}^{2}\Bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}({\bm{I}}_{p}-{\bm{X}}_{ \mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2} ^{2}+\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}\bigr{\|}_{2}^{2}-2\bigl{ \langle}\left({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}} \right){\bm{w}}_{\mathcal{T}}^{*},{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}\bigr{ \rangle}\Bigr{]}.$ $$

Since $({\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}})^{T}={\bm{X}}_{\mathcal{T}}^
{+}{\bm{X}}_{\mathcal{T}}$ and $({\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}})^{T}{\bm{X}}_{\mathcal{T}}^{
+}={\bm{X}}_{\mathcal{T}}^{+}$, we have

$$ $\displaystyle\bigl{\langle}({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*},{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}} \bigr{\rangle}_{2}$ $\displaystyle=\bigl{(}({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*}\bigr{)}^{T}{\bm{X}}_{\mathcal{T}}^{+}{ \bm{\eta}}$ $\displaystyle={\bm{w}}_{\mathcal{T}}^{*T}({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^ {+}{\bm{X}}_{\mathcal{T}})^{T}{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}$ $\displaystyle={\bm{w}}_{\mathcal{T}}^{*T}{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}} -{\bm{w}}_{\mathcal{T}}^{*T}({\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}) {\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}$ $\displaystyle={\bm{w}}_{\mathcal{T}}^{*T}{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}} -{\bm{w}}_{\mathcal{T}}^{*T}{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}$ $\displaystyle=0.$ $$

$({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}){\bm{w}}_{
\mathcal{T}}^{*}$ and ${\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}$ are thus orthogonal.
We deduce that

$$ $\displaystyle\begin{split}\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\hat{{\bm{w}}} _{\mathcal{T}}-{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}&=\mathbb{E} _{{\bm{X}}}\Bigl{[}\bigl{\|}\left({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{ X}}_{\mathcal{T}}\right){\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}+\bigl{\|}{ \bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}\bigr{\|}_{2}^{2}\Bigr{]}\\ &=\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\left({\bm{I}}_{p}-{\bm{X}}_{\mathcal{ T}}^{+}{\bm{X}}_{\mathcal{T}}\right){\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2 }\Bigr{]}+\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{\bm {\eta}}\bigr{\|}_{2}^{2}\Bigr{]}.\end{split}$ (7) $$

Leveraging the same arguments used by Belkin et al. () to prove the existence of the double descent phenomenon in the regression problem, we distinguish two cases depending on $n$ and $p$ to derive equation 7.

###### Theorem 3 .

###### Proof.

#### Classical Regime ( p < n 𝑝 𝑛 p<n italic_p < italic_n ).

Breiman & Freedman () studied this regime for the regression problem.
In the classical regime, the Moore-Penrose inverse is equal to:

$$ $\displaystyle{\bm{X}}_{\mathcal{T}}^{+}=({\bm{X}}_{\mathcal{T}}^{T}{\bm{X}}_{ \mathcal{T}})^{-1}{\bm{X}}_{\mathcal{T}}^{T},$ $$

which implies that

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\left({\bm{I}}_{p}-{\bm{X}} _{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}\right){\bm{w}}_{\mathcal{T}}^{*}\bigr {\|}_{2}^{2}\Bigr{]}=\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\bigl{(}{\bm{I}}_{p }-({\bm{X}}_{\mathcal{T}}^{T}{\bm{X}}_{\mathcal{T}})^{-1}{\bm{X}}_{\mathcal{T} }^{T}{\bm{X}}_{\mathcal{T}}\bigr{)}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2} \Bigr{]}=0.$ $$

From equation 7, we deduce that

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\hat{{\bm{w}}}_{\mathcal{T} }-{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}=\mathbb{E}_{{\bm{X}}} \Bigl{[}\bigr{\|}{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}\bigr{\|}_{2}^{2}\Bigr{] }=\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\eta}}^{T}{\bm{X}}_{\mathcal{T}}^{+T}{\bm{ X}}_{\mathcal{T}}^{+}{\bm{\eta}}\bigr{]}=\operatorname{Tr}\Bigl{(}\mathbb{E}_{ {\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}^{+T}{\bm{X}}_{\mathcal{T}}^{+}{\bm{ \eta}}{\bm{\eta}}^{T}\bigr{]}\Bigr{)}.$ $$

We observe that

$$ $\displaystyle\begin{split}{\bm{\eta}}&={\bm{y}}-{\bm{X}}_{\mathcal{T}}{\bm{w}} _{\mathcal{T}}^{*}\\ &={\bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}+{\bm{X}}_{\mathcal{T}^{c}}{ \bm{w}}_{\mathcal{T}^{c}}^{*}+{\bm{\epsilon}}-{\bm{X}}_{\mathcal{T}}{\bm{w}}_{ \mathcal{T}}^{*}\\ &={\bm{X}}_{\mathcal{T}^{c}}{\bm{w}}_{\mathcal{T}^{c}}^{*}+{\bm{\epsilon}}, \end{split}$ (8) $$

where ${\bm{\epsilon}}=[\epsilon_{1},\ldots,\epsilon_{n}]^{T}$.
Because ${\bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}$ and ${\bm{X}}_{\mathcal{T}^{c}}{\bm{w}}_{\mathcal{T}^{c}}^{*}+{\bm{\epsilon}}$ are both uncorrelated, we have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\hat{{\bm{w}}}_{\mathcal{T} }-{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}=\operatorname{Tr}\Bigl{( }\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}^{+T}{\bm{X}}_{\mathcal{T} }^{+}\bigr{]}\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\eta}}{\bm{\eta}}^{T}\bigr{]} \Bigr{)}.$ (9) $$

We have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}^{+T}{\bm{X}}_ {\mathcal{T}}^{+}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}({\bm{X}}_{ \mathcal{T}}^{T}{\bm{X}}_{\mathcal{T}})^{-2}{\bm{X}}_{\mathcal{T}}^{T}\bigr{]}$ $$

Let ${\bm{S}}$ be the unique positive definite square root of ${\bm{X}}_{\mathcal{T}}^{T}{\bm{X}}_{\mathcal{T}}$ and ${\bm{\Psi}}={\bm{X}}_{\mathcal{T}}{\bm{S}}^{-1}$, an $n\times p$ matrix and ${\bm{X}}_{\mathcal{T}}={\bm{\Psi}}{\bm{S}}$.
${\bm{\Psi}}$ is an orthonormal such that ${\bm{\Psi}}^{T}{\bm{\Psi}}={\bm{I}}_{p}$ and ${\bm{\Psi}}{\bm{\Psi}}^{T}={\bm{I}}_{n}$.
Following the same arguments than Breiman & Freedman (), we obtain

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}^{+T}{\bm{X}}_ {\mathcal{T}}^{+}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}({\bm{X}}_{ \mathcal{T}}^{T}{\bm{X}}_{\mathcal{T}})^{-2}{\bm{X}}_{\mathcal{T}}^{T}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Psi}}{\bm{S}}{\bm{S}}^{-4}{ \bm{S}}{\bm{\Psi}}^{T}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Psi}}{\bm{S}}^{-2}{\bm{\Psi}} ^{T}\bigr{]}$ $\displaystyle={\bm{\Psi}}\mathbb{E}_{{\bm{X}}}\bigl{[}({\bm{X}}_{\mathcal{T}}^ {T}{\bm{X}}_{\mathcal{T}})^{-1}\bigr{]}{\bm{\Psi}}^{T}.$ $$

${\bm{X}}_{\mathcal{T}}^{T}{\bm{X}}_{\mathcal{T}}\in\mathbb{R}^{p\times p}$ follows a Wishart distribution: ${\bm{X}}_{\mathcal{T}}^{T}{\bm{X}}_{\mathcal{T}}\sim\mathcal{W}_{p}(n,{\bm{I}}
_{p})$, where $\mathcal{W}_{p}(n,{\bm{I}}_{p})$ denotes a Wishart distribution with $n$ degrees of freedom and scale matrix ${\bm{I}}_{p}$.
The inverse of a Wishart-distributed matrix $\mathcal{W}_{p}(n,{\bm{I}}_{p})$ follows an inverse Wishart distribution: $({\bm{X}}_{\mathcal{T}}^{T}{\bm{X}}_{\mathcal{T}})^{-1}\sim\mathcal{W}_{p}^{-1
}(n,{\bm{I}}_{p})$, where $\mathcal{W}_{p}^{-1}(n,{\bm{I}}_{p})$ denotes an inverse Wishart distribution with $n$ degrees of freedom and scale matrix ${\bm{I}}_{p}$.
As a consequence,

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{(}{\bm{X}}_{\mathcal{T}}^{T}{ \bm{X}}_{\mathcal{T}}\bigr{)}^{-1}\Bigr{]}=\tfrac{1}{n-p-1}{\bm{I}}_{p}$ $$

and thus

$$ $\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}^{+T}{\bm{X}}_{\mathcal{T}} ^{+}\bigr{]}=\tfrac{1}{n-p-1}{\bm{I}}_{p}.$ (10) $$

Putting equation 10 into equation 9, we get

$$ $\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\hat{{\bm{w}}}_{\mathcal{T}}-{\bm{w}}_{ \mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}=\tfrac{1}{n-p-1}\mathbb{E}_{{\bm{X}} }\bigl{[}{\bm{\eta}}^{T}{\bm{\eta}}\bigr{]}.$ $$

From equation 8, we have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\eta}}^{T}{\bm{\eta}}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\bigl{[}\bigl{(}{\bm{y}}-{\bm{X}}_{\mathcal {T}}{\bm{w}}_{\mathcal{T}}^{*}\bigr{)}^{T}\bigl{(}{\bm{y}}-{\bm{X}}_{\mathcal{ T}}{\bm{w}}_{\mathcal{T}}^{*}\bigr{)}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{(}{\bm{X}}_{\mathcal{T}^{c}}{ \bm{w}}_{\mathcal{T}^{c}}^{*}+{\bm{\epsilon}}\bigr{)}^{T}\bigl{(}{\bm{X}}_{ \mathcal{T}^{c}}{\bm{w}}_{\mathcal{T}^{c}}^{*}+{\bm{\epsilon}}\bigr{)}\Bigr{]}$ $\displaystyle={\bm{w}}_{\mathcal{T}^{c}}^{*T}\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm {X}}_{\mathcal{T}^{c}}^{T}{\bm{X}}_{\mathcal{T}^{c}}\bigr{]}{\bm{w}}_{\mathcal {T}^{c}}^{*}+\underbrace{\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\epsilon}}^{T}\bigr {]}}_{=\mathbf{0}}\underbrace{\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{ T}^{c}}\bigr{]}}_{=\mathbf{0}}{\bm{w}}_{\mathcal{T}^{c}}^{*}+{\bm{w}}_{ \mathcal{T}^{c}}^{*T}\underbrace{\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{ \mathcal{T}^{c}}^{T}\bigr{]}}_{=\mathbf{0}}\underbrace{\mathbb{E}_{{\bm{X}}} \bigl{[}{\bm{\epsilon}}\bigr{]}}_{=\mathbf{0}}+\mathbb{E}_{{\bm{X}}}\bigl{[}{ \bm{\epsilon}}^{T}{\bm{\epsilon}}\bigr{]}.$ $$

${\bm{X}}_{\mathcal{T}^{c}}^{T}{\bm{X}}_{\mathcal{T}^{c}}$ follows a Wishart distribution, i.e., ${\bm{X}}_{\mathcal{T}^{c}}^{T}{\bm{X}}_{\mathcal{T}^{c}}\sim\mathcal{W}_{q}(n,
{\bm{I}}_{q})$, where $\mathcal{W}_{q}(n,{\bm{I}}_{q})$ denotes the Wishart distributions with $n$ degrees of freedom and scale matrix ${\bm{I}}_{q}$, respectively.
We obtain thus

$$ $\displaystyle\begin{split}\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\eta}}^{T}{\bm{ \eta}}\bigr{]}&=n{\bm{w}}_{\mathcal{T}^{c}}^{*T}{\bm{w}}_{\mathcal{T}^{c}}^{*} +n\sigma^{2}\\ &=n\lVert{\bm{w}}_{\mathcal{T}^{c}}^{*}\rVert_{2}^{2}+n\sigma^{2}.\end{split}$ (11) $$

As a consequence, we obtain

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\hat{{\bm{w}}}_{\mathcal{T} }-{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{F}^{2}\Bigr{]}=\tfrac{n}{n-p-1}\Bigl{(} \|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}+\sigma^{2}\Bigr{)}.$ $$

#### Interpolating Regime ( p ≥ n 𝑝 𝑛 p\geq n italic_p ≥ italic_n ).

The interpolating regime has been considered in Belkin et al. () for the regression problem.
We can first observe that:

$$ ${\bm{w}}_{\mathcal{T}}^{*}=({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*}+{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}$ $$

and

$$ $\displaystyle\bigl{\langle}({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*},{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\rangle}=0.$ $$

Indeed, since ${\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}=({\bm{X}}_{\mathcal{T}}^{+}{
\bm{X}}_{\mathcal{T}})({\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}})$, we have

$$ $\displaystyle\bigl{\langle}({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*},{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\rangle}$ $\displaystyle=\bigl{(}{\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*}\bigr{)}^{T}({\bm{X}}_{\mathcal{T}}^{+} {\bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*})$ $\displaystyle={\bm{w}}_{\mathcal{T}}^{*T}\bigl{(}{\bm{X}}_{\mathcal{T}}^{+}{ \bm{X}}_{\mathcal{T}}-({\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}})({\bm{ X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}})\bigr{)}{\bm{w}}_{\mathcal{T}}^{*}$ $\displaystyle=0.$ $$

We deduce that $({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}){\bm{w}}_{
\mathcal{T}}^{*}$ and ${\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}$ are orthogonal.
From the Pythagorean theorem, we have

$$ $\displaystyle\bigl{\|}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}=\bigl{\|}({ \bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}){\bm{w}}_{ \mathcal{T}}^{*}\bigr{\|}_{2}^{2}+\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_ {\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}$ $$

and thus

$$ $\displaystyle\bigl{\|}({\bm{I}}_{p}-{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{ \mathcal{T}}){\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}=\bigl{\|}{\bm{w}}_{ \mathcal{T}}^{*}\bigr{\|}_{2}^{2}-\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_ {\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}.$ $$

Putting the equation above into equation 7, we obtain

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\hat{{\bm{w}}}_{\mathcal{T} }-{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}=\mathbb{E}_{{\bm{X}}} \Bigl{[}\bigl{\|}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}-\mathbb{E }_{{\bm{X}}}\Bigl{[}\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}{ \bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}+\mathbb{E}_{{\bm{X}}}\Bigl{ [}\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}\bigr{\|}_{2}^{2}\Bigr{]}.$ (12) $$

Note that ${\bm{\Pi}}_{\mathcal{T}}={\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}={\bm
{X}}_{\mathcal{T}}^{T}({\bm{X}}_{\mathcal{T}}{\bm{X}}_{\mathcal{T}}^{T})^{-1}{
\bm{X}}_{\mathcal{T}}$ is the orthogonal projection matrix for the row space of ${\bm{X}}_{\mathcal{T}}$.
We can thus write ${\bm{X}}_{\mathcal{T}}^{+}{\bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}={\bm{
\Pi}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}$ as a linear combination of rows of ${\bm{X}}_{\mathcal{T}}$.
Then, using the fact that the ${\bm{x}}_{i}$ in ${\bm{X}}$ are i.i.d. and drawn from a standard normal distribution and by rotational symmetry of the standard normal distribution, it follows that we have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{ \bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}= \tfrac{n}{p}\bigl{\|}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}$ $$

and thus

$$ $\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^ {2}\Bigr{]}-\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{ \bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}=\bigl {\|}{\bm{w}}_{\mathcal{T}}^{*}\bigr{\|}_{2}^{2}\bigl{(}1-\tfrac{n}{p}\bigr{)}.$ $$

For $\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{\bm{\eta}}
\bigr{\|}_{2}^{2}\Bigr{]}$ in equation 12, we have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{ \bm{\eta}}\bigr{\|}_{2}^{2}\Bigr{]}=\operatorname{Tr}\Bigl{(}\mathbb{E}_{{\bm{ X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}^{+T}{\bm{X}}_{\mathcal{T}}^{+}\bigr{]} \mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\eta}}{\bm{\eta}}^{T}\bigr{]}\Bigr{)}.$ $$

As $p>n$, we have ${\bm{X}}_{\mathcal{T}}^{+}={\bm{X}}_{\mathcal{T}}^{T}({\bm{X}}_{\mathcal{T}}{
\bm{X}}_{\mathcal{T}}^{T})^{-1}$ and thus

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{\mathcal{T}}^{+T}{\bm{X}}_ {\mathcal{T}}^{+}\bigr{]}=\mathbb{E}_{{\bm{X}}}\bigl{[}({\bm{X}}_{\mathcal{T}} {\bm{X}}_{\mathcal{T}}^{T})^{-1}\bigr{]}$ $$

Similarly, ${\bm{X}}_{\mathcal{T}}{\bm{X}}_{\mathcal{T}}^{T}$ follows a Wishart distribution: ${\bm{X}}_{\mathcal{T}}{\bm{X}}_{\mathcal{T}}^{T}\sim\mathcal{W}_{n}(p,I_{n})$, and $({\bm{X}}_{\mathcal{T}}{\bm{X}}_{\mathcal{T}}^{T})^{-1}$ follows an inverse Wishart distribution: $({\bm{X}}_{\mathcal{T}}{\bm{X}}_{\mathcal{T}}^{T})^{-1}\sim\mathcal{W}_{n}^{-1
}(p,{\bm{I}}_{n})$.
Its expectation is given by:

$$ $\displaystyle\mathbb{E}[({\bm{X}}_{\mathcal{T}}{\bm{X}}_{\mathcal{T}}^{T})^{-1 }]=\tfrac{{\bm{I}}_{n}}{p-n-1}$ $$

From equation 11, we deduce that

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}{\bm{X}}_{\mathcal{T}}^{+}{ \bm{\eta}}\bigr{\|}_{2}^{2}\Bigr{]}$ $\displaystyle=\operatorname{Tr}\Bigl{(}\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{X}}_{ \mathcal{T}}^{+T}{\bm{X}}_{\mathcal{T}}^{+}\bigr{]}\mathbb{E}_{{\bm{X}}}\bigl{ [}{\bm{\eta}}{\bm{\eta}}^{T}\bigr{]}\Bigr{)}$ $\displaystyle=\tfrac{n}{p-n-1}\Bigl{(}\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{ 2}+\sigma^{2}\Bigr{)}.$ $$

For equation 12, using equations above, we have

$$ $\mathbb{E}_{{\bm{X}}}\Bigl{[}\bigl{\|}\hat{{\bm{w}}}_{\mathcal{T}}-{\bm{w}}_{ \mathcal{T}}^{*}\bigr{\|}_{2}^{2}\Bigr{]}=\bigl{\|}{\bm{w}}_{\mathcal{T}}^{*} \bigr{\|}_{2}^{2}\bigl{(}1-\tfrac{n}{p}\bigr{)}+\tfrac{n}{p-n-1}\Bigl{(}\|{\bm {w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}+\sigma^{2}\Bigr{)}.$ $$

∎

### A.2 Proof of Theorem 2

This section is dedicated to the proof of Theorem 2.

###### Theorem 4 .

Let $(p,q)\in\llbracket 1,d\rrbracket^{2}$ such that $p+q=d$, $\mathcal{T}\subseteq[d]$ an arbitrary subset of the $d$ first natural integers, and $\mathcal{T}^{c}:=[d]\,\backslash\,\mathcal{T}$ its complement set.
Let $\hat{{\bm{w}}}\in\mathbb{R}^{d}$ such that $\hat{{\bm{w}}}_{\mathcal{T}}={\bm{X}}_{\mathcal{T}}^{+}{\bm{y}}\in\mathbb{R}^{p}$ and $\hat{{\bm{w}}}_{\mathcal{T}^{c}}=\mathbf{0}\in\mathbb{R}^{q}$.
If $({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}}$, then the expected OOD risk of the predictor $\hat{f}_{\mathcal{T}}:{\bm{x}}\mapsto\phi({\bm{x}}^{T}\hat{\bm{w}})\in\hat{
\mathcal{F}}_{d}$ satisfies

$$ $\mathbb{E}_{{\bm{X}}}\bigl{[}R_{\text{OOD}}(\hat{f})\bigr{]}\geq\bigl{(} \lambda_{\min}({\bm{\Sigma}})+\lambda_{\min}({\bm{\Sigma}}^{\text{OOD}})\bigr{ )}c(n,p,\sigma^{\prime})+2\sigma^{\prime 2}$ $$

and

$$ $\mathbb{E}_{{\bm{X}}}\bigl{[}R_{\text{OOD}}(\hat{f})\bigr{]}\leq\bigl{(} \lambda_{\max}({\bm{\Sigma}})+\lambda_{\max}({\bm{\Sigma}}^{\text{OOD}})\bigr{ )}c(n,p,\sigma^{\prime})+\sigma^{\prime 2},$ $$

where $c(n,p,\sigma^{\prime})$ is defined in equation 6
and ${\bm{\Sigma}}^{\text{OOD}}\in\mathbb{R}^{d\times d}$ is the positive-definite matrix defined as

$$ ${\bm{\Sigma}}^{\text{OOD}}=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X}, \mathcal{Y}}^{\text{OOD}}}\bigl{[}{\bm{x}}{\bm{x}}^{T}\phi^{\prime}({\bm{x}}^{ T}{\bm{w}}^{*})^{2}\bigr{]}=\mathbb{E}_{{\bm{x}}}\Bigl{[}\tfrac{\partial\phi({ \bm{x}}^{T}{\bm{w}}^{*})}{\partial{\bm{w}}^{*}}^{T}\tfrac{\partial\phi({\bm{x} }^{T}{\bm{w}}^{*})}{\partial{\bm{w}}^{*}}\Bigr{]}.$ $$

###### Proof.

For ease of notation, we denote the weight vector ${\bm{w}}^{*}_{\text{OOD}}$ defined in equation 4.1 by ${\bm{w}}^{*}$.
The layout of the proof is similar to the proof of Theorem 1.
Let ${\bm{x}}\in\mathcal{X}$ and

$$ ${\bm{\Delta}}_{\bm{w}}=\hat{\bm{w}}-{\bm{w}}^{*}.$ $$

Using the Taylor’s theorem, the linear approximation of the predictor $\hat{f}_{\mathcal{T}}(\cdot)$ near ${\bm{x}}^{T}{\bm{w}}^{*}$ is equal to

$$ $\displaystyle\hat{f}_{\mathcal{T}}({\bm{x}})=\phi({\bm{x}}^{T}\hat{{\bm{w}}})$ $\displaystyle=\sigma\bigl{(}{\bm{x}}^{T}({\bm{w}}^{*}+{\bm{\Delta}}_{\bm{w}}) \bigr{)}$ $\displaystyle=\phi({\bm{x}}^{T}{\bm{w}}^{*})+{\bm{x}}^{T}{\bm{\Delta}}_{\bm{w} }\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*})+o({\bm{x}}^{T}{\bm{\Delta}}_{\bm{w}})$ $\displaystyle\approx\phi({\bm{x}}^{T}{\bm{w}}^{*})+{\bm{x}}^{T}{\bm{\Delta}}_{ \bm{w}}\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*}),$ $$

where $o(\cdot)$ denotes the little-o notation.
We have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}R_{\text{OOD}}(\hat{f})\bigr{]}$ $\displaystyle=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}},{ \bm{X}}}\bigl{[}(2\hat{f}({\bm{x}})-1-z)^{2}\bigr{]}+\mathbb{E}_{({\bm{x}}, \cdot)\sim P_{\mathcal{X},\mathcal{Y}}^{\text{OOD}},{\bm{X}}}\bigl{[}(2\hat{f} ({\bm{x}})-1-z)^{2}\bigr{]}$ $\displaystyle=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}},{ \bm{X}}}\Bigl{[}\bigl{(}2\bigl{(}\phi({\bm{x}}^{T}\hat{{\bm{w}}})-\phi({\bm{x} }^{T}{\bm{w}}^{*})\bigr{)}-\epsilon^{\prime}\bigr{)}^{2}\Bigr{]}$ $\displaystyle\quad+\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y} }^{\text{OOD}},{\bm{X}}}\Bigl{[}\bigl{(}\bigl{(}2\bigl{(}\phi({\bm{x}}^{T}\hat {{\bm{w}}})-\phi({\bm{x}}^{T}{\bm{w}}^{*})\bigr{)}-\epsilon^{\prime}\bigr{)}^{ 2}\Bigr{]}$ $\displaystyle=\mathbb{E}_{({\bm{x}},\cdot)\sim P_{\mathcal{X},\mathcal{Y}},{ \bm{X}}}\Bigl{[}\bigl{(}2{\bm{x}}^{T}{\bm{\Delta}}_{\bm{w}}\phi^{\prime}({\bm{ x}}^{T}{\bm{w}}^{*})\bigr{)}^{2}\Bigr{]}+\mathbb{E}_{({\bm{x}},\cdot)\sim P_{ \mathcal{X},\mathcal{Y}}^{\text{OOD}},{\bm{X}}}\Bigl{[}\bigl{(}2{\bm{x}}^{T}{ \bm{\Delta}}_{\bm{w}}\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{*})\bigr{)}^{2}\Bigr{ ]}+2\sigma^{\prime 2}$ $\displaystyle=\operatorname{Tr}\Bigl{(}\bigl{(}\mathbb{E}_{({\bm{x}},\cdot) \sim P_{\mathcal{X},\mathcal{Y}}}\bigl{[}\phi^{\prime}({\bm{x}}^{T}{\bm{w}}^{* })^{2}{\bm{x}}{\bm{x}}^{T}\bigr{]}+\mathbb{E}_{({\bm{x}},\cdot)\sim P_{ \mathcal{X},\mathcal{Y}}^{\text{OOD}}}\bigl{[}\phi^{\prime}({\bm{x}}^{T}{\bm{w }}^{*})^{2}{\bm{x}}{\bm{x}}^{T}\bigr{]}\bigr{)}\mathbb{E}_{{\bm{X}}}\bigl{[}{ \bm{\Delta}}_{\bm{w}}{\bm{\Delta}}_{\bm{w}}^{T}\bigr{]}\Bigr{)}+2\sigma^{ \prime 2}$ $\displaystyle=\operatorname{Tr}\Bigl{(}\bigl{(}{\bm{\Sigma}}+{\bm{\Sigma}}^{ \text{OOD}}\bigr{)}\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}{\bm{ \Delta}}_{\bm{w}}^{T}\bigr{]}\Bigr{)}+2\sigma^{\prime 2}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{ \Sigma}}{\bm{\Delta}}_{\bm{w}}+{\bm{\Delta}}_{\bm{w}}^{T}{\bm{\Sigma}}^{\text{ OOD}}{\bm{\Delta}}_{\bm{w}}\bigr{]}+2\sigma^{\prime 2}.$ $$

From the min-max theorem, we have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{ \Sigma}}{\bm{\Delta}}_{\bm{w}}+{\bm{\Delta}}_{\bm{w}}^{T}{\bm{\Sigma}}^{\text{ OOD}}{\bm{\Delta}}_{\bm{w}}\bigr{]}\geq\bigl{(}\lambda_{\min}({\bm{\Sigma}})+ \lambda_{\min}({\bm{\Sigma}}^{\text{OOD}})\bigr{)}\mathbb{E}_{{\bm{X}}}\bigl{[ }{\bm{\Delta}}_{\bm{w}}^{T}{\bm{\Delta}}_{\bm{w}}\bigr{]}$ $$

and

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{ \Sigma}}{\bm{\Delta}}_{\bm{w}}+{\bm{\Delta}}_{\bm{w}}^{T}{\bm{\Sigma}}^{\text{ OOD}}{\bm{\Delta}}_{\bm{w}}\bigr{]}\leq\bigl{(}\lambda_{\max}({\bm{\Sigma}})+ \lambda_{\max}({\bm{\Sigma}}^{\text{OOD}})\bigr{)}\mathbb{E}_{{\bm{X}}}\bigl{[ }{\bm{\Delta}}_{\bm{w}}^{T}{\bm{\Delta}}_{\bm{w}}\bigr{]}.$ $$

Note that for $\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{\Delta}}_{\bm{w}}
\bigr{]}$, we have

$$ $\displaystyle\mathbb{E}_{{\bm{X}}}\bigl{[}{\bm{\Delta}}_{\bm{w}}^{T}{\bm{ \Delta}}_{\bm{w}}\bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\|\hat{{\bm{w}}}-{\bm{w}}^{*}\|_{2} ^{2}\Bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\|\hat{{\bm{w}}}_{\mathcal{T}}-{\bm {w}}_{\mathcal{T}}^{*}\|_{2}^{2}\Bigr{]}+\mathbb{E}_{{\bm{X}}}\Bigl{[}\|\hat{{ \bm{w}}}_{\mathcal{T}^{c}}-{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2}^{2}\Bigr{]}$ $\displaystyle=\mathbb{E}_{{\bm{X}}}\Bigl{[}\|\hat{{\bm{w}}}_{\mathcal{T}}-{\bm {w}}_{\mathcal{T}}^{*}\|_{2}^{2}\Bigr{]}+\|{\bm{w}}_{\mathcal{T}^{c}}^{*}\|_{2 }^{2}.$ $$

as $\hat{{\bm{w}}}_{\mathcal{T}^{c}}=\mathbf{0}$.
The remainder of the proof follows the proof of Theorem 1 with ${\bm{\eta}}={\bm{y}}-{\bm{X}}_{\mathcal{T}}{\bm{w}}_{\mathcal{T}}^{*}$, where ${\bm{w}}^{*}_{\text{OOD}}={\bm{w}}^{*}$.
∎

###### Theorem 4 .

###### Proof.

## Appendix B Details about baselines and architectures implementation

### B.1 Baselines OOD methods

In this section, we present an overview of the baseline methods used in our experiments.We describe the principles behind these baselines, and the chosen hyperparameters. It is worth noting that extensive hyperparameter search for each method were not performed to maintain stability. Hence, once the final model is selected, hybrid methods like ViM, ASH and NECO performance may increase if such task is performed

#### Softmax Score.

This score uses the maximum softmax probability (MSP) of the model as an OOD scoring function (Hendrycks & Gimpel, ).

#### Energy.

Liu et al. () proposes using the energy score for OOD detection, where the energy function maps the logit outputs to a scalar. To maintain the convention that lower scores correspond to in-distribution (ID) data, (Liu et al., ) uses the negative energy as the OOD score.

#### ReAct.

Sun et al. () propose clipping extreme-valued activations. The original paper found that clipping activations at the $90th$ percentile of ID data was optimal. Moreover, as the authors propose, we employ the ReAct+Energy configuration.

#### KL-Matching & MaxLogit.

KL-Matching computes the class-wise average probability using the entire training dataset. Consistent with the approach outlined in (Hendrycks et al., ), this calculation is based on the predicted class rather than the ground-truth labels. MaxLogit employs the maximum logit value of the model as an OOD scoring function.

#### Mahalanobis.

This score leverages the feature vector from the layer preceding the final classification layer (Lee et al., ). To estimate the precision matrix and the class-wise mean vector, we used the entire training dataset. It’s important to note that we incorporated ground-truth labels during this computation process.

#### ViM & Residual.

Wang et al. () decomposes the latent space into a principal space $P$ and a null space $P^{\perp}$. The ViM score is calculated by projecting the features onto the null space to create a virtual logit, which is then combined with the logits using the norm of this projection. To enhance performance, they calibrate this norm with a constant which is determined by dividing the sum of the maximum logits by the sum of the norms of the null space projections, both measured on the training set.
The Residual score is derived by computing the norm of the latent vector’s projection onto the null space.
We followed the author’s suggestions for the null space, by setting it to half the size of the full feature vector, adapted to each model width.

#### ASH.

Djurisic et al. () employs activation pruning at the penultimate layer, just before the application of the DNN classifier. This pruning threshold is determined on a per-sample basis, eliminating the need for pre-computation of ID data statistics. The original paper presents three different post-hoc scoring functions, with the only distinction among them being the imputation method applied after pruning.
We employ ASH-P in our experiments as it performed the best, in which the clipped values are replaced with zeros. As specified in the original paper, we fix the pruning threshold value to $90\%$.

#### NECO.

Ammar et al. () leverages the geometric properties of Neural Collapse, measuring the relative norm of a sample within the subspace defined by the ETF to identify OOD samples. NC typically involves a collapse in the variability of class representations, leading to a more structured and simplified feature space. It is hypothesized that this collapse also impacts OOD detection, particularly through the emerging orthogonality between ID and OOD data. NECO utilizes this orthogonality to effectively distinguish between ID and OOD data by measuring the relative norm of each data point within the approximated ETF space scaled by the maximum logit value as the OOD score. We use a dimension $d=c$ to approximate the ETF sub-space for all architectures, with $c$ being the number of classes.

### B.2 Experiments setup

#### ViT Experimental Setup.

For all experiments, we trained a set of ViT models with widths
[4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 60, 80, 100, 120, 160, 200, 240, 280, 320, 360, 400, 480, 520, 600, 680, 760, 800]. The width is used as the last dimension of the output layer after the linear transformation (the class-token size). The dimension of the FeedForward layer is the width multiplied by 4. The input size is set to 32 and the patch size to 8, no dropout is used and we use 4 heads with 4 Transformer blocks.
The ViT models are first randomly initialized and then trained on CIFAR-10 using stochastic gradient descent with CE loss. The weights are fine-tuned for 60 000 steps, with no warm-up steps, 1 024 batch size, 0.9 momentum, and a learning rate of 0.03.

#### Swin Experimental Setup.

We used a standard 4 block Swin architecture, with a downscaling factor of (2,2,2,1) for each block respectively. The width ranges from 1 to 100, with a window size of 4, an input size of 32, and a filter size of 4.
The model is randomly initialized and then optimized using an Adam optimizer with CE loss for a 1 000 epoch using a batch size of 1 024. The initial learning rate is 0.0001.

#### CNN Experimental Setup.

Similar to Nakkiran et al. (), we define a standard family CNN models formed by 4 convolutional stages of controlled base width [$k$, $2k$, $4k$, $8k$], for $k$ in the range of [1, 128], with a fully connected layer as classifier. The MaxPool is set to [2, 2, 2, 4] for the four blocks respectively. For all the convolution layers, the kernel size is set to 3, stride and padding to 1.

## Appendix C Details about Neural collapse

For overparametrised model trained through the terminal phase of training (TPT), Neural Collapse (NC) phenomenon emerges, particularly in the penultimate layer and in the linear classifier of DNNs (Papyan et al., ; Ammar et al., ). NC is characterized by five main properties:

- 1.
Variability Collapse (NC1):
the within-class variation in activations becomes negligible as each activation collapses toward its respective class mean.
- 2.
Convergence to Simplex ETF (NC2): the class-mean vectors converge to having equal lengths, as well as having equal-sized angles between any pair of class means. This configuration corresponds to a Simplex Equiangular Tight Frame (ETF).
- 3.
Convergence to Self-Duality (NC3):
in the limit of an ideal classifier, the class means and linear classifiers of a neural network converge to each other up to rescaling, implying that the decision regions become geometrically similar and that the class means lie at the centers of their respective regions.
- 4.
Simplification to Nearest Class-Center (NC4): The network classifier progressively tends to select the class with the nearest class mean for a given activation, typically based on standard Euclidean distance.
- 5.
ID/OOD Orthogonality (NC5): As the training procedure advances, OOD and ID data tend to become increasingly more orthogonal to each other. In other words, the clusters of OOD data become more perpendicular to the configuration adopted by ID data (i.e., the Simplex ETF).

These NC properties provide valuable insights into DNNs learned representation structure and properties, which allows for a considerable simplification. Additionally, the convergence of NC provides guarantees in terms of OOD detection.
For further details refer to (Papyan et al., ; Ammar et al., )

## Appendix D Complementary Results on Double descent and OOD detection

### D.1 CIFAR-10 additional results

To further show the consistency of double descent for OOD detection, Figures D.1, D.2 D.3, D.4 and D.5 show the OOD detection metrics performance on six more semantic-shift OOD datasets. To illustrate the performance of other OOD-methods while maintaining visibility, we show two different methods at each dataset alongside the better-performing and most stable three: MSP, NECO, and ASH.

Figure: Figure D.1: OOD detection evolution curve w.r.t model’s width (x-axis) in terms AUC. With CIFAR10 as ID and ImageNet-O as OOD, for (from top-left to bottom-right) CNN, ResNet-18, ViT, and Swin.
Refer to caption: x9.png

Figure: Figure D.2: OOD detection evolution curve w.r.t model’s width (x-axis) in terms AUC. With CIFAR10 as ID and Textures as OOD, for (from top-left to bottom-right) CNN, ResNet-18, ViT and Swin.
Refer to caption: x13.png

Figure: Figure D.3: OOD detection evolution curve w.r.t model’s width (x-axis) in terms AUC. With CIFAR10 as ID and inaturalist as OOD, for (from top-left to bottom-right) CNN, ResNet-18, ViT and Swin.
Refer to caption: x17.png

Figure: Figure D.4: OOD detection evolution curve w.r.t model’s width (x-axis) in terms AUC. With CIFAR10 as ID and SUN as OOD, for (from top-left to bottom-right) CNN, ResNet-18, ViT, and Swin.
Refer to caption: x21.png

Figure: Figure D.5: OOD detection evolution curve w.r.t model’s width (x-axis) in terms AUC. With CIFAR10 as ID and places365 as OOD, for (from top-left to bottom-right) CNN, ResNet-18, ViT and Swin.
Refer to caption: x25.png

### D.2 CIFAR-100 results

In this section, we present results for the CIFAR-100 dataset as ID for a ResNet-18 model.
Figure D.6 illustrates the OOD detection metrics performance, Figure D.7 shows the accuracy and eigenvalues distribution (see section E.1 for discussion about eigenvalues).
We can observe similar behaviors between the ResNet-18 trained on CIFAR-10, and this current configuration on a harder dataset (CIFAR-100).

Figure: Figure D.6: OOD detection evolution curve w.r.t model’s width (x-axis) in termsAUC. For a ResNet-18 model with CIFAR-100 as ID and (from top-left to bottom-right) CIFAR-10, ImageNet-O, Texture, inaturalist, SUN and places365 as OOD.
Refer to caption: x29.png

Figure: Figure D.7: Generalisation evolution curve (left) w.r.t model’s width (x-axis), for a ResNet-18 model with CIFAR-100 as ID. Eigenvalues explained variance distribution (right) in the overparametrized regime for ResNet-18 (width 64), with CIFAR-100 as ID. Black line represents the $100^{th}$ eigenvalue.
Refer to caption: x35.png

Table 2 illustrates the evolution of AUC between the underparametrized and overparametrized regime and its correlation with the $NC1_{u/o}$ for the remaining OOD datasets. As the CNN’s $NC1_{u/o}<1$, its performance stagnates or deteriorates with overparametrization, while the other models improve. Additionally, we can see how the hybrid-based methods improve considerably and become competitive with logit-methods when $NC1_{u/o}>1$.

**Table 2: Models performance in terms of AUC in the underparametrized local minima ($\texttt{AUC}_{u}$) and the overparametrized maximum width ($\texttt{AUC}_{o}$), w.r.t $NC1_{u/o}$ value. Best is highlighted in green when $\texttt{AUC}_{u}$ is higher, red when $\texttt{AUC}_{u}$ is higher and blue if both AUC are within standard deviation range. The highest AUC value per-dataset and per-architecture is highlighted in bold.**
| Model | $NC1_{u/o}$ | Method | ImageNet-O | Textures | iNaturalist |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\texttt{AUC}_{u}$ $\uparrow$ | $\texttt{AUC}_{o}$ $\uparrow$ | $\texttt{AUC}_{u}$ $\uparrow$ | $\texttt{AUC}_{o}$ $\uparrow$ | $\texttt{AUC}_{u}$ $\uparrow$ | $\texttt{AUC}_{o}$ $\uparrow$ |  |  |  |
| CNN | 0.88 | Softmax score | 71.60+0.37 | 71.78$\pm$0.57 | 70.76$\pm$0.99 | 74.80$\pm$0.78 | 66.19$\pm$1.27 | 72.07$\pm$1.56 |
| MaxLogit | 68.91$\pm$0.96 | 65.94$\pm$0.51 | 62.01$\pm$3.75 | 45.56$\pm$1.06 | 63.38$\pm$1.89 | 61.58$\pm$2.07 |  |  |
| Energy | 65.06$\pm$1.57 | 65.76 $\pm$0.51 | 53.62$\pm$5.87 | 45.07$\pm$1.04 | 60.11$\pm$2.32 | 61.32$\pm$2.05 |  |  |
| Energy+ReAct | 59.06$\pm$2.60 | 58.08$\pm$0.80 | 42.80$\pm$7.54 | 32.67$\pm$0.86 | 51.17$\pm$2.82 | 49.41$\pm$2.47 |  |  |
| NECO | 67.45$\pm$1.14 | 68.89$\pm$0.35 | 56.54$\pm$3.83 | 55.75$\pm$3.63 | 59.79$\pm$1.96 | 68.52$\pm$3.60 |  |  |
| ViM | 67.13$\pm$1.61 | 65.76$\pm$0.51 | 49.84$\pm$3.59 | 45.09$\pm$0.79 | 51.58$\pm$2.82 | 61.37$\pm$1.96 |  |  |
| ASH-P | 66.20$\pm$1.48 | 66.26$\pm$0.57 | 56.07$\pm$5.94 | 46.23$\pm$1.07 | 60.73$\pm$2.83 | 61.52$\pm$2.11 |  |  |
| ResNet | 1.96 | Softmax score | 70.90$\pm$0.52 | 75.91$\pm$0.49 | 68.03$\pm$0.75 | 72.78 $\pm$1.14 | 65.79$\pm$2.16 | 74.85$\pm$1.39 |
| MaxLogit | 69.45$\pm$0.88 | 72.50$\pm$0.88 | 65.06$\pm$1.80 | 64.43$\pm$2.04 | 63.88$\pm$3.47 | 72.82$\pm$2.32 |  |  |
| Energy | 67.36$\pm$1.26 | 72.44$\pm$0.89 | 62.28$\pm$2.55 | 64.35$\pm$1.84 | 61.73$\pm$4.97 | 72.77$\pm$2.33 |  |  |
| Energy+ReAct | 68.02$\pm$1.41 | 72.00$\pm$0.81 | 64.15$\pm$2.68 | 66.13$\pm$1.20 | 61.68$\pm$6.83 | 71.85$\pm$2.60 |  |  |
| NECO | 70.42$\pm$0.83 | 76.11$\pm$1.42 | 67.56$\pm$1.84 | 73.18$\pm$3.20 | 64.50$\pm$3.51 | 75.12$\pm$2.21 |  |  |
| ViM | 70.94$\pm$1.54 | 75.03$\pm$0.62 | 77.88$\pm$1.66 | 81.02$\pm$1.42 | 62.56$\pm$4.91 | 67.12$\pm$2.44 |  |  |
| ASH-P | 67.36$\pm$1.26 | 71.57$\pm$0.94 | 62.28$\pm$2.55 | 62.96$\pm$2.11 | 61.73$\pm$4.97 | 71.72$\pm$2.28 |  |  |
| Swin | 1.70 | Softmax score | 56.62$\pm$3.15 | 64.78$\pm$1.48 | 49.71$\pm$3.38 | 63.51$\pm$0.33 | 49.10$\pm$6.81 | 60.26$\pm$2.08 |
| MaxLogit | 55.86$\pm$3.95 | 64.70$\pm$2.07 | 49.47$\pm$4.18 | 60.29$\pm$0.38 | 49.22$\pm$5.07 | 58.91$\pm$1.81 |  |  |
| Energy | 49.58$\pm$5.48 | 64.56$\pm$2.05 | 49.49$\pm$6.73 | 59.96$\pm$0.46 | 51.61$\pm$7.96 | 58.73$\pm$1.75 |  |  |
| Energy+ReAct | 49.81$\pm$5.51 | 65.43$\pm$2.09 | 50.66$\pm$6.07 | 62.38$\pm$0.24 | 51.87$\pm$7.51 | 59.39$\pm$1.83 |  |  |
| NECO | 57.63$\pm$4.14 | 68.19$\pm$2.71 | 56.50$\pm$3.64 | 68.06$\pm$0.66 | 49.09$\pm$5.14 | 62.58$\pm$2.21 |  |  |
| ViM | 65.18$\pm$1.83 | 73.45$\pm$2.25 | 84.47$\pm$1.46 | 78.67$\pm$1.65 | 67.86$\pm$2.44 | 63.83$\pm$2.63 |  |  |
| ASH-P | 49.46$\pm$5.66 | 64.48$\pm$2.11 | 48.96$\pm$7.85 | 59.90$\pm$0.48 | 51.33$\pm$10.79 | 58.69$\pm$1.99 |  |  |
| ViT | 2.32 | Softmax score | 64.17$\pm$0.64 | 63.64$\pm$0.80 | 67.73$\pm$1.82 | 70.27$\pm$0.36 | 52.79$\pm$0.77 | 58.23$\pm$0.51 |
| MaxLogit | 63.15$\pm$0.88 | 68.87$\pm$0.52 | 67.22$\pm$2.25 | 79.25$\pm$0.38 | 51.90$\pm$1.95 | 61.28$\pm$0.94 |  |  |
| Energy | 61.24$\pm$1.08 | 69.10$\pm$0.50 | 65.59$\pm$2.78 | 79.68$\pm$0.38 | 51.11$\pm$3.04 | 61.40$\pm$0.97 |  |  |
| Energy+ReAct | 61.30$\pm$1.21 | 69.09$\pm$0.49 | 65.64$\pm$2.86 | 79.68$\pm$0.38 | 51.63$\pm$4.61 | 61.39$\pm$0.98 |  |  |
| NECO | 65.83$\pm$1.35 | 69.95$\pm$0.37 | 69.41$\pm$2.16 | 77.32$\pm$0.49 | 52.50$\pm$2.05 | 62.96$\pm$0.89 |  |  |
| ViM | 68.76$\pm$1.52 | 67.82$\pm$0.55 | 65.58$\pm$2.69 | 74.13$\pm$0.38 | 56.41$\pm$3.81 | 60.42$\pm$0.90 |  |  |
| ASH-P | 61.24$\pm$1.08 | 68.96$\pm$0.48 | 65.59$\pm$2.78 | 79.39$\pm$0.42 | 51.11$\pm$3.04 | 61.28$\pm$0.99 |  |  |

## Appendix E Model representations and Neural collapse analysis

### E.1 Structure of the model representation

Convergence towards Neural Collapse is a clear indicator of improved model representation, which progressively converges towards a highly structured manifold (as defined by the ETF structure). As a consequence, the model can be expected to be more stable and perform better.

The model eigenvalues distribution can help observing how the model representation aligns with the ETF. The properties of this structure implies that the top $c$ eigenvalues are equally prominent, while the remaining eigenvalues should be less influential. Figure E.1 shows the distribution of each model eigenvalues at the overparametrized regime. It is worth noting that all overparametrized model widths show similar curves per-architecture.
ResNet-18 and Swin models follow the expected NC pattern, with a steep drop in importance at the $c^{th}$ eigenvalue indicating the limit of the ETF. However, ViT and CNN show a slowly decaying curve, which indicates a lack of clear separation in the model representation between highly important ID information and noisy features. This lack of a global structure in their representation, results in both models failing to reliably outperform their underpametrized minima.

This highlights the importance of the ETF structure and NC convergence in enhancing representation stability and overall performance. Establishing a clear distinction between informative and noisy features aids both in accurate ID classification and in effectively separating ID data from OOD samples.

Figure: Figure E.1: Eigenvalues explained variance distribution in the overparametrized region for ((from top-left to bottom-right)) CNN (width 64), ResNet-18 (width 64), ViT (width 400) and Swin (width 50) from left to right respectively, all with CIFAR-10 as ID. Black line represents the $10^{th}$ eigenvalue.
Refer to caption: x37.png

### E.2 Evolution of NC1

In this section, we further analyse the evolution of NC1 and the model’s learned representation with overparametrization, to further show their correlation.
Our analysis will focus primarily on the ResNet and CNN models, due to their similarities.
We will not address the transformer-based models whose performance, especially for generalization, were lower than those of ResNet or CNN.
This is because transformers typically require extensive pre-training, particularly for small datasets, and this was not the case for our experiments.

In order to visualize the variability collapse predicted by NC1, Figure E.2 shows the last-layer activations for both models at their optimal underparameterized and overparameterized widths. In ResNet, transitioning to overparameterized models leads to significant improvements in the compactness and separation of ID clusters, as well as enhanced orthogonality with OOD points. In contrast, the CNN model does not show clear improvements in ID compactness or OOD separation, making it difficult to determine which representation is better.

The same phenomenon is shown in Figure E.3, where the NC1 metric is shown against the model widths for both ResNet and CNN.
While both models exhibit a double descent pattern, CNN barely matches its underparameterized metric value, whereas ResNet continuously improves with added complexity. This discrepancy in NC convergence with overparametrization explains why only ResNet benefits from increased complexity, suggesting that without improvement in NC, increasing model complexity provides no benefits for the learned representations.

Figure: Figure E.2: Visualization of the last-layer activations on the test set for ResNet and CNN in the underparametrized local minima and the overparametrized width $128$ model, with cifar10 as ID and cifar100 as OOD dataset. ID point are shown in colors and OOD in black. ResNet underparamterized (left), ResNet overparametrized (middle left), CNN underparametrized (middle right) CNN overparametrized (right).
Refer to caption: extracted/5976901/figs/NC/plot_test_4000_epoch_cifar10_CNN_noisy_start_seed_3_width_7_OOD_cifar100.png

Figure: Figure E.3: NC1 metric evolution (Log scale), w.r.t model width increase. ResNet is shown in blue, CNN in red. Dashed lines represent the interpolation threshold for each model with matching color.
Refer to caption: x41.png

### E.3 Mahalanobis and Residual joint performance

We noticed that for all architectures, and on all OOD datasets, Mahalanobis and Residual follow the same evolution curve, with usually slightly higher AUC in favor of Mahalanobis. This behaviour is intriguing, due to the fact that each method relies on different types of information. While Mahalanobis models the ID distribution, i.e., the principal space, Residual relies on computing the null space norm, which is orthogonal to the principal space.

We associate this behavior with the noise isolation in each architecture, which is specific to the double descent training paradigm.
Indeed, in order for models to be able to perfectly interpolate all the training data and achieve (almost) zero training error, noisy samples must be represented closer to their assigned (noisy) label, rather than to their true label.
This will cause the train class clusters (using the true labels) to be less compact and separable, making their high-likelihood region to span almost the entire principal space, in which the ID data is represented.
Hence, to separate ID from OOD, learning the Mahalanobis GMM (fitted on the train data) becomes equivalent to separating the principal and null space, which is the same reasoning behind the Residual score.

This overfitting occurs at the interpolation threshold, which causes the learned distributions by Mahalanobis to be sparse and not robust to OOD data, impeding its improvement as we transition towards overparametrization. It is important to note also that both of these methods are usually below, or struggle to surpass the random choice threshold of $0.5$ AUC in the overparametrized regime (with the exception of texture dataset on ResNet-18 case).

Interestingly, both of these methods suffer much less from this behaviour under the Transformer based architecture, and even exhibit a double descent curve on most datasets.
This can be explained by the fact that even the most overparametrized Transformer variant have an error higher than 4%, considerably higher than the training error lower than $0.01\%$ that convolutional models consistently achieve.
Hence, Transformers suffer less from this effect because they have not interpolated the noise in the training data perfectly.
It is worth noting that interpolating the noise is desirable, as it is necessary for generalisation in this setup (Bartlett et al., ). Transformer-based architectures require extensive pre-training to generalise well, especially for small scale dataset, which was not performed in our experiments. This inability of transformers to perfectly interpolate the training data contributes to their lower performance in terms of generalisation in the overparametrized regime, especially in the ViT case.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.

(Update: increased score from 6 to 8 after author rebuttal)
soundness: 3
strengths: - This paper provides a notable contribution by extending the concept of double descent, originally associated with generalization error, to OOD detection. This new perspective highlights a less explored aspect of model overparameterization, offering valuable insights into the robustness of overparameterized models.

- Theoretical contribution: The use of Random Matrix Theory to derive bounds for OOD risk offers valuable insights on understanding OOD detection in the over-paramterized regime.

- The experiments are quite comprehensive, particularly in the variety of OOD detection methods employed.
summary: The paper investigates the influence of model complexity on Out-Of-Distribution (OOD) detection in machine learning models, with a specific focus on the double descent phenomenon. The authors propose a theoretical framework using Random Matrix Theory to explore OOD risks associated with binary least-squares classifiers on Gaussian data. The experimental analysis extends these theoretical insights across various neural network architectures, suggesting that overparameterization does not always enhance OOD detection.
weaknesses: - Adding the label noise (Line 305) is a bit artificial for real-world applications, probably also leads to different phenomenon in OOD detection. Is there any result on noiseless experiments for understanding the effect of model size for OOD detection?

- Could the authors provide additional remarks on connecting the derived theoretical results and the empirical OOD detection methods? E.g., is the derived lower bound in Theorem related to one/several of the practical OOD detection methods?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.  Can you comment on the significance of your theoretical results? It seems that theorem 2 is a simple extension of theorem 1 applied to both ID and OOD distributions, and theorem 1 closely resembles theorem 1 in Belkin et al. (2020), with the addition of the activation function/non-linearity $\phi(\cdot)$ (as also acknowledged in the paper). But do you use $\phi(\cdot)$ to link OOD detection and overparameterization for your overall conclusion? In other words, without $\phi(\cdot)$, what is the technical challenge for extending theorem 1 in Belkin et al. (2020) to the OOD risk?

2.  Your theorem doesn’t specify assumptions on the OOD distribution. Would your theory provide any insights on how the expected behavior of the OOD risk might change with varying degrees of distributional shift between ID and OOD distributions?

3.  The trace operation in the NC1 metric is applied to $d\times d$ matrices ($d$ being the dimension of last-layer features). Thus, I suspect that the value of NC1 would be sensitive to changes in the value of $d$. Is the last-layer dimension in your experiments in both the under- and over-parameterized settings similar?  Or do you vary them as you increase the widths of the intermediate layers of the network? In the latter case, I am wondering if comparing the NC1 values when $d$ is different is fair. 

4.  The paper makes heuristic connections between how much over-parameterization improves OOD detection and how well the features align with ETF. ETF is only predictive of the last-layer's features' geometry if the training data is balanced (e.g., see [1]). Are all your training datasets balanced across classes?  If yes, would OOD detection under overparameterization behave differently if the training data were imbalanced?

[1] Thrampoulidis et al. "Imbalance trouble: Revisiting neural-collapse geometry." Advances in Neural Information Processing Systems 35 (2022)
soundness: 3
strengths: The paper is written clearly. The authors discuss connections to previous work well and provide extensive experiments across various metrics, architectures, and datasets to support their claims.
summary: This paper examines the impact of model complexity on (supervised) out-of-distribution (OOD) detection, showing that, similar to in-distribution (ID) generalization, OOD detection exhibits a double descent behavior as models become overparameterized.

Theoretically, the authors show that, in a setup similar to Belkin et al. (2020), both ID and OOD detection risks follow a double descent pattern in linear least-squares regression. Empirically, they explore various datasets, architectures, and OOD detection techniques, showing that logit-based metrics produce a smoother double descent curve compared to feature-based ones. They also relate the improvement seen with overparameterization to improvements in neural collapse (NC) within the feature space.
weaknesses: (see below)

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: (1) I feel that Eqn. (5) is unclear, as $z$ depends on $x$. The current format gives the impression that $z$ is fixed in both terms.

(2) Why Gaussian distribution is necessary for the proof?

(3) Could you provide more details for understanding remark 4.6? Also, how Theorems 1 & 2 are related to the OOD detection method?
soundness: 2
strengths: Overall, this paper presents a robust theoretical framework for analyzing the OOD detection problem and introduces a valid method supported by theoretical derivations and extensive numerical studies.
summary: This paper investigates the influence of model complexity on the out-of-distribution (OOD) detection problem by examining its connection to the double descent phenomenon. Specifically, This paper proposes a novel OOD risk metric and establishes bounds for the excess risk of both test samples and OOD samples, assuming Gaussian data. Leveraging these theoretical findings, this paper demonstrates that the model achieves optimal performance when the number of parameters matches the number of samples. Extensive experiments have been conducted across multiple neural network architectures to validate their claims.
weaknesses: The main weakness lies in the strong assumptions on both the data and the model. 

1. The model is a one-hidden-layer network with a single neuron.

2. The data is assumed to be Gaussian and isotropic. First, the Gaussian distribution may not reflect the real data application.

3. From lines 830 to 835, the nonlinear function is approximated by a linear function. However, the validity of this approximation is unclear—specifically, the conditions under which this approximation is valid and the rationale behind deriving it. Furthermore, this simplification may reduce the analysis to a purely linear case, potentially limiting the paper's contribution.

===============================

(comments after rebuttal) I changed my score from 6 to 5 (borderline reject) because I believe this paper is not yet ready for publication. 

1. The Gaussian input assumption.
2. The simplicity of the model, which involves only one hidden layer with a single neuron.
3. A new concern was raised during discussions with the authors: the limited technical contributions compared to existing works, which also focus on linear models.
4. The current version contains multiple mistakes in the proof.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.	In Section 5.2, the article states, “We report NC metrics, where lower values imply better performance in terms of both generalization and Out-of-Distribution (OOD) detection.” I find this statement insufficiently rigorous. While NC serves as a paradigm to illustrate the optimization process in deep learning, its lower values do not necessarily imply better performance regarding generalization. However, the authors carefully demonstrate the relationship between NC and generalization. Consequently, I remain skeptical of the authors' position and believe that this statement lacks the necessary rigor.
2.	I have concerns about the conclusion of Remark 4.1 of expected OOD risk metric. If the proposed expression (5) yields a lower value, it should not be possible to simultaneously derive conclusions (1) and (2). A more detailed explanation would be appreciated. I would appreciate a more detailed explanation.
3.	Since the Double Descent phenomenon is also based on a linear model, could the depth of the model be related to this phenomenon? Or does the model complexity mentioned in the context of Double Descent pertain solely to the model's width? Additionally, what is the relationship between the Double Descent phenomenon and the depth of the model?
4.	Please refer to “Weaknesses”.
soundness: 2
strengths: 1.	This work focuses on two recent cutting-edge concepts: OOD detection and the Double Descent phenomenon. Both concepts have significant implications for deep learning applications and the understanding of deep learning optimization.
2.	The paper provides clear and rigorous results for OOD detection and the Double Descent phenomenon.
3.	Both the theoretical and experimental results appear to be valid.
summary: This work focuses on exploring the Double Descent phenomenon in Out-of-Distribution (OOD) Detection. Theoretically, it proposes an expected OOD risk metric and, by leveraging Random Matrix Theory, establishes the relationship between OOD detection and the Double Descent phenomenon within a linear binary classification framework. Experimentally, the study conducts a comprehensive empirical investigation across various datasets and models. In the experiments, model complexity is indicated by different filter sizes, allowing for the observation of the Double Descent phenomenon in OOD detection under varying filter configurations. The conclusion is valid; however, the novelty of the work is not particularly significant.
weaknesses: 1.	Although the issues considered are cutting-edge concepts, I believe that the motivation presented is weak. The paper does not explore in depth the connections between these two concepts; it primarily provides derivations and experiments based on existing results. Maybe, the authors could discuss potential implications of their findings for model selection in OOD detection tasks, or to analyze how the double descent phenomenon might inform the design of more robust OOD detection methods.
2.	The motivation for expected OOD risk metric is unclear to me; I would appreciate a more detailed explanation. The authors could provide examples of how this metric might be used in practice, or to compare it with existing OOD detection metrics to highlight its unique contributions.
3.	Experimentally, the use of different filter sizes to represent model complexity is appropriate, but it also introduces certain limitations in the experimental results.

Meta-review: This paper investigates a double descent phenomenon for out-of-distribution (OOD) detection. Similar to Belkin's original double descent work (2019, 2020), the authors theoretically demonstrate the phenomenon in least squares classifiers with strict assumptions on the training data distribution. The authors back up the theory in this simplified setting with experiments with neural networks that identify a double descent curve corresponding to the under-parameterized and over-parameterized regimes.

Connecting out-of-distribution detection with double descent is novel and will likely be of interest to the OOD community. The general finding may not be surprising given the strong connection between Belkin's proof techniques and this paper. While the paper's theory aims to identify a peak at the interpolation threshold, this paper would be significantly strengthened if the authors made assumptions about the OOD distribution, which would allow for a theoretical characterization of the double descent curve away from the peak.

During the rebuttal period, the authors made non-trivial changes to the proofs in the paper, both to simplify assumptions and to correct errors pointed out by the reviewers. While all of the changes seem benign, the fact that such errors existed in the first place gives me pause, especially since the theorems generally follow the pattern of Belkin et al. (2020). I suggest another round of careful review to ensure the changes are theoretically sound.

Altogether, this paper could benefit from another round of reviewing. This would ensure that the theory is correct while allowing the authors to potentially extend their analysis to incorporate the OOD distribution, ultimately leading to a stronger and more impactful paper.

Decision: Reject