Deep Neural Networks Tend To Extrapolate Predictably
Authors: Katie Kang
Amrith Setlur
Claire Tomlin
Sergey Levine
Venue: ICLR 2024 poster (2024)
Keywords: extrapolation
OOD generalization
deep neural networks
decision-making
Primary area: general machine learning (i.e., none of the above)

Abstract: Conventional wisdom suggests that neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. Our work reassesses this assumption for neural networks with high-dimensional inputs. Rather than extrapolating in arbitrary ways, we observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. Moreover, we find that this value often closely approximates the optimal constant solution (OCS), i.e., the prediction that minimizes the average loss over the training data without observing the input. We present results showing this phenomenon across 8 datasets with different distributional shifts (including CIFAR10-C and ImageNet-R, S), different loss functions (cross entropy, MSE, and Gaussian NLL), and different architectures (CNNs and transformers). Furthermore, we present an explanation for this behavior, which we first validate empirically and then study theoretically in a simplified setting involving deep homogeneous networks with ReLU activations. Finally, we show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.

Published as a conference paper at ICLR 2024
DEEP NEURAL NETWORKS TEND TO EXTRAPOLATE PREDICTABLY
Katie Kang', Amrith Setlur", Claire Tomlin', Sergey Levine UC Berkeley, CCarnegie Mellon University
ABSTRACT
Conventional wisdom suggests that neural network predictions tend to be unpre- dictable and overconfident when faced with out-of-distrbuution (OOD) inputs. Our work reassesses this assumption for neural networks with high-dimensional inputs. Rather than extrapolating in arbitrary ways, we observe that neural network predic- tions often tend towards a constant value as input data becomes increasingly OOD. Moreover, we find that this value often closely approximates the optimal constant solution (OCS), i.e., the prediction that minimizes the average loss over the training data without observing the input. We present results showing this phenomenon across 8 datasets with different distributional shifts (including CIFARIO- and ImageNet-R, S), different loss functions (cross entropy, MSE, and Gaussian NLL), and different architectures (CNNs and transformers). Furthermore, we present an explanation for this behavior, which we first validate empirically and then study theoretically simplified setting involving deep homogeneous networks with ReLU activations. Finally, we show how one can leverage our insights in practice to enable risk-sensitive decision-maker in the presence of OOD inputs.
INTRODUCTION
The prevailing belief in machine learning posits that deep neural networks behave erratically when presented with -uttof-distribution (OOD) inputs, often yielding predictions that are not only incorrect, but incorrect with high confidence (Guo et al., 2017; Nguyen et al., 2015). However, there is some evidence which seemingly contradicts this conventional wisdom - for example, Hendrycks & Gimpel (2016) show that the softmax probabilities outputted by neural network classifiers actually tend be less confident on OOD inputs, making them surprisingly effective OOD detectors. In our work, we find that this softmax behavior may be reflective of a more general pattern the way neural networks extrapolate: as inputs diverge further from the training distribution, neural network's 's predictions often converge towards a fixed constant value. Moreover, this constant value often approximates the best prediction the network can produce without observing any inputs, which we refer to as the optimal constant solution (OCS). We call this the "reversion to the OCS" hypothesis:
Neural networks predictions on high--imennsooal 0OD inputs tend to revert towards the optimal constant solution.
In classification, the OCS corresponds to the marginal distribution of the training labels, typically a high-entropy distribution. Therefore, our hypothesis posits that classifier outputs should become higher-entropy as the input distribution becomes more OOD, which is consistent with the findings in Hendrycks & Gimpel (2016). Beyond classification, to the best of our knowledge, we are the first present and provide evidence for the "reversion to the OCS" hypothesis in its full generality. Our experiments show that the amount of distributional shift correlates strongly with the distance between model outputs and the OCS across 8 datasets, including both vision and NL.P domains, loss functions, and for both CNNS and transformers. We additionally report instances where this phenomenon did not hold, including for adversarial inputs.
Having made this observation, we set out to understand why neural networks have tendency to behave this way. Our empirical analysis reveals that the feature representations corresponding to
Correspondence to: katiekang @wecs berkeley.edu Code: https: //t:::///tttiii thub. cwwp.cccm/ee/eeemmoeeooeeeeeeeoe wwt.eaatieeaangggeeeeeoe 998 i ous_ext on

---

Published as a conference paper at ICLR 2024
In-distribution
#
OCS
"Cheee you
qreat peccate benswee
Inputs
Representations
Outputs
Out-of-iittrbution
vitue bomel p01t
cataphect carp all ovei
Meat litte bate leanturieg
Figure 1: A summary of findings. On in-distribution data (top), neural network outputs tend to vary significantly based input labels. In contrast, OOD data (bottom), observe that model predictions tend not only be more similar to one another, also gravitate towards optimal constant solution (OCS). also observe that OOD inputs tend to map to representations with smaller magnitudes, leading to predictions largely dominated by the (constant) network biases, which may shed light why neural networks have this tendency.
OOD inputs tend to have smaller norms than those of in-distribution inputs, leading to less signal being propagated from the input. a result, neural network outputs from OOD inputs tend to be dominated by the input-independent parts of the network (c.g.. bias vectors at each layer), which observe to often map closely to the OCS, We also theoretically analyze the extrapolation behavior of deep homogeoneous networks with ReLU activations, and derived evidence which supports this mechanism in the simplified setting.
Lastly, we leverage observations to propose simple strategy enable risk-sensitive decision- making in the face of OOD inputs. The OCS can be viewed as "backup default output" to which the neural network reverts when it encounters novel inputs. If we design the loss function such that the OCS aligns with the desirable cautious behavior as dictated by the decision-making problem, then the neural network model will automatically produce cautious decisions when its inputs are OOD, We describe a way to enable this alignment, and empirically demonstrate that this simple strategy can yield surprisingly good results OOD selective classification.
2 RELATED WORK
A large body of prior works have studied various properties of neural network extrapolation. One line work focuses on the failure modes of neural networks when presented with OOD inputs, such as poor generalization and overconfidence (Torralba & Efros, 2011; Gulrajani & Lopez-Paz, 2020; Recht et al., 2019; Ben-David et al., 2006; Koh et al., 2021). Other works have noted that neural networks are ineffective in capturing epistemic uncertainty in their predictions (Ovadia al., 2019;
Lakshminarayanal et al., 2017: Nalisnick et al., 2018; Guo et al., 2017; Gal & Ghahramani, 2016), and that number of techniques high confidence (Szegedy et al., 2013; Nguyen et al., 2015; Papernot et al., 2016; Hein et al., 2019).
manipulate neural networks produce incorrect predictions with
However, Hendrycks et al. (2018) observed that neural networks assign lower maximum softmax probabilities to OOD than to in-distribution point, meaning neural networks may actually exhibit less confidence on OOD inputs. Our work supports this observation, while further generalizing to arbitrary loss functions. Other lines of research have explored OOD detection via the norm of the learned features (Sun et al., 2022; Tack et al., 2020), the influence of architectural decisions on generalization (Xu et al., 2020; Yehudai et al., 2021; Cohen-Karli et al., 2022; Wu et al., 2022), the relationship between in-distribution and OOD performance (Miller et al., 2021; Back al., 2022; Balestriero et al., 2021). and the behavior of neural network representations under OOD conditions (Webb et al., 2020; Idnani et al., 2022; Pearce et al., 2021; 1; Dietterich & Guyer, 2022; Huang et al., 2020). While our work also analyzes representations in the context of extrapolation. our focus is on understanding the mechanism behind "reversion to the OCS", which differs from the aforementioned works.
2

---

Published as a conference paper at ICLR 2024
Rotation:
Rotation:
30
Rosation: 60
focaiasi
Noise: 0
Noise: 2
Noisc:
6
6
No Input
No
Input
Eniropy
Gaussiun
Caussian
Figure 2: Neural network predictions from training with cross entropy and Gaussian NLL on MNIST (left) and CIFARIO (right). The models were trained with 0 rotation/noise. and evaluated on increasingly OOD inputs consisting of the digit 6 for MNIST, and of automobiles for CIFARIO. The blue plots represent the average model prediction over the evaluation datasct. The orange plots show the OCS associated with cach model. We can sec that as the distribution shift increases, the network predictions tend towards the OCS.
Our work also explores risk-sensitive decision decision-making using selective classification as a testbed. Selective classification is well-studied problem, and various methods have been proposed to enhance selective classification performance (Geifman & EI- Yaniv, 2017; Feng et al., 201 1; Charoenphakdee al., 2021; Ni al., 2019; Cortes al., 2016; Xia & Bouganis, 2022). In contrast, our aim is not to develop the best possible selective classification approach, but rather to providing insights into the effective utilization of neural network predictions in OOD decision-making.
REVERSION TO THE OPTIMAL CONSTANT SOLUTION
In this work, we will focus on the widely studied covariate shift setting (Gretton al., 2009; Sugiyama et al., 2007). Formally, let the training data D {(x,, y4 1:)) be generated by sampling Hy ~ Prrain (x) and y/ ~ P(yla:). At test time, we query the model with inputs generated from (oo # Pamin(2) (z), whose ground truth labels are generated from the same conditional distribution, P))]z), as that in training. We will denote a neural network model as fe R + R", where d and are the dimensionalities of the input and output, and A θ represents the network weights. We will focus on settings where d is figh-dimensiona.. The neural network weights are optimized by minimizing a loss function L using gradient descent, 0 = arg min N Σn [(fe(x:), 34).
Main hypothesis. our experiments, we observed that as inputs become more OOD, neural network predictions tend to revert towards constant prediction. This means that, assuming there is little label shift, model predictions will tend be more similar to one another for OOD inputs than for the training distribution. Furthermore, we find that this constant prediction is often similar to the optimal constant solution (OCS), which minimizes the training loss if the network is constrained to ignore input. More precisely, we define the OCS as Joonstant arg min yE3 N EiSien L(J, v:). Based on observations, we hypothesize that the likelihood of samples from POOD (x) under Puraiu (a) decreases, (I) for x ~ POOD (2) tends to approach föonstant* As an illustrative example, we trained models using either cross-entropy (NLL) on the MNIST and CIFAR 10 datasets. The blue plots in Fig. 2 show the models' predictions its inputs become increasingly OOD, and the orange plots visualize the OCS associated with each model. We can see that even though we trained on different datasets and evaluated on different kinds of distribution shifts, the neural network predictions exhibit the same pattern of extrapolation: as the distribution shift increases, the network predictions move closer to the OCS.
cartinuouu-valledd Gaussian negative log-likelihood
3.1 EXPERIMENTS
We will now provide empirical evidence for the "reversion to the OCS" hypothesis. Our experiments aim to answer the question: As the test-time inputs become more OOD, do neural network predictions move closer to the optimal constant solution?
Experimental setup. We trained models on 8 different datasets, and evaluated them on both natural and synthetic distribution shifts: CIFAR IECARRO//IFARRO 1O-C, ImageNet/ ImageNet-R(enditee and Sketch, DomainBed OfficeHome, SkinLesionPPxelss, UTKFace, BREEDS living- 17, BREEDS non-living-22 and WILDS Amazon. See Appendix B.1 for a more detailed description of each dataset. Models with image inputs ResNet (He et al., 2016) VGG (Simonyan & Zisserman,
3

---

Published as a conference paper at ICLR 2024
In-distribution
Out-of-distribution
CIFARIO-C
ImageNet
Dist. to 00S
CE
MSE
Dist. to OCS
CE
MSE
Stat Nuses Motion Dile
OOD Score
ede GT
OOD Score
(35
12 es
315 OOD Score
OOD Score
111
XX
OfficeHome
IkinLesionPixch
FTKe
Dist. to OCS
CE
MSE
Caussian NLL
Dist, to OCS
Gaussian NLL
1.6
Ar Plaae
Dist.
OOD Score
OOD Score
OOD Score
OOD Score
living-17 17
non- mion-living-22
26
Amazon
Dist. to OCS
CE
MSE
CE
MSE
CE
MSE
L3
28
135
1.3
50
the OOD Score
eal Li
OOD Score
OOD Score
OOD Score
OOD Score
OOD Score
Figure 3: Evaluating the distance between network predictions and the OCS the input distribution becomes more OOD, Each point represents different evaluation dataset, with the red star representing the (holdout) training distribution, and circles representing OOD datasets. The vertical line associated with each point represents the standard deviation over 5 training runs. As the OOD score of the evaluation dataset increases, there is clear trend of the neural network predictions approaching the OCS.
2014) style architectures, and models with text inputs use DistilBERT (Sanh et al., 2019), a distilled version of BERT (Devlin et al., 2018).
We focus on three tasks, each using a different loss functions: classification with cross entropy (CE), selective classification with mean squared error (MSE), and regression with Gaussian NLL. Datasets with discrete labels are used for classification and selective classification, and datasets with continuous labels are used for regression. The cross entropy models are trained to predict the likelihood that the input belongs to each class, as is typical in classification. The MSE models are trained to predict rewards for selective classification task. More specifically, the models output a value for each class as well as an abstain option, where the value represents the reward of selecting that option given the input. The ground truth reward is +1 for the correct class, -4 for the incorrect classes, and +0 for abstaining. We train these models by minimizing the MSE loss between the predicted and ground truth rewards. We will later use these models for decision-nen -aaiing in Section The Gaussian NI.I models predict mean and a standard deviation, parameterizing Gaussian distribution. They are trained to minimize the negative log likelihood of the labels under its predicted distributions.
Evaluation protocol. To answer our question, we need to quantify (I) the dissimilarity between the training data and the evaluation data, and (2) the proximity of network predictions to the OCS. To esti- mate the former, we trained a low-capacity model to discriminate between the training and evaluation datasets and measured the average predicted likelihood that the evaluation dataset is generated from the evaluation distribution, which we refer to the OOD score. This score is 0.5 for indistinguishable train and evaluation data, and 1 for a perfect discriminator. To estimate the distance between the model's prediction and the OCS, we compute the KI. divergence between the model's predicted distribution and the distribution parameterized by the OCS, 1 ΣN DKL (y)x))):))] P/ (v)), for models trained with cross-entropy and Gaussian NLL. For MSE models, the distance is measured N our evaluation protocol, and Appendix B.4 for the closed form solution for the OCS for each loss.
using the mean squared error, 1 EN [)fo) fconstamil 1| See Appendix B.3 for more details on
Results. In Fig. 3, we plot the OOD score (x-axis) against the distance between the network predictions and the OCS (y-axis) for both the training and OOD datasets. Our results indicate a clear trend: as the OOD score of the evaluation dataset increases, neural network predictions move closer
4

---

Published as a conference paper at ICLR 2024
MNIST (CB)
MNIST (CB)
MINIST (CE)
MINIST (MSE)
Rotation
75
Normalized
Woy Nore
1.
8.9
Projectro d,v)
0.59
Output
1.00
Outpnt
Meal:
0.30
8.7
Lagut
Outpoi
10
20
40
60
76
0.00
Layer (i)
Rocation
Class
CIFARIO (CB)
CIFARIO (CE)
CIPARIO (CE)
CIFARIO (MSE)
Noise
Normalized
WiA (x) Norm
Projected oj(k)
1:50
10w
0:46
0.00
Outgot
0.00
Output
de 10 -
0.25
Layer
Noise
Class
Classs
Figure 4: Analysis of the interaction between representations and weights as distribution shift increases. Plots in first column visualize the norm of network features for different levels of distribution shift at different layers of the network. In later layer of the network, the norm of features tends to decrease as distribution shift increases. Plots in second column show the proportion of network features which lie within the span of the following lincar layer. This tends decrease distributional shift inereases. Error bars represent the standard deviation taken over the test distribution. Plots in the third and fourth column show the accumulation of model constants as compared to the OCS for a cross entropy and a MSE model; the two closely mirror one another.
to the OCS. Moreover, our results show that this trend holds relatively consistently across different loss functions, input modalities, network architectures, and types of distribution shifts. We also found instances where this phenomenon did not hold, such as adversarial inputs, which we discuss in greater detail in Appendix A. However, the overall prevalence of "reversion the OCS" across different settings suggests that it may capture a general pattern in the way neural networks extrapolate.
4 WHY DO OOD PREDICTIONS REVERT TO THE OCS?
In this section, we aim to provide insights into why neural networks have a tendency to revert to the OCS. We will begin with an intuitive explanation, and provide empirical and theoretical evidence in Sections 4.1 and 4.2. In our analysis, we observe that weight matrices and network representations associated with training inputs often occupy low-dimensional subspaces with high overlap. However, when the network encounters OOD inputs, we observe that their associated representations tend to have less overlap with the weight matrices compared to those from the training distribution, particularly in the later layers. As a result, OOD representations tend to diminish in magnitude as they pass through the layers of the network, causing the network's output to be primarily influenced by the accumulation of model constants (e.g. bias terms). Furthermore, both empirically and theoretically. we find that this accumulation of model constants tend to closely approximate the OCS. We posit that reversion to the OCS for OOD inputs occurs due to the combination of these two factors: that accumulated model constants in a trained network tend towards the OCS, and that OOD points yield amaller-magnittee representations in the network that become dominated by model constants.
4.1 EMPIRICAL ANALYSIS
We will now provide empirical evidence for the mechanism we describe above using deep neural network models trained on MNIST and CIFAR 10. MNIST models use a small 4 layer network, and CIFAR10 models use a ResNet20 (He et al., 2016). To more precisely describe the quantities we will be illustrating, let us rewrite the neural network as f() = gi+1 (w) + b,)). where (x) is an intermediate representation at layer W; and b; are the corresponding weight matrix and bias, a nonlinearity, and li+1 denotes the remaining layers of the network. Because we use a different network architecture for each domain, we will use variables to denote different intermediate layers of the network, and defer details about the specific choice of layers to Appendix C.2. We present additional experiments analyzing the ImageNet domain, well the effects of batch and layer normalization in Appendix C.
First, we will show that (z) tends to diminish for OOD inputs. The first column of plots in Fig. 4 show Ey- (z) (o (z(()) 121/Ea= [ (o [x) for PooD with different level of rotation or noise. The x-axis represents different layers in the network, with the leftmost being the input
5

---

Published as a conference paper at ICLR 2024
and the rightmost being the output. We can that in the later layers of the network, ( consistently became smaller as inputs became more OOD (greater rotation/noise). Furthermore, the diminishing effect becomes more pronounced the representations pass through more layers.
Next, we will present evidence that this decrease in representation magnitude occurs because oy (x) for x: ~ Punin(2) (x) tend to lie more within the low-dimensional subspace spanned by the rows of Wj than (y(x) for x ~ POoD (z). Let Voo denote the top (right) singular vectors of W. The middle plots of Fig. show the ratio of the representation's 's norm at layer j that is captured by projecting the representation onto Vio i.e., ||(b)()) Veo Topl 2 //lo (1)) Il" as distribution shift increases. We can see that as the inputs become more OOD, the ratio goes down, suggesting that the representations lie
increasingly outside the subspace spanned by the weight matrix.
Finally, we will provide evidence for the part of the mechanism which accounts for the optimality of the OCS. Previously, we have established that OOD representations tend to diminish in magnitude in later layers of the network. This begs the question. what would the output of the network be if the input representation at an intermediary layer had magnitude of O? We call this the accumulation of model constants, i.e. 9++ 1 (arbb) )). In the third and fourth columns of Fig. 4, we visualize the accumulation of model constants at one of the final layers k of the networks for both a cross entropy and a MSE model (details in Sec. 3.1), along with the OCS for each model. We can see that the accumulation of model constants closely approximates the OCS in each case.
4.2 THEORETICAL ANALYSIS
We will now explicate our empirical findings more formally by analyzing solutions of gradient flow (gradient descent with infinitesimally small step size) on deep homogeneous neural networks with ReLU activations. We adopt this setting due to its theoretical convenience in reasoning about solutions at convergence of gradient descent (Lyu & Li, 2019; Galanti et al., 2022: Huh et al., 2021), and its relative similarity to deep neural networks used in practice (Neyshabur et al., 2015; Du et al., 201 8).
Setup: We consider a class of homogeneous neural networks F == {f(W: :>) W W), with L lay- ers and ReLU activation, taking the functional form f(W; z) = W L-1.... (W2) where W; E Rmxm, Vi E (,....: L - 1}. W1 E Rmx1 and Wr, E R1xm. Our focus is on binary classification problem where we consider two joint distributions Perain, POoD over in- puts and labels: X x V. where inputs are from X' == {x E Rd ||>||e S 1}, and labels are in y == +1}. We consider gradient descent with a small learning rate on the objective: L(W:D)= = Σ ((a))ED ((f(W;))) y) where (f( 3) + exp (-yf(W;z)) is the exponential loss and D is an IID sampled dataset of size N from Perain- For more details on the setup, background on homogeneous networks, and full proofs for all results in this section, please see Appendix D.
We will begin by providing a lower bound on the expected magnitude of intermediate layer features corresponding to inputs from the training distribution:
Proposition 4.1 (P (Purain observes high norm features) When f(W; fits D, i.e., y: V, m
ViE[N], then w.h.p 1 å over D, layer j representations f;(W; (W;x) satisfy E R [Il (W;
(1/Co)(T- - Ö( /og((//8//// + C. log m/N=)), 3 constants Co, Ci s.f. [[5 S Co 1/4 C 2 C
SL/2
Here, we can see that the trained network perfectly fits the training data (y: (J: (W: 22),, ViE(N), and the training data size N is sufficiently large, then the expected Eg norm of layer activations ([WW;) (x) on P Pirain is large and scales at least linearly with 7.
Next, we will analyze the size of network outputs corresponding to points which lie outside of the training distribution. Our analysis builds on prior results for gradient flow on deep homogeneous nets with ReLU activations which show that the gradient flow is biased towards the solution (KKT point) of a specific optimization problem: minimizing the weight norms while achieving sufficiently large margin on each training point (Timor et al., 2023; Arora et al., 2019; Lyu & Li, 2019). Based on this, it is easy to show that that the solution for this constrained optimization problem is given by neural network with low rank matrices in each layer for sufficiently deep and wide networks. Furthermore, the low rank nature of these solutions is exacerbated by increasing depth and width, where the network approaches an almost rank one solution for each layer. If test samples deviate from this low rank space of weights in any layer, the dot products of the weights and features will collapse in the subsequent layer, and its affect rolls over to the final layer, which will output features

---

Published as a conference paper at ICLR 2024
with very small magnitude. Using this insight, present an upper bound on the magnitude of the final layer features corresponding to OOD inputs:
Theorem 4.1 (Feature norms can drop easily on POOD) Suppose 3 a network ( with L' layers and m' neurons satisfying conditions in Proposition 4.1 (y=1)). When we optimize the training objective with gradient flow over class of deeper and wider homogeneous networks F with L L', m' m. m', the resulting solution would converge directionally to a network f(W; xx) for
which the following is true: 3a set of rank 1 projection malrices {At) such that if representations for any layer satisfy Eroor AA (W; x))22 S c, then 3C2 for which Erood (W;;))) & Colc+
15'///++//a
This theorem tells us that for any layer j. there exists only a narrow rank one space A, in which OOD representations may lie, in order for their corresponding final layer outputs to remain significant in norm. Because neural networks are not optimized on OOD inputs, we hypothesize that the features corresponding to OOD inputs tend to lie outside this narrow space, leading to a collapse in last layer magnitudes for OOD inputs in deep networks. Indeed, this result is consistent with our empirical findings in the first and second columns of Fig. 4.1, where we observed that OOD features tend to align less with weight matrices, resulting in a drop in OOD feature norms.
To study the accumulation of model constants, we now analyze a slightly modified class functions == (f(W::) + b : E R. f(W::) E F), which consists of deep homogeneous networks with bias term in the final layer. In Proposition 4.2, we show that there exists set of margin points (analogous support vectors in the linear setting) which solely determines the model's bias b.
Proposition 4.2 (Analyzing network bias) If gradient on F converges directionally to W, then x En yk for margin points ((Tk, yk) yk f[W; = arg mi y) ((W;x)))
If the label marginal of these margin points mimics that of the overall training distribution, then the learnt bias will approximate the OCS for the exponential loss. This result is consistent with our empirical findings in the third and fourth columns on Fig. 4.1, where we found the accumulation of bias terms tends to approximate the OCS.
RISK-SENSITIVE DECISION -MAOONGG
Lastly, we will explore an application of our observations decision-making problems. In many crissonnmmaing scenarios, certain actions offer high potential for reward when the agent chooses them correctly, but also higher penalties when chosen incorrectly, while other more cautious actions consistently provide moderate level of reward. When utilizing learned model for decision-making, is desirable for the agent to select the high-risk high-reward actions when the model is likely to be accurate, while opting for more cautious actions when the model is prone to errors, such when the inputs are OOD. It turns out, if we leverage "reversion to the OCS" appropriately, such risk-sensitive behavior emerge automatically. If the OCS of the agent's 's learned model corresponds to cautious actions, then "reversion to the OCS" posits that the agent will take increasingly cautious actions as its inputs become more OOD. However, not all decision-mall making algorithms leverage "reversion to the OCS" by default. Depending on the choice of loss function (and consequently the OCS), different algorithms which have similar in-distribution performance can have different OOD behavior. In the following sections, we will use selective classification as an example of a decision-making problem to more concretely illustrate this idea.
5.1 EXAMPLE APPLICATION: SELECTIVE CLASSIFICATION
In selective classification, the agent can choose to classify the input or abstain from making decision. As an example, we will consider a selective classification task using CIFARIO, where the agent receives reward of +1 for correctly selecting class, reward of -4 for an incorrect classification, and a reward of 0 for choosing to abstain.
Let consider
approach that leverages "reversion to the OCS" and one that does not, and discuss their respective OOD behavior. An example of the former involves learning a model to predict the reward associated with taking each action, and selecting the action with the highest
7

---

Published as a conference paper at ICLR 2024
Noise: 0
Noise:
Noise:
Noise: $
Noise:
Noise:
focuetant
Prediction
Frequency Averago
01234387H8A
012349075UA
0128438789A
0123490759A
01254067880
0124598789A
01234b0759A
Action
130
0120456789A
0123456759A
012345679DA
0120456789A
0123456TSBA
0193456780A
0123456759A
Figure 5: Selective classification via reward prediction on CIFARIO. We evaluate on holdout datasets consisting of automobiles (class 1) with increasing levels of noise. X-axis represents the agent's s actions, where classes are indexed by numbers and abstain is represented by "A". We plot the average reward predicted by the model for each class (top), and the distribution of actions selected by the policy (bottom). The rightmost plots represent the OCS (top). and the actions selected by an OCS policy (bottom). As distribution shift increased, the model predictions approached the OCS, and the policy automatically selected the abstain action more frequently.
predicted reward. This reward model, fo R + RIAI, takes as input an image, and outputs vector the size of the action space. We train So using dataset of images, actions and rewards, D = [( 4t ) by minimizing the MSE loss, N Eisien (Se(z)) rr) and select actions using the policy x (xx) arg arg A fo (z) The OCS of fo is the average reward for each action over the training points, i.e., (f )a LisTe r.: (a:ma]
EiSTEn Tayma In our example, the OCS is -3.5 for selecting each class and O for abstaining, so the policy corresponding to the OCS will choose to abstain. Thus, according to "reversion to the OCS", this agent should choose to abstain more and more frequently as its input becomes more OOD, We illustrate this behavior in Figure 5. In the first row, we depict the average predictions of a reward model when presented with test images of a specific class with increasing levels of noise (visualized in Figure 1). In the second row, we plot histogram of the agent's 's selected actions for each input distribution. We can see that as the inputs become more OOD, the model's predictions converged towards the OCS, and consequently, the agent automatically transitioned from making high-risk, high-reward decisions of class prediction to the more cautious decision of abstaining.
One example of an approach which does not leverage "reversion to the OCS" is standard classification via cross entropy. The classification model takes as input an image and directly predicts probability distribution over whether each action is the optimal action. In this case, the optimal action given an input is always its ground truth class. Because the OCS for cross entropy is the marginal distribution of labels in the training data, and the optimal action is never to abstain, the OCS for this approach corresponds
policy that never chooses to abstain. In this case, "reversion to the OCS" posits that the agent will continue to make high-risk high-reward decisions even as its inputs become more OOD, As a result, while this approach can yield high rewards on the training distribution, is likely to yield very low rewards on OOD inputs, where the model's 's predictions are likely to be incorrect.
5.2 EXPERIMENTS
We will now more thoroughly compare the behavior of reward prediction agent with standard classification agent for selective classification on variety of different datasets. Our experiments aim answer the questions: How does the performance of a decision-making approach which leverages "reversion to the OCS" compare to that of an approach which does not?
Experimental Setup. Using the same problem setting as the previous section, we consider a selective classification task in which the agent receives a reward of +1 for selecting the correct class, -4 for selecting an incorrect class, and +0 for abstaining from classifying. We experiment with datasets: CIFAR10. 10, DomainBed OfficeHome, BREEDS living-17 and non-living-26, We compare the performance of the reward prediction and standard classification approaches described in the previous section, as well as third oracle approach that is optimally risk-sensitive, thereby providing an upper bound on the agent's 's achievable reward. To obtain the oracle policy, we train a classifier on the training dataset to predict the likelihood of each class, and then calibrate the predictions with temperature scaling on the OOD evaluation dataset. We then use the reward function to calculate the theoretically optimal threshold on the classifier's maximum predicted likelihood, below which the
8

---

Published as a conference paper at ICLR 2024
Oracle
Reward Prodiction
Standard Classification
Impulsee
Shot Noise
Defocus Blur
Motion Blur
Speckle
1.0
1.0
1.0
1.0
Abetain
0.5-
0.5
0.5
0.5
0.5
8.0
(L0
0.0
0.0
5
0(1)
0(5)
000)
Noise
Noise
Noise
Noise
Noise
1.0
OfficeHome
Abstain Ratio
1.0
living-17
Abetain Ratio
non-living-26
0.6
9.6
0.0
Product(t)
Photo
Clipart
Art
0.0
Train(s)
OOD
Train
OOD
Figure 6: Ratio of abstain action to total actions; error bars represent standard deviation over random seeds; (1) denotes the training distribution. While the oracle and reward prediction approaches selected the abstain action
more frequently imputs became more OOD, standard classification approach almost never selected abstain.
Impuise Noise
Shot Noise
Defocus Blur
Motion Blur
Speckle Noise
Reward
-2
-2
(01)
0(0)
Noise
Noise
Noise
Noise
Noise
OfficeHome
living-17 17
non-living-26
Reward
Reward
Reward
16
&
(Loduct(L)
Photo
Clipart
Art
Train(t)
OOD
Train(t)
OOD
Figure 7: Reward obtained by each approach. While all three approaches performed similarly the training distribution, reward prediction increasingly outperformed standard classification inputs became more OOD,
abstaining option is selected. Note that the oracle policy has access to the reward function and the OOD evaluation dataset for calibration, which the other two approaches do not have access to.
Results. In Fig. 6, plot the frequency with which the abstain action is selected for each approach. As distribution shift increased, both the reward prediction and oracle approaches selected the abstaining action more frequently, whereas the standard classification approach never selected this option. This discrepancy arises because the OCS of the reward prediction approach aligns with the abstain action, whereas the OCS of standard classification does not. In Fig. 7, we plot the average reward received by each approach. Although the performance of all three approaches are relatively similar on the training distribution, the reward prediction policy increasingly outperformed the classification policy distribution shift increased. Furthermore, the gaps between the rewards yielded by the reward prediction and classification policies are substantial compared to the gaps between the reward prediction and the oracle policies, suggesting that the former difference in performance is nontrivial. Note that the goal of our experiments was not to demonstrate that our approach is the best possible method for selective classification (in fact, our method is likely not better than SOTA approaches), but rather to highlight how the OCS associated with an agent's 's learned model influence its OOD decision-making behavior. To this end, this result shows that appropriately leveraging "reversion to the OCS" can substantial improve an agent's performance on OOD inputs.
CONCLUSION
We presented the observation that neural network predictions for OOD inputs tend to converge towards the model's optimal constant solution, and provided empirical evidence for this phenomenon across diverse datasets and different types of distributional shifts. We also proposed a mechanism to explain this phenomenon and simple strategy that leverages this phenomenon to enable risk-sensitive decision decision-making, making. While understanding of "reversion to the OCS" remains rudimentary, hope our observations will prompt further investigations on how we may predict and even potentially steer the behavior of neural networks OOD inputs.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
Christina Baek, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. serement-onnthe-lnge Predict- ing the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35: 15:19494-1,,89 2022.
Randall Balestriero, Jerome Pesenti, and Yann LeCun. Learning in high dimension always amounts to extrapolation. arXiv preprint arXiv:2110.09485, 2021.
Peter L Bartlett, Dylan Foster, and Matus J Telgarsky Spectrally-normaaizzed margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Nontawat Charoenphakdee,, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pp. 1507-1517. PMLR, 2021.
Edo Cohen-Karlik, Avichai Ben David, Nadav Cohen, and Amir Globerson. On the implicit bias of gradient descent for temporal extrapolation. In International Conference on Artificial Intelligence and Statistics, pp. 10966-10981 PMLR, 2022.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pp. 67-82 Springer, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 245. leee, 2009.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Thomas G Dietterich and Alex Guyer. The familiarity hypothesis: Explaining the behavior of deep open set methods. Pattern Recognition, 132:108931, 2022.
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018
Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H Abdi. Towards better selective classification. In The Eleventh International Conference on Learning Representations, 2011.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Tomer Galanti, Zachary S Siegel, Aparna Gupte, and Tomaso Poggio. Sgd and weight decay provably induce a low-rank bias in neural networks. arxiv, 2022.
Yonatan Geifman and Ran El-Ya.iv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 14 2014.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009.
10

---

Published as a conference paper at ICLR 2024
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schön. How reliable is your regression model's 's uncertainty under real-world distribution shifts? arXiv preprint arXiv:2302.036699 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41-50, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903. 12261, 2019.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and -ittr---distibutiin examples in neural networks. arXiv preprint arXiv: 1610.022100 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 18/2.04600 2018.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021.
Haiwen Huang, Zhihan Li, Lulu Wang, Sishuo Chen, Bin Dong, and Xinyu Zhou. Feature space singularity for cut-of-distributtio detection. arXiv preprint arXiv:201 1.111..1444444 2020.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103.10427, 2021.
Daksh Idnani, Vivek Madan, Naman Goyal, David Schwab, and Shanmukha Ramakrishna Vedantam. Don't forget the nullspace! nullspace occupancy mechanism for out of distribution failure. In The Eleventh International Conference on Learning Representations, 2022.
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning. pp. 5637-5664. PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. arxiv, 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2019.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy the line: the strong correlation between distrr-iittiitution and in in-distribution generalization. International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021.

---

Published as a conference paper at ICLR 2024
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? arXiv preprint 4rX1::1810.009136, 2018.
Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 427-436, 2015.
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. Advances in Neural Information Processing Systems, 32, 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), P), pp. 372-387. IEEE, 2016.
Tim Pearce, Alexandra Brintrup, and Jun Zhu. Understanding softmax confidence and uncertainty. arXiv preprint arXiv:2106.04977 2021.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International conference on machine learning, pp. 5389-5400 PMLR, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 2019.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprini arXiv:2008. 04859, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2822-2878. 2018.
Masashi Sugiyama, Matthias Krauledat, and Claus-Robent Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference Machine Learning, pp. 20827-20840. PMLR, 2022.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1rXiiv:332..61 2013.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. Advances in neural information processing systems, 2020.
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. In International Conference on Algorithmic Learning Theory, pp. 1429-1459, PMLR, 2023.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
12

---

Published as a conference paper at ICLR 2024
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, pp. 10506-10518, 2019.
Taylor Webb, Zachary Dulberg, Steven Frankland, Alexander Petrov, Randall O'Reilly, and Jonathan Cohen. Learning representations that support extrapolation. In International conference on machine learning. pp. 10136-10144 10146. PMLR, 2020.
Yongtao Wu, Zhenyu Zhu, Fanghui Liu, Grigorios Chrysos, and Volkan Cevher. Extrapolation and spectral bias of neural nets with hadamard product: polynomial net study, Advances in neural information processing systems, 35:26980-26995, 2022.
Guoxuan Xia and Christos-Savvas Bouganis. Augmenting softmax information for selective classifi- cation with -itoo--distribution data. In Proceedings of the Asian Conference on Computer Vision, pp. 1995-2012, 2022.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward graph neural networks. arXiv preprint arXiv:2009. /848, 2020.
Gilad Ychudai, Ethan Fetaya, Eli Meirom, Gal Chechik, and Haggai Maron. From local structures to size generalization in graph neural networks. In International Conference on Machine Learning, pp. 11975-11986. PMLR, 2021.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/eegressonn by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5810-5818, 2017.
13

---

Published as a conference paper at ICLR 2024
A INSTANCES WHERE "REVERSION TO THE OCS" DOES NOT HOLD
In this section, we will discuss some instances where "reversion to the OCS" does not hold. The first example is evaluating an MNIST classifier (trained via cross entropy) on an adversarially generated dataset using the Fast Gradient Sign Method (FSGM) (Goodfellow et al., 2014). In Fig. 8, we show our findings. On the left plot, we can see that the outputs corresponding to the adversarial dataset were farther from the OCS than the predictions corresponding to the training distribution, even though the adversarial dataset more OOD. On the right plot, we show the normalized representation magnitude of the original and adversarial distributions throughout different layers of the network. We can see that the representations corresponding to adversarial inputs have larger magnitudes compared to those corresponding to the original inputs. This is departure from the leftmost plots in Fig. 4, where the norm of the representations in later layers decreased as the inputs became more OOD. One potential reason for this behavior is that the adversarial optimization pushes the adversarial inputs to yield representations which align closer with the network weights, leading to higher magnitude representations which push outputs father from the OCS.
Adversarial MNIST (CE)
Adversarial MNIST
Dist. to OCS
Normalized lize
W [xxx)) Norm
Adversarial
1.1
Original
2.20
1.0
2.00
0..03
0.60
0.20
EL-0
0.90
Output
OOD Score
Layer
Figure 8: On the left, we evaluate the distance between network predictions and the OCS as the input distribution becomes more OOD for an MNIST classifier. The red star represents the (holdout) training distribution, and the blue circle represents an adversarially generated evaluation distribution. Even though the adversarial distribution is more OOD, its predictions were farther from the OCS. On the right, we plot the normalized representation magnitude across different layers of the network. The representations corresponding to adversarial inputs had greater magnitude throughout all the layers.
Our second example is Gaussian NLL models trained on UTKFace, and evaluated on inputs with impulse noise. Previously, in Fig. 3, we had shown that "reversion to the OCS" holds for UTKFaae with gaussian blur, but we found this to not necessarily be the case for all corruptions. In Fig. 9, we show the behavior of the models evaluated on inputs with increasing amounts of impulse noise. In the middle plot, we can see that as the OOD score increases (greater noising), the distance to the OCS increases, contradicting "reversion to the OCS". In the right plot, we show the magnitude of the representations in an internal layer for inputs with both gaussian blur and impulse noise. We can see that while the representation norm decreases with greater noise for gaussian blur, the representation norm actually increases for impulse noise. We are not sure why the model follows "reversion to the OCS" for gaussian blur but not impulse noise. We hypothesize that one potential reason could be that, because the model was trained to predict age, the model learned to identify uneven texture proxy for wrinkles. Indeed, we found that the model predicted higher ages for inputs with greater levels of impulse noise, which caused the predictions to move farther from the OCS.
14

---

Published as a conference paper at ICLR 2024
Untidnal (0)
UTK Face
FTKFeae
W Norm
3.00
200
2,30
150
Dist: to
Gazadan Bler Impede Note
Geussian Bhe (6) Enpube Neise (3)
1.50
100
3
4
OOD Score
Noise
Figure 9: On the left, we visualize an example of UTKe inputs in its original form, with gaussian blur, and with impulse noise. In the middle, evaluate the distance between network predictions and the OCS as the input distribution becomes representing the (holdout) training distribution, and circles representing OOD datasets with increasing levels impulse noise. The vertical line associated with each point represents the standard deviation over 5 training runs. As the OOD score of the evaluation dataset increases, the model predictions here tended to move farther from the OCS. On the right, plot the magnitude of the representation imputs with different levels of gaussian blur and impulse noise. The representation magnitude for inputs with greater gaussian blur tend to decrease, while the representation magnitude with greater impulse noise tend to increase.
OOD. Fach point represents different evaluation dataset, with the red
specific layer of the model corresponding
B EXPERIMENT DETAILS
B.1 DATASETS
The datasets with discrete labels include CIFARIO (Krizhevsky et al., 2009), ImageNet (Deng et al., 2009) (subsampled to 200 classes to match (nageNet-R(rendition) (Hendrycks et al., 2021)), DomainBed OfficeHome (Gulrajani & Lopez-Paz, 2020), BREEDS LIVING-17 and NON-LIVING- 26 (Santurkar et al., 2020), and Wilds Amazon (Koh et al., 2021), and the datasets with continuous labels include SkinLesionPixels (Gustafsson et al., 2023) and UTKFace (Zhang al., 2017). We evaluate CIFAR 10 models on CIFAR10-C (Hendrycks & Dietterich, 2019), which includes syn- thetic corruptions varying levels of intensity. We evaluate ImageNet models ImageNet-R and ImageNet-Sketch (Wang et al., 2019), which include renditions of ImageNet classes in novel styles and in sketch form. OfficeHome includes images of furniture in the style of product, photo, clipart, and art. We train on product images and evaluate on the other styles. BREEDS datasets consist of training and OOD test datasets consisting of the same classes but distinct subclasses. Wilds Amazon consists of training and OOD test datasets of Amazon reviews from different sets of users. To amplify the distribution shift, we subsample the OOD dataset only include incorrectly classified points. Skinl esionPixels consists of dermatoscopic images, where the training and OOD test datasets were collected from patients from different countries. UTKFace consists of images faces, and we construct OOD test datasets by adding different levels of Gaussian blur to the input.
B.2 TRAINING PARAMETERS
First, we will discuss the parameters we used to train our models.
Task
Network Architecture
2 convolution layers followed by fully connected layers
MNIST
ReLU nonlinearities
CIFAR1O
ResNet20
ImageNet
ResNet50
OfficeHome
ResNet50
BREEDS
ResNet18
Amazon
DistilBERT
Skinl_esionPixeli
ResNet34
UTKFace
Custom VGG style architecture
15

---

Published as a conference paper at ICLR 2024
Learning
Learning Rate Scheduler
Weight
Task MNIST
Optimizer Adam
Rate
Decay
Momentum
0.001
Step; ? 0.7
0.01
Multi step
CIFARIO
SGD
0.1
milestones=[1000 150]
0.0001
0.9
Step: 7: 0.1
ImageNet
SGD
0.1
step size 30
0.0001
0.9
OfficeHome
Adam
0.00005
Linear with warm up
BREEDS
SGD
0.2
(warm up frac=0.05)
0.00005
0.9
Linear with warm up
Amazon
AdamW
0.00001
(warm up frac=0)
0.01
SkinLlesioPPixels
Adam
0.001
UTKFace
Adam
0.001
0
Task
Data Preprocessing
MNIST
Normalization
CIFARTO
Random horizontal flip, random crop, normalization
ImageNet
Random resized crop, random horizontal flip, normalization
Random resized crop, random horizontal flip,
OfficeHome
color jitter, random grayscale, normalization
Random horizontal flip,
BREEDS
random resized crop, randaugment
Amazon
PaniliBERTTokknnnn Normalization
SkinLesionPPiells
UTKFace
Normalization
B.3 EVALUATION METRICS
Next, we will describe the details of our OOD score calculation, For image datasets, we pass the image inputs through pretrained ResNet18 ImageNet featurizer to get feature representations, and train a linear classifier to classify whether the feature representations are from the training distribution the evaluation distribution. We balance the training data of the classifier such that each distribution makes up 50 percent. We then evaluate the linear classifier on the evaluation distribution, and calculate the average predicted likelihood that the batch of inputs are sampled from the evaluation distribution, which we use as the OOD score. For text datasets, we use a similar approach, but use a DistilBERT classifier and Tokenizer instead of the linear classifier and ImageNet featurizer.
There are some limitations to the OOD score. Ideally, we would like the OOD score to be measure of how well model trained on the training distribution will generalize to the evaluation distribution. This is often the case, such as the datasets in our experiments, but not always. Consider a scenario where the evaluation dataset a subset the training dataset with a particular type of feature. Here, neural network model trained on the training dataset will likely generalize well to the evaluation dataset in terms of task performance. However, the evaluation dataset will likely receive a high OOD score, because the evaluation inputs will be distinguishable from the training inputs, since the evaluation dataset has a high concentration of a particular type of feature. In this case, the OOD score not a good measure of the distribution shift of the evaluation dataset.
Additionally, with regards to our measure of distance between model predictions and the OCS, we note that this measure is only informative if the different datasets being evaluated have around the same distribution of labels. This is because both the MSE and the KI. metrics are being averaged over the evaluation dataset.
B.4 CHARACTERIZING THE OCS FOR COMMON LOSS FUNCTIONS
In this section, we will precisely characterize the OCS for a few of the most common loss functions: cross entropy, squared error (MSE), and Gaussian negative log likelihoog (Gaussian NLL.).
16

---

Published as a conference paper at ICLR 2024
Cross entropy. With cross entropy loss, the neural network outputs vector where each entry associated with a class, which we denote as fo()); This vector parameterizes a categorical efa distribution: (a(()()) Eisism efolmy The loss function minimizes the divergence between Po(y)z) and P(y]z), given by
m Mil log
elo()) 2jm efolm);)
C(fo(r).V) Σ 1ly
While there can exist multiple optimal constant solutions for the cross entropy loss, they all map to the same distribution which matches the marginal empirical distribution of the training labels,
P (v:)
N EiSien 1 = yi)
E1sss e "ostuny
For the cross entropy loss, the uncertainty the neural network prediction can be captured by the entropy of the output distribution. Because Pj Janen (y) usually has much higher entropy than Po ())=) evaluated on the training distribution, Pa (y x) on OOD inputs will tend have higher entropy than on in-distribution inputs.
With this loss function, the output of the neural network parameterizes the mean and standard deviation of Gaussian distribution, which we denote as fo(x) [=e(z)) og (z)]. The objective is to minimize the negative log likelihood of the training labels under the predicted distribution, Po (y(la) ((oo())
Gaussian NLI.
Ho(z))²
[(fe(z), v) = log(ar(z))) oo (x))
Let us similarly denote famstam Meconstant: Fconstaml- In this case, Heonstant E Eisien Vt. and
constant 1 Eisien Pernstam )². Here, Constamt usually much higher than the standard deviation of P(yla) for any given z. Thus, our observation suggests that neural networks should predict higher standard deviations for OOD inputs than training inputs.
MSE. Mean squared error can be seen as a special case of Gaussian NLL in which the network only predicts the mean of the Gaussian distribution, while the standard deviation is held constant, i.e. Pa(y|z) ~ (N)(x)) 1). The specific loss function is given by:
((fo(r). = (v - fo(x))
Here, Soonstant = exhibited greater uncertainty, predictions from an MSE loss do not capture any explicit notions of uncertainty. However, observation suggests that the model's predicted mean will still move closer to the average label value as the test-time inputs become more OOD.
N Eisien Di- Unlike the previous two examples, in which OOD predictions
C EMPIRICAL ANALYSIS
C.I ADDITIONAL EXPERIMENTS
In this section, will provide additional experimental analysis to support the hypothesis that we put forth in Section 4. First, in order to understand whether the trends that we observe for CIFARI0 and MNIST would scale to larger models and datasets, we perform the same analysis as the ones presented in Figure on ResNet50 model trained ImageNet. and evaluated on ImageNet- -keethh and -ingeeeetn(iiiiiinn) Our findings are presented Figure 10. Here, we can see that the same trends from the CIFAR10 and MNIST analysis seem transfer to the ImageNet models.
Next, we aim to better understand the effects of normalization layers in the network on the behavior of the model representations. We trained models with normalization (NN), batch normalization (BN), and layer normalization (LN) on the MNIST and CIFAR datasets, and evaluated them on OOD test sets with increasing levels of rotation and noise. Note that the model architecture and all
17

---

Published as a conference paper at ICLR 2024
ImageNet (CE)
ImageNet (CE)
ImageNet (CE)
ImageNet (MSE)
Wab()) Norm
Projected (l00)
Ratio
ocB
Meai Ormtaste
Norwalized
Oulput
Output
0.3
1.0
0.75
Inpot
Outgon
Origgnall Ri R/cadition) Slantch
Class
Class
Layer (i)
Noisc
Figure 10: Analysis of the interaction between representations and weights for distribution shift increases, for model trained on ImageNet and evaluated on ImageNet-Sketch and (nageNet-R(enditon)))
MNIST (NN)
MNIST (NN)
MNIST (NN)
Normalized Wioh xc)x) Norm
1.0
Projected dy(x)
Norm Ratio
1.00
Crestants
0.9
0.35
Output
0.75
0.8
0.30
0.50
0.7
0.25
Input
Output
15
30
45
60
75
MNIST (BN)
MNIST
(BN)
MNIST (BN)
Rotation
75
Normalized
Wighx [ Norm
0.9
1.0
Projected dj(x)
Norm Ratio
0.40
Output
Model Coentinte
OCS
60
0.8
0.35
0.50
45
30
0.7
0.30
0.25
15
Input
Output
30
45
60
75
0.00
3
6
MNIST (LN)
MNIST (LN)
MNIST (LN)
Normalized
Wioh(X) ( Norm
1.0
Projected dj(x)
Norm Ratio
0.45
Model
Output
0.75
0.9
0.40
0.50
0.8
0.35
0.25
Input
Output
45
0.00
Layer ii)
Rotation
Class
Figure 11: Analysis of the interaction between representations and weights as distribution shift increases, for model trained on MNIST and evaluated on increasing levels of rotation. The models being considered includes four layer neural network with normalization (top), batch normalization (middle), and layer normalization (bottom).
other training details are held fixed across these models (for each datasets) with the exception of the type of normalization layer used (or lack thereof). We perform the same analysis as the ones presented in Figure 4, and present our findings in Figure 11 and 12. We found similar trends across the different models which are consistent with the ones we presented in Figure 4
C.2 ANALYSIS DETAILS
In this section, we will provide details on the specific neural network layers that we used in our analysis in Sections 4.1 and C.I. We will illustrate diagrams for the neural network architectures that we used for each of our experiments, along with labels of the layers associated with each quantity we measure. In the first column each analysis figure, we measure quantities at different layers of of the network:
denote these by io.... in. where each i represents one tick in the X-axis from left to right. We use j to denote the layer used in the plots second column of each figure. We icE (and kMSE )o denote the layer used in the plots in the third (and fourth) columns of each figure, respectively. We illustrate the networks used in Figure 4 in Figure 13, the one used in Figure in Figure 14, the ones used in Figure 11 in Figure 15, and the used in Figure 12 in Figure 16.
18

---

Published as a conference paper at ICLR 2024
CIFAR1O (NN)
CIFARIO (NN)
CIFAR10 (NN)
1.25
1.00
Normalized
Wee(x) (x) Norm
Projected dj
Norm Ratio
Output
0.75
1.0
0.50
0.75
Input
Output
2
Layer (i)
Noise
Class
CIFARIO (BN)
CIFARIO (BN)
CIFAR10 (BN)
Noise
Normalizer Z
Wiog [ Norm
Projected oj
Norm Ratio
Model Cocatants
OCS
1.0
Output
0.75
0.50
0.75
0.25
Input
Output
2
0.00
3
5
Layer (i)
Noise
Class
CIFARIO (LN)
CIFARIO (LN)
CIFARI0 (LN)
1.00
Normalized Wjop x Norm
Projected oj(x)
Norm Ratio
Model
OCS
1.0
Output
0.75
0.25
Input
Output
2 Noise
0.00
Layer
Class
Figure 12: Analysis of the interaction between representations and weights distribution shift increases, for model trained on CIFARIO and evaluated on increasing levels of noise. The models being considered includes AlexNet with no normalization (top), batch normalization (middle), and layer normalization (bottom).
MNIST 4 Layer Network
Convolational Layer
Inpul
Ouipul
Pally
Connected Layer
1g
1 KASSE
1/ Kee
CIFAR10 / ResNet20
Inpul
Output
Kuss
Figure 13: Diagram neural network models used our experimental analysis Figure 4. along with labels of the specific layers we used in our analysis.
D PROOFS FROM SECTION 4.2
In the first subsection, deseribe the setup, and gradient flow with some results from prior works that we rely upon in proving our claims on the in-distribution and dittof-distriuttion activation magnitudes. In the following subsections we prove our main claims from Section 4.2.
19

---

Published as a conference paper at ICLR 2024
ImageNet/ / ResNet50
Input
Output
4
Ruse
4y)
Figure 14: Diagram of neural network models used in our experimental analysis in Figure 10. along with labels of the specific layers used in our analysis.
MNIST 4 Layer Network
No normalization (NN)
Convolutional
Layer
Input
Output
Connected
Layer
Batch Norm
1
Layer Norm
Batch normalization (BN)
Input
Output
Layer normalization (LN)
Imput
Output
Figure 15: Diagram of neural network models used in experimental analysis in Figure 1,, along with labels of the specific layers used in our analysis.
Setup. We are learning over a class of homogeneous neural networks F= (f(W; ;)) E W). with L layers, and element wise activation o<x) x z1xx 2 0) (ReLU function), taking the functional form:
z(W:z) Wro (W2o
where W; E Rm Vi E L 1). W1 E R= and output dimension set to 1, i.c., Wr. IR Rlxmm We say that class F is homogeneous, if there exists constant C' such that, for all
20

---

Published as a conference paper at ICLR 2024
CIFARIO / AlexNet
No normalization (NN)
Convolationa
Layer
leput
Ouipu
Connected
Layer
Butch Noem
Layer Norm
Batch normalization (BN)
Jeput
Output
Layer normalization (LN)
Iupu
Ouppe
Figure 16: Diagram of neural network models used in our experimental analysis in Figure 12, along with labels of the specific layers used in our analysis.
W, have:
f(a W;x) a f(W
Our focus is on binary classification problem where we have joint distribution over inputs and labels: X x y. Here, the inputs are from set X = {a Rdd ||ズ|| I B}, and labels are binary ) : 1-1,+1}, We have an IID sampled training dataset D = { y:) containing pairs of data points 2; and their corresponding labels y:-
For loss function é R + R. the empirical loss of f(W; on the dataset D is
x
L(w; D) = Σ (v.f(W:z))..
Here, the loss f. can be the exponential loss E(g) e and the logistic loss ((g) -log(1 + e- To refer to the output (post activations if applicable) of layer j in the network fWW;,), for the input æ, we use the notation: xw;
Gradient flow (GF). We optimize the objective in Equation equation using gradient flow. Gradient flow captures the behavior of gradient descent with an infinitesimally small step size (Arora et al., 2019; Huh et al., 2021; Galanti et al., 2022; Timor et al., 2023). Let ((t) be the trajectory of gradient flow, where we start from some initial point ((0) of the weights, and the dynamics of W(t) given by the differential equation:
(t
W ( L(w; D).
(2)
dt
Note that the ReLU function differentiable at 0. Practical implementations of gradient methods define the derivative of (0) to be some constant in [0, 1). Following prior works (Timor et al., 2023). in this work we assume for convenience that o' followling limit exists:
We say that gradient flow converges the
lim W()).
f-000
21

---

Published as a conference paper at ICLR 2024
In this case, denote W (co)= = lim W(t). We say the following limit exists:
say that the gradient flow converges in direction
lim (())/
8-000
Whenever the limit point lim (t) exists, we refer the limit point as the ERM solution by running gradient flow, and denote as W.
Gradient flow convergence for interpolating homogeneous networks. Now, we a result from prior works that states the implicit bias gradient flow towards max-margin solutions when suffi- ciently deep and wide homogeneous networks trained with small learning rates and exponential tail classification losses. As the loss converges to zero, the solution approaches a KKT point of an optimization problem that finds the minimum l2 norm neural network with margin of at least each point in the training set. This is formally presented in the following Lemma adapted from (Ji & Telgarsky, 2020) and (Lyu & Li, 2019).
Lemma D.1 (Gradient flow is implicitly biased towards minimum (l112) Consider minimizing the average of either the exponential or the logistic loss (in equation 1) over a binary classifi- cation dataset using gradient flow (in equation 2) over the class of homogeneous neural networks CC, then gradient flow
F with ReLU activations. If the average loss on D converges to zero converges in direction to a first order stationary point (KKT point) of the the following maximum mar- gin problem in the parameter space of W:
min J(W=)EEF
Σ JELL]
IW&l
S.t. Vi [n] (f((; 21
Spectrally normalized margin based generalization bounds (Bartlett et al., 2017). Prior work on Rademacher complexity based generalization bounds provides excess risk bounds based spectrally normalized margins, which scale with the Lipschitz constant (product of spectral norms of weight matrices) divided by the margin.
Lemma D.2 (A dadptation of Theorem 1.1 from (Bartlett et al., 2017)) For the class homoge- neous of ReLU networks in F with reference matrices (AI....AAp and all distributions P inducing binary classification problems, i.e., distributions over Rd x [[-1,+1}, with probability â over the III) sampled dataset DD and margin y O, the network f(r; has expected margin loss upper bounded as:
Rw..A 7n
log log(1/8)
E(s.))rr(yf(w;;)) 27) E (vif(w;z)) 27)
log(m) +
where the covering number bound determines RW.A = (II |[Wallop)
Σ
I.. W.T-ATI 1
D.1 LOWER BOUND FOR ACTIVATION MAGNITUDE ON IN-DISTRIBUTION DATA
Proposition D.1 (Pirain observes high norm features) When f(W;2) (x) fits D, i.e. xw: ViE(N), then w.h.p 5 over D, layer j representations (,WW;)) satisfy E (W; C2 log m//)))) 3 constants Co, C1 s.t. [[W I C C1 co.
(1/C,)(r ( /og(1/66//
ch+l-1
Proof.
Here, we lower bound the expected magnitude of the in-distribution activation norms at fixed layer j in terms of the expected activation norm at the last layer, by repeatedly applying Cauchy-Schward
22

---

Published as a conference paper at ICLR 2024
inequality and the property of ReLU activations: ||o(x)]l2 & |||||, alternatively.
E Pusii
a+)
Erenin
[ ((WoWW)).....((((()))))) ))1l
I WilppEEr
(R 112
S [Wi.llppEPp....
(ReLU activation property)
Doing the above repeatedly gives us the following bound:
E E (f(ū
II R-
lIWallop)
S < G+1//
(I,(
co [W:
Next, generalization bound on the margin loss to further lower bound the expected norm of the last layer activations. Recall, that we use gradient flow to converge to globally optimal solution of the objective in equation such that the training loss converges to 0 as co. Now, the spectrally normalized generalization bound from Lemma D.2 to get:
log m N
Σ
W. - -,
(1g(1/5) N
B(c))-- x)) 22) s Ō
II W Willln
+
|
2/3
This implies that with probability 8 over the training set D, on least O( (og(1/6)/m) fraction of the test set, the margin is at least y, i.e., if we characterize the set of correctly classified test points as C.y. then:
Bllf(W=)]] (z.y) y) e C..l Elyy f(W;=)] () e Cwrl
2 Ely f(W;z) (x,y) E Cwrl 2 7
We are left with lower bounding: B[l((W;:))) generalization guarantee we know:
(z, v) & C which is trivially 2 Now, from the
((1((z))) v) E Cw))
log m
1/L
I| - A7 11.
+
log(1 18)
NN
2/3
N
log Co
I
A 1l
1og(1/8)
25
7/N
|| 2//3
log m ?N
Σ
A7 1| I|WT
log(1/8)
5
Ci
+
|| | 2/3
N
where the final inequality uses
1/2
W 2/3
2/3
which is the typical AM-GM inequality. We also use the inequality: C1 2 33//2 This bound tells us that:
(108 log m C
log(1/0)
B(1((z,)) v) (w)) 1
7N
N
Plugging the above into the lower bound derived completes the proof of Proposition D.1.
23

---

Published as a conference paper at ICLR 2024
D.2 UPPER BOUND FOR ACTIVATION MAGNITUDE ON OUT-OF-DISTRIBUTION DATA
Theorem D.3 (Feature norms can drop easily on PooD) If shallow network ('W;;)) with L' layers and m' neurons satisfying conditions Proposition 4.1 (사=1), then optimizing the training objective with gradient flow over class of deeper and wider homogeneous network F with >>> I', m. > m' would converge directionally to a solution f(W;x), for which the following is true: 3 a set of rank projection matrices {Ai such that if representations for any layer j satisfy Eooo 11 ll2 S then 302 for which E Pooo (W;x)) & Cole + Cz Cz (Lt1/1).
Proof.
Here, we show that there will almost rank one subspaces for each layer of the neural network, such that if the OOD representations deviate even slightly from the low rank subspace any given layer, the last layer magnitude will collapse. This phenomenon is exacerbated in deep and wide networks, since gradient descent on deep homogeneous networks biased towards KKT points of minimum norm, max-margin solution (Lyu & Li, 2019), which consequently leads gradient flow on sufficiently deep and wide networks towards weight matrices that are low rank (as low as rank 1).
We can then show by construction that there will always exist these low rank subspaces which the OOD representations must not deviate from for the last layer magnitudes to not drop. Before we prove the main result, we adapt some results from Timor et al. (2023) to show that gradient flow is biased towards low rank solutions in our setting. This formally presented in Lemma D.4.
Lemma D.4 (GF on deep and wide nets is learns low rank W1,...,W2 We are given the IID sampled dataset for the binary classification task defined by distribution Perains i.e., D == (z,, ye) Wi) S Rd x {-1,1]. Here, ||æi|22 S 1, with probability 1. Let there exist homoge- neous network in class F', of depth L' 2 2, and width m' 2 if there exists neural network f(W';z), such that: V(I E DD f(W";z,, 2., Y: 2 7s and the weight matrices Wi.... W'i: satisfy ||W, |E S for some fixed constant C' > 0. If the solution f(W*, x) of gradient flow any class F of deeper L L' and wider m > m' networks xW) converges to the global optimal of the optimization problem:
min ((W2)EF
Σ
s.t.
W vf((;z))) 21,
(4)
2
JE[L]
then for some universal constant C1, the following is satisfied:
max [IW; 1oo///1W/1
1
W op
2 CyLL
L L++1
(5)
iE(T)
T.
W
Proof.
We will prove this using the result from Lemma D. and major parts of the proof technique is a re-derivation of some of the results from Timor et al. (2023), in our setting. From Lemma D. we know that gradient flow on F necessarily converges in direction to KKT point of the optimization problem in Equation 4. Furthermore, from Lyu & Li (2019) we know that this optimization problem Constraint Qualification (MFCQ) condition, which means that
satisfies the Songasarian-Fromore the KKT conditions are first-order necessary conditions for global optimality.
We will first construct a wide and deep network f(W; x) E F, using the network f(W": x)) from the relatively shallower class F', and then argue about the Frobenius norm weights of the constructed network to be larger than the global optimal of problem in Equation 4.
Recall that aW E J satisfies the following:
V(a, v:) E D, f(W (
yi 2 7.
Now, we can begin the construction of f(W E F. Set the scaling factor:
(Vi/c) -
24

---

We set the weight matrices Published as a conference paper at ICLR 2024
Then, for any weight matrix W, for i E I....,,' L'' - 1, set the value for W; to be:
W, a a- W (v2/c) L W.
Let be the vector of the oulput layer L' in shallow f(W";z). Note that this is an m-dimennionnal vector, since this is the final layer for f(W; But in our construction, layer L' is layer that includes fully connected matrix WL E Rmxm matrix.
So for layer L', we the new matrix be:
V
U
W
0
Om
where Om is the m-dimensional vector of Os.
This means that for the L'-th layer in ((W; we have the following first two neurons: the neuron that has weights which match the corresponding layer from f(W"x), and the neuron that has weights given by negation. Note that since the weights in f(W:z) are constructed directly from the weights of f(W'; z), via scaling the weights through the scaling parameter a defined above, we can satisfy the following for every input x for the output of layer T/ in fWW;
a f(W;z)
c ((W;
fv(W;x))
On
Om
Next, define the weight matrices for the layers: i E {T! + +,...., L- }} to be:
(I/ 1, 1..... L.). L...........
V2 C
W =
where Im is the m identity matrix.
The last layer L in f(W; is set to be .0......o ToT E Rm For this construction, we shall now prove that
2
2
00
/(W;,)) ;2) ((:;+) for every input x.
For any input the output of layer T'' in f(W'x)) is:
ReLU (a f(W; ReLU (-a* f(W:z))
0m
Given our construction for the layers that follow we get for the last but one layer:
L-I'-1
2
ReLU (a+ f(W;z))
1-
ReLU (-at f(W;x))
0,1
25

---

Published as a conference paper at ICLR 2024
4
-
Hitting the above with the last layer
2
v2
.0....,00 00T we get:
1-1'11
2
(
ReLU (a f(W;z))
1-1'11
V2
ReLU (-a* x)W
(L-1'
1- E²'
2
v2
+(W:z) f(W;z).
Thus, ,(W; = I(W": Vx.
If W- [W1.... Wz] be the parameters of wider and deeper network f(W; and if fW"; " be the network with the parameters W achieving the global optimum of the constrained optimization problem in equation 4.
Because we have that f(W*; is of depth L > L' and has m neurons where m > m' and is optimum for the squared l norm minimization problem, conclude that ||W* ||2 < [[]] Therefore.
||* < |
(V2/0) " limal?
(1770)) ( [
+ +( L' -1) ((v2/)) 4' 4/ 22 (122/)) 4/
(2(I'- (1776) (1) 20
(2[ 1/ -1) +2) (12//)) uuuuti?
c² +1) (11//) (113/0) 4//
+ + L1) (117))))) "/40)))
(3/09)-111 (2 + +1) + (3/02) 2(L L')
-2 -2/88/²)) / (L' + 1) (1/0)- 2(L L') = 22/02) "/16 (L 1).
Since f is global optimum of equation 4, also show that it satisfies:
|| = || i<j. i [z]
By the lemma, there is C* > 0 such that C* = W for all E [T]. To see why this is true, consider the x(W where W. n,W and W; = 1/,,W,. for some j and i, [4]. By the property of homogeneous networks we have that for every input x, we get: f(W;z) = f(W*;z). We how the sum of the weight norm squares change with small change in yy:
d (n) [W:l + (1/13) =
dn
26

---

Published as a conference paper at ICLR 2024
17 1, since W* is the optimal solution. Taking the derivative get: dy [
1113 (2) = | 2 for any i < and i E [L].
For this expression be zero, we must have W = W
Based on the above result we can go back to our derivation of || 2 2230-1111 (I. + 1). Next, we can see that for every i E [T.] we have:
-1/L
2
-'1L
C*²
L 22 C
(L+1)
C²2 2 C
2
L
1//b
1
C
1/0" >
11/2+1
V2
V2
Now we use the fact that Væ X, the norm ||
S (x:) S |ل|||
II SE(L]
Willop
II
WWillop
S
Σ
W
1lop
op
xE[Z]
iE(Z]
Thus: 1 Exery I lap > 1. Plugging this into the lower bound
C²:
7,
Σ
|W
W
C
T.
Σ
W
2
V2
v2
C
14/+1-11
(6)
iE(T)
4
22
C
L+1
This further implies that Vi [L]:
V2
V2-"/
Wille ||W lop V2.
T- T-1//
C
v2 ( E'
Setting C²
we get the final result:
L +
|W & llop C.IL
vi e [n].
L
From Lemma D.4 we know that at each layer the weight matrices are almost rank 1, i.e., for each layer E [L], there exists vector Uy. such that ||bl| 1 and W,, s dyUzu for some oj > 0. More formally, we know that for any L L' and 2 m' satisfying the conditions in Lemma D.4. for every layer j we have:
|I( - -, VI - of
Next, substitute the previous bound that derived: || 11 I a, o,C' following, when gradient flow converges to the globally optimal solution of equation
.
get the
|I( - oi- WW < o, CizIL LLY/L S 10/4
L+1
V
27

---

Published as a conference paper at ICLR 2024
Now are ready to derive the final bound the last layer activations:
Eg-Pooo ((WW; = EOOD
(((x)) ...))l
-ooo WL WLo WW
[V)(W;))]
(W
=llf((Wllz PooD WWo
[ ...))l
Froo (l)
II ==j+1
// S FooD [l)( Co
where the final inequality repeatedly applies Cauchy-Schwartz on a norm one vector: // 12222 along with another property of ReLU activations: ||o(u))l2 S ||l|22-
Eooo IW) $ V
E Poon 11
$ V
EPooD -W
off2 + 111 13,
Since, Eroon I |lun1] W,lls and
E
1 5 (62 + 11[I - (f
So (22 ORRI L+1/L) S <a (L+1/L)
Recall that or S C./L. From the above, we get the following result:
EzoPoon [[W;; 2)12 s ColE (W) S Cole + C v1+1/L),
which completes the proof of Thcorem D.3.
D.3 BIAS LEARNT FOR NEARLY HOMOGENEOUS NETS
In this subsection, we analyze a slightly modified form of typical deep homogeneous networks with ReLU activations. To study the accumulation of model constants, analyze the class of functions == --) + b b R, f(W:-) E F). which consists of deep homogeneous networks with a bias term in the final layer. In Proposition 4.2, we show that there exists a set of margin points (analogous to support vectors in the linear setting) which solely determines the model's bias
Proposition D.2 (Analyzing network bias) If gradient flow on f converges directionally to W, b, then En yk for margin points ((Ix, yk) yk f(W; = arg min Yy f(W;;,))
Proof.
Lemmas C.8, C.9 from Lyu & Li (2019) can be proven for gradient flow over J as well since all need to do is construct h...... fy such that h; satisfoes forst order stationarity for the ith constraint in the following optimization problem, that is only lightly modified version of problem instance P in Lyu & Li (2019):
min L(W
Σ will
W.b
s.t. V.f(W; 2 b, We [N]
28

---

Published as a conference paper at ICLR 2024
Note that the above problem instance also satisfies MFCQ (Mangasarian- Fromovitz Constraint Qualification), which can also be shown directly using Lemma C.7 from Lyu & Li (2019).
Mrngasarian-Fromoviizz
As consequence of the above,
show that using gradient flow to optimize the objective:
min W.b.
1 N
Σ (2.
exp (-y y. (f +),
also converges in direction the KKT point of the above optimization problem with the nearly homogeneous networks F. This result is also in line with the result for linear non-homogeneous networks derived Soudry et al. (2018).
Finally, at directional convergence, the gradient the loss aL(WDD quence of the asymptotic analysis in Lyu & Li (2019).
converges, as a direct conse- aw
Let us denote the set margin points at convergence as M {( yx) (WW minje[[] f {. These, preciscly the set of points for which the constraint in the above optimization problem is tight, and the gradients of their objectives are the only contributors in the construction of h1 ..,..., hx for Lemma C.8 in Lyu & Li (2019). Thus, it is easy to that convergence the gradient directionally converges to the following value, which is purely determined only by the margin points M.
0 L(W,b:D) aW
ΣnEM Uk Vw ((W:x)) [[ER Us fwW mx)]ll
lim
1-00
b: aW
Similarly we can take the derivative of the objective with respect to the bias b, and verify its direction. For that, note that: ( 0b
- exp(ex(f((M))))) +b)), but more importantly, exp(=v(f(W;xz) + b) evaluates to the same value for all margin points in MM. Hence,
8 L(W.b; b;
ΣKEM Us |EREM Vel
lim
26 1 b;
While both b and W have converged directionally, their norms keep increasing, similar to analysis in other works (Soudry et al., 2018; Huh et al., 2021; Galanti et al., 2022; Lyu & Li, 2019; Timor et al., 2023). Thus, from the above result it is easy that is just given by the sum of the labels of the margin points in the binary classification task. This direction also matches the OCS solution direction (that only depends the label marginal) if the label marginal distribution matches the distribution of the targets on the support points. This completes the proof of Proposition D.2.
that bias keeps increasing along the direction
29


Reviews:
[{"review_id": "TECQGKVav2", "reviewer": "Reviewer_DSRT", "summary": "The paper proposes the \"reversion to OCS\" hypothesis: Neural network predictions often tend towards a constant value as input data becomes increasingly OOD; that constant value closely approximates the optimal constant solution (OCS), which is the prediction that minimizes the average loss over the training data without observing the input. This hypothesis is verified by empirical results on a wide range of datasets and architectures with different input modalities and loss functions. Moreover, the paper also provides theoretical results to explain the observed behavior. Specifically, the feature norm of each neural layer can drop easily with OOD inputs, which shows why the model's output converges to the OCS when the input becomes more OOD. Finally, the authors leverage this insight to enable risk-sensitive decision-making.", "questions": "1. Do behaviors of $W_i \\phi_I(x)$ mentioned in Section 4.1 remain on realistic datasets, such as ImageNet and Amazon? \n\n2. From the insight of the paper, can we say that methods that try to improve models’ performances on OOD inputs are actually moving the models’ outputs away from the OCS?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Finding of the paper is interesting. \n\n* Paper writing is careful and clear. \n\n* The paper includes detailed evidence, both empirically and theoretically, for their claims.", "weaknesses": "There is no significant weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tXNR2skk9l", "reviewer": "Reviewer_KwvU", "summary": "The paper aims to show that, contrary to the (somewhat) common belief, OOD inputs mostly lead to predictions closer to the average of training labels (more specifically, the optimal constant solution, OCS, minimizing the training loss without the inputs), as opposed to overconfident incorrect results reported in many previous works. The paper first demonstrates the strong (negative) correlation between the distance to OCS and the OOD score (estimated by separately trained low-capacity models) over 8 image & text datasets using ResNet, VGG and DistilBERT (Fig 3), then shows the reason is that OOD inputs tend to have smaller projected feature norms $||Wh||$ (Fig 4), which could be theoretically explained within homogeneous DNNs with ReLU activations (Thm 4.1). Finally the paper presents an OCS-based selective classification algorithm using MSE loss, and validates its performance (abstain ratio, reward) against the standard (CE) classification and an oracle (with access to evaluation data) on 4 CV datasets (Fig 6 & 7).", "questions": "* In Fig 7, reward prediction seems to be noticeably better than standard classification on all CIFAR10 noises and particularly OfficeHome even at the training distribution (t), i.e. supposedly not OOD. Is this solely because of abstention (as shown in Fig 6), or does MSE loss somehow works better than CE loss in this case? More generally, how do their accuracies (instead of rewards) compare e.g. without considering the abstained samples?\n* In Fig 4, the normalized norm in early layers for CIFAR10 (bottom left panel) seems to be systematically increasing with noise, contrary to later layers. Is this an expected behavior?\n* Is there a particular reason to use -4 for the incorrect results? Does any number that brings the OCS below 0 (-3.5 in the paper) work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ [Originality] The paper novelly reassesses DNN’s OOD behavior both empirically and theoretically (OCS, OOD score, Sec 4.2), and reports interesting results (dispute of the common belief, OCS-based selective classification).\n+ [Quality] The paper is of sufficient quality in my opinion, with proper empirical (Fig 3 & 4) and theoretical (Thm 4.1, Prop 4.2) validations of the main claims, and experimental evidence of the proposed algorithm’s effectiveness (Fig 6 & 7, although can be further improved, see Weaknesses).\n+ [Clarity] The paper is overall clear and easy to follow, although certain details, e.g. the construction of the $(x,a,r)$ dataset in Sec 5.1, can be further elaborated for better understandability.", "weaknesses": "- [Evaluation] While it’s understandable that this work doesn’t focus on achieving SOTA results, it’s still highly desirable to see how the proposed algorithm compares to existing selective classification (or OOD detection) baselines, and/or how they can be combined to further boost performance (discussion would be fine too).\n- [Significance] While this paper is good in most aspects (as summarized in Strengths), its significance however is a bit insufficient in my opinion and can be substantially improved by addressing the following issues:\n1) Evaluation as stated above. More evaluation can help strengthen the applicability of the paper.\n2) The failure of the claim in some of the distribution shifts (adversarial perturbation and impulse noise in Appendix A) raises concerns about the generalizability of this work. What kinds of OOD shifts (extrapolations) are actually supported and not supported by this work? Is there a way to more formally and/or finely characterize them? More evaluation datasets as well as systematically generated OOD shifts e.g. [1] could be helpful.\n\n[1] A Fine-Grained Analysis on Distribution Shift, ICLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bae8h9uYsk", "reviewer": "Reviewer_Spup", "summary": "This paper focuses on the extrapolation of neural networks. It formulates a hypothesis that neural networks tend to move towards a constant value as the distribution shift on the input data increases. The first empirical observation exhibits that the norm of the feature representations decreases when the data are OOD and not sampled from ID. Then, the paper argues that this translates into the input-independent parts of the network (e.g. bias vectors) to dominate the output representation, thus explaining the more uniform prediction. The paper further argues that this constant value is close to the optimal constant solution (OCS). Then the paper focuses on why the OOD predictions have this tendency to move towards the OCS; the paper associates this behavior with the low-rank assumption on the learnt subspaces. The paper conducts a series of experiments with standard neural networks (e.g. resnet) to verify this hypothesis empirically. An application in high-risk decision making is also presented to exhibit the usefulness of this observation.", "questions": "Beyond the weaknesses above, the following questions come to mind about this submission: \n\n- In fig. 4, I am wondering what the norm is for ID data; is it the case that the norm is also decreasing on those networks as well? \n\n- In theorem 4.1 it mentions a “shallow” network with L’ layers. Is this a contradiction? \n\n- Even though not strictly mandatory, I am wondering whether structures that differ from the feedforward structure of MLPs differ in the solution they find. For instance, graph neural networks or multi-path networks. In other words, I am not sure whether the proved extrapolation properties (see Xu et al 2021 and Wu et al 2022 that are already cited) affect the formed hypothesis.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-written, while understanding and improving the OOD of existing networks is important. \n\n+ I personally like sec. A in the appendix, where the paper demonstrates some cases that have an unexpected performance with respect to their hypothesis. \n\n+ The hypothesis seems new to me, while there are empirical results to support the hypothesis.", "weaknesses": "- Some of the training details are opaque in the main paper, which might lead into a simpler explanation over the observed empirical performance. For instance, could the learning algorithm or the data augmentation or the normalization impact this hypothesis? \n\n- I am skeptical about the hypothesis formed in the following sense: even if we assume a zero input, most modern networks rely on a normalization scheme, e.g. batch or layer normalization. Then, in a trained network, the “centering” provided by the learnt means and variances of the network will not result in a zero-mean representation for the next layers. As such, I am wondering how the normalization plays into the formed hypothesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nOqNqrQfIY", "reviewer": "Reviewer_5N2a", "summary": "It is well-established that neural networks are miscalibrated and overconfident on Out-Of-Distribution (OOD) data. At the same time, it is also known that classifiers tend to be less confident on OOD compared to In-Distribution data [1]. \nIn this paper, the authors investigate the reasons why neural networks are less confident on OOD data, and propose the “reversion to the OCS” hypothesis as a possible explanation of this behavior. The Optimal Constant Solution (OCS) is a constant output to which neural networks converge when diverging from ID data. In the case of a classifier, the OCS corresponds to the uniform distribution over the target classes.\n\n**Contributions**\nIn relation to the OCS hypothesis, the paper's contributions are the following:\n- Empirical validation that the learned feature representations on OOD data have smaller norms (Fig. 4). This leads to the fact that the output of OOD is mostly driven by the input-independent components of the model (biases). Accordingly, the OCS is then mostly identified by the biases of the network (Fig. 4). Lastly, the role of the feature norms and network biases on OOD data was investigated theoretically in the restricted setting of Deep ReLU networks (Section 4.2).\n- Empirical validation that as data deviates further from ID data, the output converges to the OCS (Fig. 3).\n- In Section 5, the OCS hypothesis was leveraged as a tool for risk-sensitive decision-making. More in detail, we can define a classifier with a rejection option such that the OCS corresponds to preferring rejection to guessing one class. In this way, following the “reversion to the OCS” hypothesis, the further the classifier with rejection is from ID data, the more is prone to prefer the rejection option.\n\n**Limitations**\nThe authors also addressed the limitations of the OCS hypothesis in the appendix, by showing that the hypothesis does not hold for adversarial examples (Fig. 8) and for specific types of noise (Fig. 9).\n\n**References**\n1. Hendrycks, D., & Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks.", "questions": "- Do you see the OCS hypothesis as an \"average\" behavior of neural networks on OODs? In other works, do you expect that high-confidence OODs have to be constructed (e.g. adversarial examples), while the other ones will mostly respect the OCS hypothesis?\n- In the OCS hypothesis, you clearly stated the assumption that OODs are high-dimensional. Is the OCS not valid on low-dimensional toy examples? \n- In Figure 6, the OfficeHome dataset is the only one that has lower abstain ratio for the oracle compared to the predicted rewards. Is this because of the high number of target classes, compared to the other dataset, that makes it easier to achieve low confidence outputs?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- I found the paper very well written and clear. The experiments are all well presented and detailed.\n- A lot of research has been recently devoted to addressing the issue of detecting overconfident OOD data and quantifying uncertainty in neural networks. This paper shows a result, that, in my opinion, might have been overlooked by the existing literature: while some OOD inputs might have high confidence, the more an input is far from ID data, the more the output *should* converge to some constant value.\n- Existing OOD detectors, among the many, have exploited the norm of the learned features [1,2] and the confidence of the output [3] to detect OODs. This paper connects the two methodologies through the OCS hypothesis, by showing that OOD are less confident and detectable thanks to the lower feature norms and the reliance to the bias terms.\n\n**References**\n1. Sun, Yiyou, et al. \"Out-of-distribution detection with deep nearest neighbors.\" International Conference on Machine Learning. PMLR, 2022.\n2. Tack, Jihoon, et al. \"Csi: Novelty detection via contrastive learning on distributionally shifted instances.\" Advances in neural information processing systems 33 (2020): 11839-11852.\n3. Liang, S., Li, Y., & Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks.", "weaknesses": "**Weaknesses**\n- Lack of references: the norm of the learned features was already known to be a discriminant characteristic between ID and OOD data (see, e.g., [1] and [2] cited above).\n- Lack of discussion on when the OCS does not hold. Although I understand that the results on the OCS hypothesis are still preliminary, I would like some additional remarks on when it might fail (as you showed in the appendix, but also as shown in works such as [1]). I suggest to briefly address this aspect in the introduction (see also Questions).\n- I don't understand the conclusions drawn from the decision-making scenario. Ideas contained in the Hendrycks baselines paper ([1] in the summary) implicitly assume the OCS: if we choose an abstain threshold on the output confidence of a standard classifier, we already expect a non-zero abstain ratio on OOD data.  In general, anomaly detectors on neural networks based on the output confidence are based on these ideas. Could you explain better the intuitions provided by section 5 on exploiting the OCS? How does the OCS hypothesis possibly change the way in which we define safer AI models with rejection options, compared to simple threshold-based ones?\n\n**Typos**:\n- Page 1: \"Therefore, the our hypothesis...\"\n\n**Concluding remarks**\nOverall, I liked the paper thanks to its clarity and valid contributions, but I have some doubts on section 5 and the connections to related works. My final rating will depend on the other reviewer's comments and the authors' response. \n\n**References**\n1. Hein, Matthias, Maksym Andriushchenko, and Julian Bitterwolf. \"Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)