Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture
Authors: Sajad Movahedi
Antonio Orvieto
Seyed-Mohsen Moosavi-Dezfooli
Venue: ICLR 2025 Spotlight (2025)
Keywords: Optimization
Deep Neural Networks
Deep Learning
Deep Learning Theory
Machine Learning Theory
Primary area: optimization
TL;DR: We try to understand the role of architecture and data in the geometry of deep neural networks in the input space and how it evolves during training.

Abstract: In this paper, we propose the *geometric invariance hypothesis (GIH)*, which argues that the input space curvature of a neural network remains invariant under transformation in certain architecture-dependent directions during training. We investigate a simple, non-linear binary classification problem residing on a plane in a high dimensional space and observe that&#151;unlike MLPs&#151;ResNets fail to generalize depending on the orientation of the plane. Motivated by this example, we define a neural network's **average geometry** and **average geometry evolution** as compact *architecture-dependent* summaries of the model's input-output geometry and its evolution during training. By investigating the average geometry evolution at initialization, we discover that the geometry of a neural network evolves according to the data covariance projected onto its average geometry. This means that the geometry only changes in a subset of the input space when the average geometry is low-rank, such as in ResNets. This causes an architecture-dependent invariance property in the input space curvature, which we dub GIH. Finally, we present extensive experimental results to observe the consequences of GIH and how it relates to generalization in neural networks.

Published as a conference paper at ICLR 2025
GEOMETRIC INDUCTIVE BIASES OF DEEP NETWORKS: THE ROLE OF DATA AND ARCHITECTURE
Sajad Movahedi 2,3, Antonio Orvieto 2,3 & Seyed-Mohsen Moosavi-Dezfooli* 'ELLIS Institute Tübingen. 2MPI for Intelligent Systems, 'Tübingen AI Center, 4 "Apple (sa jad. movahedi, antonio)@tue ellis.eu, airosaviiaaplle. .cm com
ABSTRACT
In this puper, we propose the geometric invariance hypothesis (GIH), which argues that the input space curvature of neural network remains invariant under transformation directions during training. We investigate a simple,
in certain non-linear binary classification problem residing on plane in high dimensional MLPs- Re.Ps-ResNets fail to generalize depending on the
space and observe orientation of the plane. Motivated by this example, we define neural network's 's av- erage geometry and average geometry evolution as compact prchitecture-depente summaries of the model's ouput- geometry and its evolution during training. By investigating the average geometry evolution at initialization, we discover that the geometry of neural network evolves according to the data covariance projected onto its average geometry. This means that the geometry only changes in subset of the input space when the average geometry is low-rank, such in ResNets. This causes an mrchitectur--depeneent invariance property in the input space curvature, which we dub GIH. Finally, we present extensive experimental results to observe the consequences of GIH and how it relates to generalization in neural networks. The code for this paper is available at pitttps: http//gtttttttt thub. com/ wr--dffuuttt faustusuH
INTRODUCTION
Ever since the advent of deep learning, the importance of architecture and its impact on performance, es- pecially through introducing inductive biases, have been well-known and established (Battaglia et al.] (2188] Tay et al. 2023a b White al.) 2023). However, the modern machine leamingg toolbox still lacks unified tool that can provide an interpretable and tangible connection between architecture, data, and inductive This has fragmented the approaches to architecture design into heuristic-based modifications (He
biases.
et al.
2016 loffe & Szegedy [2055] Hochreiter & Schmidhuber 1997). biology-inspired design (Bahdanau 2015 Vaswani et al. [2017] LeCun et al. 2015)) or efficiency-guided decisions (Dosovitskiy et al. Gu & Dao 2023), among other things. Many consequential characteristics of deep neural network such as decision boundaries, smoothness, and the types of features the model relies on for prediction can be determined from the input space. We propose the geometry of model in the input space as new tool to relate neural architectures and their inductive biases.
ctt al.
202
Consider a simple D-dimensional non-linear binary classification problem, where the discriminative fea- tures of the two classes are two 2-dimensional concentric circles (Figure T occupying a plane determined by two orthogonal directions (u, u2) E R.P. The other D 2 dimensions the variation in those dimensions is caused by i.i.d. and zero mean Gaussian noise. Solving such a problem with a nonlinear decision boundary was one of the motivations behind the exodus of the machine leamingg community from linear models (logistic regression, SVMs, etc.) to kernel methods and then neural networks (Bishop 2007 Goodfellow et al. 2016). Conventional wisdom tells us that no matter on which directions the circles reside on, modern neural network is complex enough to learn this decision boundary with ease (Zhang al., 2017 (002T) For an MLP, this indeed the case: any model that is wide enough and uses non- con-linee with at least a single hidden layer can learn this decision boundary for any orthogonal ug However, a standard ResNet18 (He et al., 2016) is seldom capable of generalizing for randomly selected un Uy despite its elaborate architecture design and much larger size. This observation indicates the existence of an architecture dependent geometric inductive bias in the input space, which raises the question:
unstructured, meaning
Work done while Imperial College London.

---

Published as a conference paper at ICLR 2025
How does the geometry of a neural network in the input space evolve during training. and what is the role of architecture and data in it.
In our quest to answer this question, we discover a phenomenon, which we dub the Geometrie Invariance Hypothesis (GIH). GIH claims that for a family of neural networks F, there exists an architecture-depender subspace Sy C RD where the geometry of neural networks in F can only change directions u y during training. In the context of our example. let us assume two datasets DA and D3 sharing x-axis 11 =v but have different y-axes U2 = V E ESy for DA and U1 =v for Dg. with F corresponding to the ResNet18 architecture. As evident by Figure I in this case the neural network is able to achieve generalization on DA. but not on Dg. More specifically, the decision-boundarte boundaries (green curves) show us that in DA the model is able to find the optimal, circular-shaped decision boundary regardless of the stochasticity introduced by initialization or biiiibbachinng However, in Dg, the model missing the signal on the directions corresponding to U2, relying on noise on other dimensions ESy in order to reduce the train loss, and thus failing to generalize.
100
Train Loss
Trein Long
Test Accuracy
Train Loss
Test Accuracy
20
# Epoch
* Epoch
VAA
B
1l VB
DA
Dp
v
C2
G2
Geometric Invariance Hypothesis in ResNet. A simple binary classification problem in R.D. The data is structured as two concentric circles in the plane defined by axes un U2 and otherwise uncorrelated. The axis u, is aligned with the aver- age geometry, i.e. uy G3 > O, and shared between both datasets D and D B. The axis uy is aligned with G in DA the left and not aligned with G in D on the right (see projections on bottom left corners). The green curves demonstrate the decision boundaries of the model for different initializations and mini- batches. Observe that the model can only generalize if both uı and u2 are aligned with the average geometry.
Figure 1:
In order to formalize a notion of "geometry in the input space" for a neural architecture and how it "evolves" during training, we define two distinct but related maps for neural network with scalar output, which we dub the average geometry (G'z at time t during training and for the family of models F) and the average geometry evolution (A'r at time during training), respectively. These two maps correspond to the covariance of the gradient of a neural network w.r.t. the input and its derivative w.r.t. time according to the gradient flow model (Barrett & DDhriin 202T). We find these functions to be related to decision boundaries through a first-order Taylor approximation, and also to the Hessian of the loss w.r.t. the input. Through investigating A at initialization (t (t=00p 0) for a simple MLP and CNN architecture, we observe that the changes in the input geometry of neural network at initialization correspond to the projection of the train data covariance onto its geometry at initialization. Hence, we hypothesize a similar behavior for the remainder of training, which brings us to our definition of GIH, whereupon the role of S, is taken by the subspace spanned by the eigenvectors of GD. We confirm GIH through experimentation
synthetic data and CIFAR-10.
Related Works. The inductive bias of neural networks are seen as one of the missing links explaining the generalization ability of deep neural networks (Soudry et al. 2018 Barrett & Dherin 2021 [Smith cct al. 2021), while also being an important tool in architecture design (Battagla et al.) [018 Tay et al.
2023a). The majority of research focused identifying and explaining the inductive biases of deep neural networks are focused on the behavior of the model in the parameter-space (Jacot et al.] 2018 Atanasov
let al. 2022 Mahankali et al.) 2023 Tu et al. [2224] in the frequency domain (Basri al. 2019 Teney et al. 2024). or in the function domain (Nye & Saxe 2018 Pérez et al. 2019 Shah et al. 2020). The
closest of such attempts to our work the neural anisotropy directions (NADs) paper (Ortiz-Jiménez al.
2020). which provides empirical evidence for an existing geometric inductive bias in deep neural networks in linear classification. Specifically, they empirically show that the eigenvectors and the eigenvalues of the covariance of the gradient of a neural network w.r.t. the input provides robust method for identifying and ranking directions (called NADs) which the network prefers to classify the data in linear classification task. As a follow-up, it was later shown that these directions can be connected to the set eigenfunctions of the NTK of a neural network in linear task (Ortiz-liménez et al. (022].. In this paper, we improve upon these works by providing an explanation for this behavior through analytical and empirical investigation, while also removing the linearity assumption from the results. As a result of this investigation, we are
Note that this paper, we are specifically focusing on geometric inductive hiases initialization.
2

---

Published as a conference paper at ICLR 2025
also able to justify the proposed method in -Ortiz-Jiménez et al. 2020 for identifying the NADs. For a detailed summary of the related works, please refer to App. A.I
Contributions. Our contributions can be summarized as follows:
We first define a function summarizing the input space geometry of a model and its evolution. We then factorize the contributors to the evolution of geometry during training into data-dependent term and a model-dependent tern. We investigate each element and their interaction theoretically, from which we conjecture general form. Then, the results are confirmed empirically.
Following these insights, we propose The Geometric Invariance Hypothesis (GIH), which states that the geometry of deep learning models in the input space becomes invariant in certain directions determined by the architecture.
We provide extensive experimental results aimed at establishing the validity of GIH and how it relates to generalization in neural networks. Specifically, we show that through GIH, we can deter- mine the location of the decision boundaries a neural network can lear. Furthermore, we observe that GIH can help us detect the features and train samples the model relies on for generalization.
For details on the experimental settings and discussion on the limitations of our work, please refer to App. A.12 and App. A.13 respectively.
2 THE GEOMETRY OF THE INPUT SPACE
In this section, will first provide the notations in our paper. Then, we provide preliminary definitions and results which we will use to formalize and motivate our claims in the rest of the paper.
2.1 NOTATIONS
We denote the pre-activation output of the classification layer of the model, which for simplicity we assume be scalar (i.e., dealing with binary classification problems) as fx(x,, parameterized by with We assume the model to be neural network, trained on Dr = ((,, (,)) where x, is the input and yμ is the label. We denote family of models (i.e., architecture) as F, with fo(.) EF. We denote the loss function as L(0)= Em= E (x)))))))) where L(.) the sum of squared errors (SSE, refer to Ap2. for details) our theoretical analysis, and cross-entropy in our experiments. Furthermore, we denote the gradient w.r.t the input and the parameters by vxfo(x) and V (x)) respectively, and the second-order derivative w.r.t. the parameters and data as v (xx). For ease of analysis, we rely entirely on the gradient flow model for modeling the training procedure: i.e., we denote t E Ro as the t 0 moment in training and assume the optimization process to be continuous. So investigate the changes in a particular function 97(x,0), which is a function of input and parameters 0 for the family of model F, we usually rely on daz Vog=(x,0) 0, where 0 de dr
x E ER..
dr
We also use probing distribution over the input to get the average behavior of particular function (x, which we denote x P. The probing distribution can be any distribution of choice that covers the area of interest in the input space. For instance, we can use train or validation samples, or standard Gaussian distribution, which is our choice for P in this paper. We also define A ma. B as |Corr(A, A, B) approaching at the limit Vi m, - oc, where and B are both matrices of the same dimension, Corr(;-) is the Frobenius cosine similarity, |-| is the absolute value operator, and A is function of scalars {m²} H More informally, we use A x when we claim [Corr(A, | for the two matrices A and B to be much larger than the magnitude of the cosine similarity between two randomly generated vectors of the same dimension to the point of one being able to approximate the other.
2.2 DEFINITIONS
In order to facilitate answering our central question, we will try to define two functions that help us understand the geometry of a neural network architecture in the input space, and how changes during training. Given that our focus in this paper is on classification tasks, a good starting point would be the gradient of the model w.r.t. the input (xfe From robustness literature, we know that Vxfo(x) (x) can inform about the decision-boundarier in the immediate vicinity of x (Madry et al., 2018 Moosavi-Dezfooli et al. (0166). This can be easily shown through a first-order Taylor expansion of the output of the model. Specifically, let be a vector with small 1.2 norm. Then we can write
3

---

Published as a conference paper at ICLR 2025
= 8 (x) using Taylor expansion. Therefore, 5 in the direction of Vx) maximizes the change in the output of the model. potentially flipping its prediction. Consequently, by maximizing 6 (x) we obtain valuable information about the local changes in fo(x) through 6.
However, a 6 computed based on Vx)f(x) for a single value of 0 only informs us about a decision- noise caused by stochastic factors such as mini-batching and initialization. Therefore, we propose to average the gradient over a distribution of 0: Eg (x)]. While trying to maximize 6" Eg Vo (x)] can be way to find direction that can potentially flip the prediction of fx(x), it only contains first-order information about how the distribution of 9 affects the local changes in fo(-) at x, causing potential loss of information. For instance, let us assuine a case where (x) is zero-mean w.r.t. 8, i.c., x corresponds to a local extremum. In this case, Eo[V [o))(x) does not contain any information about the local changes fo (x). . On the other hand, one can instead look at the covariance of V Vxfo(xx (x)
boundary for single parameterization of the model, subjecting
w.r.t. 0 to obtain information about the directions towards which the decision boundaries have significant presence. Therefore, a much more informative function about the local changes in fx) in the input space would be lE [x) (x) ']. Finally, since we rarely care about the behavior of a model for single input, we also use a probing distribution P to sample from. With this motivation, the geometry of a neural network architecture in the input space:
introduce the following definition for function informing us about
Definition 2.1. Let F be a family of neural networks. We define the average geometry of F at x and train time & as:
Gx) [x<fo(x
(I)
where Te determines the distribution of the trajectory of 0 during training on the train data Dr at moment 4, with the source of stochasticity usually being initialization and mini-batching. Furthermore, we define the average geometry of F induced by the probing distribution P as G -x (x))
In App. A.3
report that Gx)( is indeed directly related to the curvature of the loss w.r.t input, making the connection between the function we propose and the geometry of a neural network architecture in the input space clear.
We can capture the geometry of a family of models in the input through G)-(x) at time / during training. However, this function does not tell us how the model geometry changes at time f. thus only providing half of the picture. So we need another quantity that can inform us about the evolution of the geometry in the input space. Given our line of reasoning that resulted in the definition of average geometry, we start with the function V (x) (0o(xx)Vxxff (x)) Following the gradient- fflle model, we can capture the evolution of this func- tion by getting its derivative w.r.t. time, which gives us: V2 (x)6V T + Vxfo(x)0 V2 (x)) T. With a similar motivation, we get the expectation of this quantity w.r.t. the parameters A and x sampled from a probing distribution, resulting in the following definition:
Delinition 2.2. Let F be family of neural networks. We define the average geometry evolution of F at x and train time as:
(x(x)) V=nfo(x)dV<f((x)) +Ea
Eo-T. ( [x
(2)
where T; determines the distribution of the trajectory of I during training on the train data Dr at moment f, with the source of stochasticity usually being initialization and mini- batching. Furthemnore, we define the average geometry evolution of F induced by the probing distribution P as > Ey-p (x).
In App. A.3
we will see that A (x) is indeed directly related to the evolution of the curvature of the loss w.r.t. input, making the connection between the function and the evolution of geometry of a neural network architecture in the input space clear. For the SSE loss we have (x (+xx)=D(xxx)) (x)) where:
m
(x( - > Eo, [ (x) Vola, (x,, V (x) (Fp, (x,,) -w.)]]
u=1
Following ( we can argue that the changes in the local geometry are the result of the interaction between two main components: 1) the data, and 2) the current average geometry. The first component affects the geometry through the changes in the parameter (0) by gradient-descent, while the second component affects the geometry through the mixed-derivative V2 fo(x) and the gradient (x) While such an interaction may seem extremely complex at first, we will see that by probing the average geometry and
average geometry evolution with an appropriate probing distribution P. we observe (through theoretical and empirical means) that the expected value of this interaction is actually suppisingly simple and akin
4

---

Published as a conference paper at ICLR 2025
to linear projection, leading to interesting results the behavior of the average geometry G and generalization in a neural network.
For simplicity, refer to both of the average geometry and the average geometry evolution of standard Gaussian probing distribution and at initialization (i.c., t= as Gy and Ar, respectively. For t> 0, we denote these two functions as G' and Aj.. We name S Tu the unnormalized data covariance.
3 CHANGES IN THE AVERAGE GEOMETRY DURING TRAINING
In this section, we study the factors that affect the changes in the geometry of the model in the input space through theoretical and empirical means. More specifically, we will start by theoretically detecting the data-dependent factor in AF by investigating it in an "isotropic" model. The term "isotropic," as will become clear in this section, indicates a lack of geometric inductive bias in the model. We then theoretically look at the interaction between the data-dependent and model deeeeddependent factors in a "nonn-sotrropic" model, which we set to be a convolutional neural network with a pooling layer. Motivated by these theoretical results, we will conjecture that at the initial stage of training, the changes in the average geometry of the model follow a linear dynamic driven by an interaction between the initial geometry (Gx²) due to the model and the covariance of the data. In App. A.44 we provide theoretical and empirical results supporting the claim that this dynamic is independent of the labels, and can be considered nos-iindependet.
3.1 ISOTROPIC MODEL.
We provide theoretical results for the average geometry evolution A F at initialization according to the gradient flow model, over the probing distribution P 1 We do not make assumptions about the data, and instead focus on models that can be considered "isotropic" (Battaglia et al. 2018).
First, let us start with simple example of a linear regression (i.e.. F corresponding linear regression mod- els), which we write as fx(x) = x. We assume that the parameters are imitialized as zero-mean and i..d., as the common practice. Since Vx (x) =0, the average geometry at initialization for this model will be =x= [000] I. This property indicates a "lack of geometric inductive biases" at initialization, which as shown in App. A.5 is also shared with MLPs of any depth with ReLU non- loneaiiress..es. In Section 3.2 we will elaborate on the significance of this property. Following (3). and given that V fx = I, we
-m le,, [(0 x, yu, xx,0 = for an i.i.d and zero-mean Gaussian initialization of 0 with standard deviation σs. following Stein's 's lemma. Therefore, we can see that in the absence of non-linearity and more layers, the average geometry evolution of an isotropic model at initialization corre- sponds to the covariance of the data. When introducing these two elements, in the form of an MLP with a single hidden layer and ReLU non- non-linearity, we have the following theorem that shows a similar behavior:
have A== 2
Theorem 3.1. Let F be the family of MLPs with a single hidden layer of size n and ReLU non non-linearity. Assuming that we use the SSE loss, then as the imput dimension D and the model width n become larger, the average geometry at initialization DJ approaches the data covariance S up to constant, i.e., AF n.D. +S S. The convergence rate is O (DD)
We provide a sketch of the proof and encourage the reader to read the whole proof in App. A.11.2 The average geometry evolution of this model at initialization corresponds to n² terms of the second and fourth moments of the parameters the first layer over the space shared by halfspaces defined by the train data and the inputs sampled from the probing distribution, i.e. Exp [ x==0==00xxxx. xxx=00 x,,x, and
Eg L To T x,,) "0oT] As the input dimension becomes large enough. these two halfspaces will almost surely become orthogonal, resulting in the second and the fourth moments of the first-layer parameters over the intersection of these two halfspaces becoming equal to the second and the fourth moments up to constant. The second moments yield n² -n terms corresponding to S, while the fourth moments yield n²² 222- terms corresponding to S and n terms corresponding to Ex, [| [2] I. which vanish in favor of the S terms the depth of the first layer becomes large enough.
Theorem 3.1 gives us an interesting insight into our factorization of A underscoring that in the absence of
geometric inductive biases introduced by the model, the geometry adapts according to the data covariance.
GGiven the homogeneous quality of the architectures used in our theoretical analysis, we will set og =1 for the probing distribution.
5

---

Published as a conference paper at ICLR 2025
Therefore, the data-dependent component in is equivalent to the empirical covariance of the data estimated on the train samples.
3.2 THE LINEAR RELATIONSHIP BETWEEN GEOMETRY, DATA, AND ARCHITECTURE
We now shift our focus examining the effect of the model on the geometry and the interaction between the data-dependent and the model-dependent components in Az. Specifically, we aim to introduce architecture components, such as convolution operations and pooling layers, that as noted in App. A.. introduce geometric inductive biases to the model. We start our investigation by looking at a neural network with a linear activation function:
Theorem 3.2. Let F be the family of convolutional neural networks with a single hidden layer, linear activation function, and a global average pooling layer on each channel. Assuming that we use the SSE loss, then the model width n becomes larger, the average geometry evolution at initialization Ax approaches G5 (x) up to a constant, i.e., (x) "y (x(x))GG((() The convergence rate O(1/n). For proof, please refer to App. A.11.3
From Theorem 3.2 we observe that in the initial stages of training, G will change towards the linear
projection of S onto G in a linear convolutional neural network with pooling. Considering the results of
Theorem 3.T and Theorem[3.2 we can now better understand the effect of the two components present in (3) causing the shift in geometry at the beginning of training: the data in the form of covariance, and the architec- ture in the form of average geometry. We provide empirical confirmation for the two theorems in Figure] 7
Given the results of Theorem 3.1 and without making assumptions about the architecture. From Theorem 3.1 we know that for the gradient-flow model t 0, the changes in the geometry in the presence of isotropic model (more explanation in App. A correspond to the covariance of the data. From Theorem [32 we understand that the data covariance linearly projected on the average geometry at initialization G for the non-isotropic model used in the theorem. Then, without any assumptions about the architecture we have the following:
and Theorem 3.2,
we will conjecture about A for neural network
Conjecture 1. Let F be family of neural networks. Then, assuming we use the SSE loss, the average geometry evolution at initialization A is highly correlated with the projection of data covariance onto the average geometry at initialization, i.c., A GG As a result, the average geometry will change according to this form at the initial stages of training, i... G approaches GGSGF for t close to O.
So the changes in the geometry are caused by the projection of the covariance of the data onto the initial average geometry. As result, in the case of the existence of structure in the initial average geometry Gy, one can fully expect the geometry to be invariant (or numerically, changing extremely slowly) in certain directions with low correlation to the initial average geometry.
Considering that the average geometry evolution and our analysis of A'S is based on the gradient-flow model, we cannot confirm Conjecture direecly in a discrete training regime. Therefore, in order to present empirical evidence confirming our results so far, we instead rely on looking at the average geometry G for the entirety of the training period using gradient descent. In this case, an increasing correlation between G and G (or S for an MLP) near the initialization point in training will support our theoretical analysis. We provide the experimental results supporting Conjecture D in Figure 2 In this experiment, we plot the correlation between average geometry at time (G*x) and the projection of data onto the initial aver- age geometry of the model, as proposed in Conjecture [ We will use the normalized Frobenius dot product measure the correlation between two matrices. For our experiments, we use the CIFAR-10 data. We con- struct a binary variant of CIFAR-I0 in which the task is to distinguish animal from non-animal inputs, which we dub CIFAR-2. For each class of CIFAR-2, we sample 5000 data points randomiy Furthermore, we re- port the results for an MLP with 2 hidden layers of size 100, LeNet, ResNet18 without batch normalization, and VIT without layer normalization. More details on the implementation can be found in the appendix.
As we can observe in Figure 2 there is clear evidence for the changes in the average geometry being highly correlated with S and GG at the beginning of training, which is what we claim in Conjecture !
"Note that per the results of Corollary A.4 in case of normalization, G will depend on the variance each individual patch of the input. In our experiments, we found that in practice it is difficult to exactly compute this "normalized" variant of G due to its dependence on among other things the kemel size of the convolution operations the patch size of the VIT. For this reason, and for the sake of simplicity, omit the normalization layers from the architectures we experiment for non-synthetic data.

---

Published as a conference paper at ICLR 2025
0,40 Text Accuracy
22
Constarion
Teat Accuracy
9,30,
2
decoeccenin 201
Conaturinn x &
Text Accurscy
6,30,
6,40, Text Accurscy
12 S
0.2
Eporh
Tpech
Spech
Fterh
(a) MLP
(b) LeNet
(c) ResNer18
(d) VIT
Figure 2: The correlation between Gzz=
G F.N((t.))) and S and Gy for the (a) MLP, (b) LeNet, (c)
ResNet18 without batch normalization, and (d) ViT without layer normalization CIFAR-2. Note that D=32 322 x3 = 3072, which means the expected cosine similarity of two randomly generated vectors in the input space (11v33072) 0.002. Therefore, we consider the correlations significant.
Another interesting observation to note is that the values of S and SG virtually identical in the case of MLPs, which confirms our view of MLPs being isotropic, or in other words, having no geometric inductive biases, which manifests itself in the form of G I. On the other hand, this not the case in the other models, with the correlation between Gÿ and Gy being significantly larger than MLP during the initial training stage compared to S. This difference in correlation indicates the existence of structure in Gy for these models, which we consider to be the geometric inductive biases of the model.
4 THE GEOMETRIC INVARIANCE HYPOTHESIS
In this section, we will introduce the Geometric Invariance Hypothesis (GIH), providing empirical evidence supporting and investigating its implications on the generalization ability of deep networks. Notably, we study the relationship between the complexity of the decision boundary and GIH, establishing a connection between GIH and the simplicity bias hypothesis (SBH) (Pérez et al. 2019
4.1 THE PROJECTION OF DISCRIMINANT FEATURES ONTO THE INITIAL GEOMETRY
In the previous section, we observed an intriguing phenomenon wherein at the initial stages of training. the average geometry of the model changes according to the projection of the covariance of the data onto the initial average geometry Gy. Note that as we get further into the training process, set of features used by the model to narrow down to "discriminant features," " which can be viewed as the features upon which the model can discriminate data. Assuming that the changes in the geometry are still caused by the projection of a subspace of the data support (relating to the discriminant features) onto the initial average geometry, and the initial average geometry itself is structured, then the input geometry will remain invariant in the directions with low correlation with G In App. A.5
expect the
we consider the choices in
the architecture that can result in structure in Gy. Furthermore, App. A.6 we will observe the structure in G for the neural networks used in this paper.
This observation indicates the possibility that the initial geometry also plays role in the later stages of training. More specifically, we want to know whether the model's geometry in the input space will remain invariant during later stages of training in directions with low correlation with G as well. We formalize this concept in the form of the following hypothesis, which we dub the Geometric Invariance Hypothesis" (GIH):
Conjecture 2. (Geometric Invariance Hypothesis) Let F be a family of neural networks. Let Fig( A) correspond to the subspace defined by the top eigenvalues of the matrix A. We conjecture that Eig( A C Eig( G, In other words, the geometry of the model remains invariant in directions not in Big(G= during training.
In order to support Conjecture 2 we first define a measure monitor the changes in the average geometry, which we call the geometric velocity: G, = 1 - Corr (G,,G""") where Corr(-:) is the normalized Frobenius dot product. For this experiment, we will look at G3 and training accuracy when the data sampled from the Gaussian distribution N (0, (tGG////Gaaala G)llz). where |G |2 the spectral norm of
"Note that while we use the term invariance to refer to the geometry of the model having small changes in certain directions, in practice this "invariance" comes in the form of a large condition number in G As result, the invariance happens a spectrum, and the changes in G in a specific direction gradually decreases as the correlation between and G5 hecomes smaller.
7

---

Published as a conference paper at ICLR 2025
120
120
0.5
G,
100
C. 1et
100
1 Cize
Cov
3
Tonn
Gy RziG. Cov
20 Spoch
25
3G
73
100
20 Spoct
60
Spocn 40
(a) MLP
(b) LeNet
(c) ResNet18
(d) VIT
Figure The train accuracy (green lines) and velocity G:.:-) (blue lines) of the (a) MLP, (b) LeNet, (c)
R8 without batch normalization, and (d) ViT without layer normalization on two synthetic datasets: G5 covariance x ~ N (0,G= /llGG/l/GGar |=]|l and flip (Gx) covariance x ~ (0,fiip(G / [ with random labels. A horizontal line for velocity indicates no change in the geometry.
100-
Lineat
Quadrabic
10-1
Test oy (Np)
1000
Linear
Quadratic
110k
100
Lineat
Quadratic
10-
80-
Eiganvatue
Siassiddal
80-
Elgenvalue
Siruseidal
Elgenvatee
Accunaty
80-
Eiganvalue
Smussids!
10
10-
10
"Am Maople / M
10
42
60
10-13
1000 Eigerrealue Indea
3000
3000
2000
3000
10000
2000
3000
Eigervalue Indrex
Eigerriius Indea
(a) LeNet
(b) ResNet18
(c) VIT
Figure 4: Test accuracy of (a) LeNet, (b) ResNet18 without batch normalization, and (c) ViT without layer normalization on the CIFAR-2 data with for various types of decision boundary. The شd x-axis point corre- sponds to dataset wherein the discriminant feature is the 5th eigenvalue of GS in descending order.
Gx. We compare training accuracy and the velocity of this dataset to another dataset where the data is sampled from another Gaussian distribution ((0.fli(G==)/) ( with flip(G==) being G, with its eigenvalues flipped (i.e., in the reverse order). The experimental results are observable in Figure 3
We observe very little change in the average geometry of the model when trained on the data with Bip(GG>) covariance compared to the data with G5 covariance. This larger shift in the geometry supports Conjecture [ showing that the geometry of the model will change very little in directions with a low correlation with G In Section 4.2 we will see that neural networks trained using gradient descent may fail to generalize when the normal to the decision boundary aligns with directions exhibiting geometric invariance, as defined by Conjecture 2
4.2 THE GEOMETRIC INVARIANCE HYPOTHESIS AND THE GENERALIZATION GAP
In this section, we try to investigate the relationship between GIH and generalization. In particular, we define three types of labeling for the CIFAR-10 dataset with varying complexity: linear as y= y=sgn((x u) bb). quadratic as s=sgm((s (x" bb). and sinusoidal as y=sgn(sindl u) +b). where corresponds to standard normalized inpuf and corresponds the discriminant feature. We train and evaluate a model on each labeling function for u set to the eigenvectors of Gy-SG in descending order, which can be seen in Figure 4 In order to make sure that observation is not caused by lack of separability for certain selections of u. we set the learning rate and number of epochs in a way to ensure all models will reach 100% train accuracy. Also, in order to simulate the effect of label noise, we add small amount of noise to the input when calculating the labels (but not to the input itself during training in
As we can observe in Figure 4 there is a clear correlation between the eigenvalues corresponding to u and the test accuracy of the model on all labeling functions, with datascts labeled based on a u with a smaller eigenvalue having larger generalization gap. These results further extend (Ortiz-Jiménes et al. 2020), which mainly focused on linear decision boundaries. Note that as we saw in the previous subsection, for datasets with a low correlation between the discriminant feature and G,, the model is incapable of finding the actual decision boundary. As result, it will instead rely on features in other directions to "memorize" the train sam- ples, resulting in a lack of generalization. Consequently, the GIH explains this phenomenon, which was first observed in Ortiz-Jimééee et al. 2020 Another interesting observation to make is that for more complex
*We added normalzatiio component the sinusnidal laheling in order control the complexity of the problem by controlling the variance of x, which is set to 1.0 on each channel separately. "We sample i..d. Gaussian noise scaled on each channel separately, setting the variance to 0.2 times the variance of the channel.
8

---

Published as a conference paper at ICLR 2025
100
100
120
160
Tort Accuracy
80
Uner
10
Uner
80
80
Unee
207
Eigervalue
(a) LeNet
(b) LeNet
(c) ResNet18
(d) VIT
Figure 5: The test accuracy of linear and non-linear components of (a) MLP, (b) LeNet, (c) ResNet18, and (d) VIT on the synthetic data with both linear and non-lineat components. The x-axis corresponds to the eigenvalue index desoending order.
decision boundaries, GIH will have "sharper" impact generalization, meaning that the decision bound- ary needs to reside in us with larger eigenvalues in order for the model to detect the decision boundary. This observation appears to be separate from GIH, and best explained by the SBH. However, as will see in the next subsection, the geometric inductive biases need to be considered when talking about SBH. In App. you can observe the relationship between GIH and generalization in the presence of "isotropic" data.
4.3 RETHINKING THE SIMPLICITY BIAS HYPOTHESIS
The simplicity bias hypothesis (SBH) argues that neural networks prefer to learn features with simpler relationships with the predictive variable. However, we observed in the previous subsection, depending on the direction of these features, the model may not be capable of learning them at all. So in order to reconcile the SBH with GIH, we design the following experiment involving synthetic data formulated as x= a. +w, with Gaussian noise w W~ (0.0%
~~.,0, (I- -I-up --- - -uuu usu )) for orthogonal ||a-u+8-uss|l2 with a and | being standard normal variables, and T1 # for uniformly distributed ye (-11).
U1 U3. We define zy as zyy Ma =Tyy 0-u.++Buu
Therefore, in this experiment we have a linear discriminant and non-linear discriminant feature in the form of uj and zy. We select and Ug from eigenvectors of G with large eigenvalues, while changing u₁ according to the eigenvectors of Gys. Figure S reports the test accuracy for the linear and linear components. In agreement with GIH, we can see that when u, has high correlation with Gy, the model always selects the linear component since the linear component has perfect accuracy while the non-linear component does not do better than random guess. However, as the correlation between and G decreases, we observe that the model becomes gradually more reliant on the non-linear component. Therefore, SBH holds true when both components But when the linear component has a low correlation with Gy, the model learns the non-linear component contrary what SBH would predict.
the same scale in terms of correlation with Gz.
WHAT CAN GEOMETRY TELL US ABOUT THE SAMPLES
In this section, we provide an application- based approach to the implications of the GIH. Specifically, we focus on the implications of the GIH on the feature space and the distribution the samples and try to see the distribution of features and data can be related to the generalization ability of the model through GIH.
S.l, Gign dualdam
Accuncy 194
80
83
$4;, Egn
54, Egg
40
0.75 Explained Taair Date Variance
0.25
0.50
0.25 Trair Data Variance
0.35
Explained
Explained Trai Data Variance
(a) LeNet
(b) ResNet18
(c) VIT
Figure 6: Test accuracy of (a) LeNet, (b) ResNet18 without batch normalization, and (c) ViT without layer normalization for the feature distribution experiment. We report results for CIFAR-2. with features climi- nated up to the index number of the generalized cigenvectors of G and S. For a fair comparison and using random orthogonal directions, we delete features until similar variability is removed from the data support.

---

Published as a conference paper at ICLR 2025
Feature importance. A natural question that arises from the geometric invariance hypothesis is: will eliminating features with the lowest possible correlation with G5 from the data have an impact on the performance? In order answer this question, we design the following experiment. We find the directions with the lowest correlation with G and the highest correlation with S by solving the generalized eigenvalue decomposition problem over G and S. Using the resulting eigenvecton we perform simple orthogonal projection on the data to eliminate the features residing in those directions. More concretely, for a sample x and the set of spanning vectors VGH (vT, V sr.. VD we set X5 GIH
x = Qi x, where Qu ID-
E V.V. We perform the orthogonalization on both the train and the test data, essentially eliminating the features from the data. As baseline method, we also sample a set of spanning vectors V Rand from a random Gaussian distribution orthogonalized using the Gram- tram--Shmiidt procedure. However, for fair comparison, for a given Q C we first find an index I/ where the explained variance by VV..v...... V......%). }
on S becomes no
more than the explained variance by {vT v2 ,.... vi }. and then set Rand = =ID Σ and evaluate the model on the orthogonalized dataset. The results from CIFAR-2 are reported in Figure
-= V.v. We then train
From the experiment results we observe that eliminating features based on their correlation with the geometry initialization causes a lower drop in performance than eliminating them randomly. Therefore, we can safely claim that it is beneficial to take GIH into account when performing dimensionality reduction on data.
Model
Percentage Removed Data (%) 90
30
50
70
LeNet-R LeNot-S
61.7+00.4
56.6+0.3
52.4-0.3
45.1
62.5+002
58.5+044
53.5=0.3
45.9+002
ResNei18-R
75.8106.
718:0.4
65.1=033
52.420.4
ResNet18-S
77.0=0.2
73.2#0.1
06.2:003
5441002
VIT-R VIESS
57.5.00.1
51.510.1
45.5 0.1
35.6....1
58.410.11
52.2:0.2
46.5:00.1
36.2:0.1
Sample importance. In this experiment, we try to understand whether there a relationship between the contribution of a datapoint to the performance of the model and the amount of variation it has in the invariant directions of the geometry. Specifically, given a datapoint x, we measure its correlation with the initial geometry
Table 1: The test accuracy of LeNet, ResNet18 without batch normalization, and ViT without layer normalization for the data distribution experiment. We report mean performance along with 68% confidence interval for CIFAR-1O, with datapoints eliminated randomly
(R) or based on their score value (S). The best performance is boldface.
G as: Score(x) = (x/1|x[|>) G5 (x/||x[[2). Then, we sort the datapoints according to their Score(-) value and eliminate the ones with the smallest score. We compare the results to a baseline wherein we randomly eliminate the datapoints. You can see the results of this experiment in Table [ For more detailed experiment please refer to Table in the appendix.
As evident by the experimental results, there is a clear relationship between the contribution of a datapoint to the performance and its Score(-) value. While the difference in performance between the random baseline and our geometry-based score is somewhat modest (at most about 2%), we note two factors are not considered in our Score(-): 1) whether the features present in datapoint with high correlation with the initial geometry are actually discriminant or not, and 2) whether by eliminating a datapoint, we lose rare feature that is not frequent in the other train samples. Considering these factors may improve the performance further, which is out of the scope of this paper.
CONCLUSION AND FUTURE WORK
In this paper, have provided theoretical and empirical results that investigate the changes in the input space geometry of a neural network during training. Based on these results, we proposed the Geometric Invariance Hypothesis, which argues that depending on the architecture, the changes in the input geometry of the model can be limited to small subspace of the input space. After empirically supporting this hypothesis, we provided several experimental pieces of evidence to show the impact of geometric invariance on the generalization of deep neural networks. Considering the practical impacts of GIH, as discussed in Section we expect our paper to provide valuable information to the study of inductive biases in neural networks, the role of data and its conditioning on generalization, and architecture design.
For future work, we point to several possible extensions of our results. Firstly, we note that the theoretical results provided can be applied to more complex architectures. Secondly, other optimization methods and their implications for GIH can be considered. Thirdly, we note the possible impact of GIH on other subfields of machine learning, such as AutoML and neural architecture search, the student-teacher framework, and foundation models as another important issue to consider.
We perform orthogonalization procedure on the resulting eigenvectors ensure they can span the input space.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We would like to thank Hadi Daneshmand, Guillermo Ortiz-Jiménez, and Felix Sarnthein for the helpful discussions and comments. We also thank Mostafa Dehghani for his contributions this work in its earlier phase. Antonio Orvieto acknowledges the financial support of the Hector Foundation.
REFERENCES
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché Buc, Emily B. Fox. and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pp. 7411-7422, 2019. URL htttt:::/pproceeeeee nigs neurips pp. cc/ wwtppppp//011 http://675555000000000111111111.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon I. Lacoste-Julien. A closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 233-242. PMLR, 2017. URL
mwr. www../.resssvvvv arpit17a.hhm.
Alexander B. Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel leamners The silent alignment effect. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
https://openrevierr http:/ppperiiiie.tteetrrreeeonee www.neeee//otttt.eemoomoo 1Nv
Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli, and Thomas Hofmann. Uniform convergence, adversarial spheres and a simple remedy. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 490-499, PMLR, 2021. URL http://proceedinng; mlr prwss/ wtt.3/99/aaaana...ee//e. httml
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. htLlhhttp//axiiii wwtp abs/ 110.14199.00 0473
David G. T. Barrett and Benoit Dherin. Implicit gradient regularization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: htttpss: net 3ir
Peter I.. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities:
Risk
bounds and structural results. J. Mach. Leam. Res., 3:463 482, 2002. URI. http:
html
Ronen Basri, David W. Jacobs. Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference
Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4763-4772 2019. URL https://proceediing hewprroceeednnggneniiip.p......p wwwrips.cc.m http:ipps..pppppe/1/////eeee -bwp:eabbbbbbbbbbbabbbaaaaaaae .btt.att..ttmm
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li. and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/ 806.01261, 2018. URL http://arxie wwwpaawwww......ba.aooaaaaa

---

Published as a conference paper at ICLR 2025
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern ma- chine learning and the bias variance trade-off. CoRR, abs/1812.11118, 2018. URL
http://arxiv org/ http/ab//i10 www.a////........ 11118
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kemnels. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Aiché-Buc. Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, www.seiiiisnnn..n............... nww.ips..p....... oC/ pappp/2019/1 9/aah// Abst html
pp. 12873- 2019. URL https: http:pppsp/ppppeeeeeecooo C
Christopher M. Bishop. Pattem recognition and machine learning, 5th Edition. Infor-
mation science and statistics. Springer, 2007.
ISBN 9780387310732.
URL
https
// wwW. wor worldcat
Ping-yeh Chiang, Renkun Ni, David Yu Miller, Arpit Bansal, Jonas Geiping, Micah Goldblum, and Tom Goldstein. Loss landscapes are all you need: Neural network generalization can be explained without the implicit bias of gradient descent. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kiguli, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openrevi ew. net met/pdffid-qu CRmRb7y9
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In 9th International Confenence on Learning Representattinss ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http:ttts///eeeeeeee//
net wew.//oooomdddiioio.aae wit.eYnIITbFyyTyy
lan Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep Leamiing. Adaptive computation and machine leamiin. MIT Press, 2016. ISBN 978-0-262-03561-3. URL
http://ww>/ www.deeeeeeeeebbbeo.oe org//
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CORR, abs/312..07522, 2023.
doi: 10.48500AAXXIV231200072 URL https://aol.org/10000 48550 (1t.5550////ii... www.i..21...0...... 00752
Charles Guille-Escuret, Hiroki Naganuma, Kilian Fatras, and Joannis Mitliagkas. No wrong turns:
The simple geometry of neural networks optimization paths. CoRR, abs/2306.11922, 2023. doi:
10.48550/ARXIV233066 1922. URLphttss://di.coo http://............/////o///////
40554485500/ii// 2306 11922
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016. Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016.
doi: 10.1109/VVR.2206.990
URL https::/ /tt.:/o...oooo org/10..0000..0 http:////...111/////221 wwt.:/119/..... 2016
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/ loos/1606.08415, 2016. URL htttp://aaaxi wwwpsiiiiorr/.ooo.oo abs abs/aaas.6... 08415
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput. 183 1997. doi: 10. 19962//6...999........ 8.1755
URL/tttps::/acc.org
neco. 1997.9
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.). Proceedings of the
32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448-456. JMLR.org, 2015. URL
http://proceedinng ml http://ceeeenaooo.....errr////// wwtppp/wwwwssessseeeeeppp
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio. Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò
Cesa- Basa--ianchi, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580 8589, 2018 / paper/201
ec/
5a lo http:///eo...aocae2222222222a330
html
12

---

Published as a conference paper at ICLR 2025
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin L. Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of increasing complexity. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPSS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3491-3501, 2019. URL pttttps://ppocceediigg neurips hwtpp/cccpppppeeepeeee / (tt.2/33ccaa99777777c833aaaa
Abstract html
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffeer. Gradient based learning applied to document recognition. Proc. IEEE, 86(1)) 222222244, 1998. doi: 10.1109/5.72679.. URL
https://boi.crr//0..110000 11tp///............ 726791
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning. Nut, 521175533333 444, 2015. doi: 10. 00.008/NATURE14599. URL http://:///ei..oo1111111111.1110
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Achh-BBuc, Buc, Emily B. Fox, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8580, 2019. URL https: http:://oooeedddeeeeeoo neurips cc/ httm
Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets. CoRR, abs/21 13492, 2021. [tt.:tss::://axx org/ wwwg//aaaao.. hta///bs/221111 13492
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In 6th International Conference on Learning Repre- sentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018 URL https:/:::////
hat_htttss//ppereeviiee http:///ppeeeeeeeeeeee net. for.mi1==r.z1B/AAA
Arvind V. Mahankali, Haochen Zhang, Kefan Dong, Margalit Glasgow, and Tengyu Ma. Beyond NTK with vanilla gradient descent: A mean-field analysis of neural networks with polynomial width, samples, and time. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023. NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL b3748ccdac32dd9 1a..00551337cb0. foa
wwwpscccppppp.....ccco caper waples/ppppr//eeeeeeeee /2023/222////// //aah/h
Abst cabbsttract--ooeeeeeeee Conference html
James Martens. New insights and perspectives on the natural gradient method. J. Mach. Learn. Res., 21:146:1-146:6, 2020. httt::///ill..crr//pppes//////// 67..mmml
Seyed-Mohsen Moosavi-Dezfood Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference Computer Vision and Pattern Recog- nition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 2574 2582. IEEE Computer Society, 2016. doi: 111109//VVR.2200.222
UtLhhtts::/ http::::///oi..rr////// 10. 1109/ 11t1////R.//001111 Cww.
Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc,
Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11611-11622, 2019. URL httpt: htt.t::ps://pproeednngeoo neur pps abw.:elbatbbbbbb-bbbabt.........
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5947 5956, 2017. URL wttpttttss/////poeee/iaaao
www.dninwnnn.........e.mu. cttppppppppcccpppeeeee.ep /aper/ 227.77/7aaaaooollaeee77773333333 Abst ract. html
13

---

Published as a conference paper at ICLR 2025
Maxwell L. Nye and Andrew Saxe. Are efficient deep representations learnall?? In 6th In- temational Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3. 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https: /ttp:///pperrvveeseeee nww//forurm wot.ooormmmm--.AAAAAFY.....
Guillermo Ortiz-Jiménez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Neural anisotropy directions. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL cff02a 44d64441 45w4aa44555dd5d75515aaaa52 Abst httl
neur wttpscccppppeee/.....// /
Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neu- ral networks actually say about generalization? In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan Advances in Neural Information Process- ing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, neur ps
December 6- 14, 2021, virtual, pp. 8998 9010, 2021. URL htttttpps///procceenenge
Guillermo Valle Pérez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019.
URL httppp:://ppe.reveeeeeeeee/////e
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli, The pitfalls of simplicity bias in neural networks. Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020, URL URUnttps::/prooeeeiing www.nnnu.iip..... wwwpscccpppppr.e 20000/hhasht
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Leurning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. http:///aa org/ abs 0atps/149......
Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL QrDolHyo
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, and Nathan Srebro. The implicit bias of gradient descent on separable data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver; BC, Canada, April 30- May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https::/openreviiee http::/ppe..aeeeee..eeeeooooooap
net fow.m?id=r1g7:1900
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does induc- tive bias influence scaling? ? In Houda Bouarnrr Juan Pino, and Kalika Bali (eds.), Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 12342-12364 Association for Computational l Linguistics, 2023a. doi: 10. 118633/11/222222 1Tt1://22222222NNGEEO -EN.DINGG-----.255 URL htt.::ttts://ttt www.:oo.org100. Ttw.111111116111 113886653vll1
findings-emn 825
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A sur- URL
vey. ACM Comput. Swrv, 1566:1009 2023b.
doi: 10. 145/333335.
https://8oi.ezg//12 2tt.://0//...11155331111111
Damien Teney, Armand Nicolicioiu, Valentin Hartmann, and Ehsan Abbasnejad. Neural redshift: Random networks are not random functions. CoRR, abs/2403.00221. 2024. doi: 1044550/ARXI22440302244
URL org/ /rg//11 48550/arXiv. v203..224.0... 02241
14

---

Published as a conference paper at ICLR 2025
Zhenfeng Tu, Santiago Aranguri, and Arthur Jacot. Mixed dynamics in linear networks: Unifying the lazy and active regimes. CoRR, abs/2405.17550, 2024. doi: 1048550/ARXIV.2455.75555 URL
https://aoi.orgg org/10. 48550/ 0tt550/aax...225511112
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https::/pprocee:// ngs neur i 1ps cctpapppp///111//// /2017 //aahh/ 3e rast .tt.latttttmtmm.....m.mm.....
Yuxiao Wen and Arthur Jacot. Which frequencies do cmns need? emergent bottleneck structure in feature learning. CoRR, abs/2402.08010, 2024. doi: 104850AARXIV24400000000 URL
08010
Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken,
Arber
Zela, Debadeepta Dey, and Frank Hutter.
Neural architecture search: Insights
from
1000 papers. CoRR, 22////311.87277 2023. doi: 1048500ARXXI.233000007777 48550/ wwt.://..vvv../... 2301..231.... 08727
URL
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep
learning requires rethinking generalization. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
net wwwt Foo.ommmm=sSyydBB.....
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):007-155, 2021. doi: 10. 10.1446/344676. URL htthts::: /T//apppe/////...o////0000011110
15

---

Published as a conference paper at ICLR 2025
Table The test accuracy of LeNet, ResNet18 without batch normalization, and VIT without layer normalization for the data distribution experiment. We report results for CIFAR-10, with datapoints eliminated randomly (R) or based on their score value (S). We report the mean performance along with a 68% confidence interval for each setting, The best performance is boldface.
Percentage
Data (%)
20
30
40
50
(X)
70
80
90)
LeNet-R
65.010.44
63.9510.3
61.710.1
58.910.4
56.6103
51810.3
524110.3
19.5:002
45.1100.3
LeNets
65.9+0.5
65.14066
62.5++0.2
60:9+0046
58.5=044
56.440.1
535+03
50.8=0.2
45.9+-0.2
ResNect8-R
78.6200.2
7682003
15.8:0066
78.9103
7184004
68810.1
65.120.3
60.3200
52/110.4
ResNett8-S
7966:022
784++03
77.02:02
75.9:00.
7322=04
70.0+:03
68:2::03
61.5+044
54.4102
600:0022
57.5
54.440.1
51.4001
45.5
35.610.1
VIT-S
62.9+0.1
60.6++0.1
68.4++0.11
65.4+0.11
62.2-0.2
49.7++0.1
46.5++0.1
42.1+0.00
36.2+0.1
0.66
Combu
0.64
0.9980
0.9975
0.62
0.9270
0.60
Hoddon Siz
Hidden Sic
(a) Theorem 3.1
MLP
(b) Theorem 3.2 CNN
Figure 7: The correlation between AF (a) for the MLP introduced in Theorem 3.1 and S, and (b) the CNN introduced in Theorem 3.2 and GG initialization. The experiment is performed on CIFAR-2. The x-axis corresoonds to the width of the model (i.e., hidden size in MLP and number of channels in CNN), while the y-axis shows the correlation based on Frobenius cosine similarity.
A APPENDIX
A.1 RELATED WORKS
Since the seminal work of (Zhang et al. 2017). there has been a lot of research aimed at reconciling the classical notion of bias-variande hypothesis and the uniform convergence theory with the success of deep neural networks (Neyshabur et al. 2017 Nagarajan & Kolter 2019 Belkin et al. 2018 Kalimeris et al., 2019. Bachmann et al. 2021 Chiang al. 2023). Regarding this issue, there are two schools of thoughts that either attribute this observation to an inductive bias introduced by the gradient-based optimization methods used in deep learning (Neyshabur et al. 2017 Barrett & Dherin 2021 Smith et al., 2021), or the "volume hypothesis" arguing for the inherently larger volume of "good" solutions that generalize well (Chiang et al. 2023). However, there is one certainty with regards to this issue: the existence of "inductive biases" in these networks that result in a large decrease in the hypothesis class of solutions provided by the model, causing generalization in practice.
In search of these inductive biases, some research has been focused the optimization of neural networks (Neyshabur et al. [017] Arora et al. 2019 Guille-Escuret et al. (023). which inevitably deals with the geometry of these models in the parameter space (Arpit et al.] 2017). On the other hand, the introduction of neural tangent kernels allowed a link to be established between the rich literature of kemee methods (Bartleu& & Mendelson 2002) and the asymptotic behavior of neural networks infinite width (Jacot et al. 2018). This also gave rise to some research concerned with inductive biases of deep models in the kernel space, which is also mainly focused on the geometry of parameter space (lacot al.) 2018 Bietti& & Mairal, 2019 Ortiz-Jiménez et al. 2021). However, there are also some research on the spectral inductive biases of neural networks, which provide interesting but intangible results for the type of frequencies used by these models in practice (Basri et al. 2019 Wen & Jacot 2024].
The limited research on the geometry of deep neural networks in the input space is our main motivation for this work. The closest line of research to paper are the simplicity bias hypothesis (SBH) (Pérez et al. 2019), and the neural amisotropy directions (Ortiz-Jiménez et al. 2020). The SBH (Arpit et al. 2017 Pérez et al. 2019 Kalimeris al. 2019) has become a comerstone of deep learning theory, with implication for generalization and robustness of deep neural networks (Kalimeris et al.) 2201:) (Shah et al.
2020). Specifically, SBH argues that neural networks learn functions of increasing complexity from the
16

---

Published as a conference paper at ICLR 2025
data. Therefore, in the presence of multiple solutions the problem with various complexities, deep neural networks opt for the simplest one. In this paper, we see that while SBH is still a valid hypothesis for the inductive biases of neural networks. the geometry of the model in the imput space- which is determined by its architecture also plays a role in the "preference" of deep neural networks for simpler functions. Meaning that, depending on the architecture, there are more complex solutions that may be learned by the model before simpler ones. This behavior is due to the geometry of the model remaining invariant in certain directions of the input space, We dub this behavior geometric invariance, and our hypothesis that introduces this behavior as geometric inductive bias of neural networks, the Geometric Invariance Hypothesis (GIH).
The neural anisotropy directions (NADs) as defined in (Ortiz-Jiménez al. 2020) touch on the concept of geometric invariance for training on linearly separable datasets. Specifically, y. "Orti-JJiénne et al. (020) observe that the case of linearly separable data specific direction u1 depending on the architecture, neural networks are incapable of generalizing when u has low correlation with the covariance of the gradient of the model w.r.t. input initialization. This observation can be seen as a specific version of GIH limited to the linearly separable data. However, unlike our paper which motivates GIH from the perspective of investigating the geometry a neural network, (Ortiz-Jiménez et al. 2020) explain their observations regarding GIH through the concept of diseriminative dipoles, which corresponds a pair of data points residing on the opposite sides a linear decision boundary. Therefore, given that our paper does not make assumptions about the task, GIH can be seen generalization of the concept of NADs.
A.2 SUM OF SQUARED ERROR LOSS AND ITS DYNAMICS
Following related works (Jacot et al. 2018 Lee et al. 2019 Arora et al. 2019) we use the sum of squared error (SSE) loss between the labels and the prediction of the model for the simplicity of analysis. However, as we see in the experiments, in practice our analysis holds for cross-entropy loss as well. The SSE loss can be written as:
1 2 [ -
(fo(x))-91)
(4)
#=1
In this case, we can write the dynamics of the parameter as:
0 (o(xx) mp)-
(5)
A.3 LOSS CURVATURE THE INPUT SPACE
An important source of information about the geometry of neural network in the input space is the Hessian of the loss w.r.t. the input, which contains information about the curvature. Let V be the Hessian operator. Using the chain rule on {(f(())y) we write:
de V=((<((),,)= of (fo()),) (
Pe ayz W
We can factorize the two components determining the curvature of the loss in the input space in two: 1) global model curvature ( (Hessian, for scalar fo (x)), and 2) local model curvature (gradient puter--roduct.. Assuming that we'ne e're in the wee-pparameterized setting with fa(-) belonging to family of neural networks, we can expect the magnitude of the first derivative of the loss function w.r.t the model to reach near 0 at some point during the training procedure. Therefore, starting at some point in the training, continue have:
82 (=((f(()),))= 2²² (x
(6)
So we can look at the gradient outer-product of the model as a low-rank approximation of the geometry of the model in the input space 8 In order to isolate the effect of architecture and other non-stochastic components
'Note that this is very similar to the Gauss -ewstn approximation of the loss Hessian w.r.t. the parameters, which is extensively used in optimization (Martens| (020). However, here using this approximation solely understand the geometry of the model in the input space.
17

---

Published as a conference paper at ICLR 2025
120
120
102
22
Iran Acconoo PM
Condectio
Iran Accurioy me
Condectin
Tran Acounsoy t0
Conducion
Tran Aww.i/y 10
Tish
Accuracy
72
0.3
700
166
166
Eporh
Spech
Spech
Spech
(a) MLP
(b) LeNet
(c) VGGIL
(d) ResNet18
Figure 8: The correlation between G (..(.. and S and G5SGr for the (a) MLP, (b) LeNet, (c) VGGI1 without batch normalization, and (d) ResNet18 without batch normalization on CIFAR-2 with random labeling.
G
the model (e.g. the optimization method, initialization method, etc.) that influence the "geometric induc- tive biases" of the model, we marginalize the effect of the parameters out. Assuming that the second- derivative of the loss function w.r.t. the model output has a small variation over 7 then we can write:
EE-Ti ((foo(
(z)
Therefore, by looking at the average geometry at point x and time through (x), we will have an increasingly accurate approximation of the loss curvature in the input at point and time / up to constant. Given probing distribution P. we can consider P.O--. V (x).))) as the average geometry, which we estimate by G = (x)].
Based this approximation of the geometry in the input space, we will formulate function informing us about the changes to get the changes in the loss curvature through the gradient flow model:
geometry as well. Let us start with the derivative (6) w.r.t time in order
dv dt (fo(x).
Se oes (f((()))))) (Vofo(x) Vxf(xx)
04 āوz (o (oll6/00))/VVRoooo(((VVXV.
0e مfo (
vx,o(())
Assuming that the hirdddeerivatvee of the loss function w.r.t the model output is close to zerc and assuming the independence of Te and the second derivative of the loss, we can get the expected value of the two sides w.r.t. the parameters to isolate the effect of inductive biases, which gives us:
d (fo(x).v)
E
o (x)...
dt
Therefore, based on the loss curvature evolution function and given probing distribution P, we define the dd de (fo(x). which we estimate as A' up to a constant.
average geometry evolution as Ex-PP,0011 p-T
A.4 INDEPENDENCE OF (x FROM LABELS
this section, we try to investigate the dependence between the average geometry evolution at initialization Ay and the train labels Vı. i.e., the relationship between the changes in the geometry at initialization and the task. Firstly, we provide the following proposition to show that at the first gradient descent step, this independence can be proved provided that the model has a single-layer perceptron for its classification layer and trained via SSE loss.
Proposition A.1. Let F be family of neural networks with a single layer perceptron initialized i..d from zero-mean Gaussian as the classification layer, trained via SSE, Then A is independent of train labels.
Proof. Let fe(x) be the model defined by the proposition:
n
fo(x) > wihh
"This can happen, instance, when we have the SSF loss where the second-derivative is constant. For instance, in the cross entropy loss the third derivative will be equal to the second derivative of the softmax function, which is bounded by 1/8, while is equal to the SSE loss.
18

---

Published as a conference paper at ICLR 2025
where hx(x) is the hidden representation in the penultimate layer of the model, (x) is its 1 element, and w corresponds to the weights of the classification layer. Therefore, 0 corresponds to the concatenation of w and o. The required derivatives are:
vx Σ V.V.
i=
050 (x) (x),
Owl
VVxf
(x) (x))((x),
Our
Vofo(x)= = wi V (x))
vx,of((x)= Σ we W o
So we can write 3x(x)=Axxx) T, where we have:
Ay(x)= Σ
Eg Luii hit (xx) 15 1/ Vx)
16g
+ Eg Wل Wis (wee, hi (xx) y/s (x Vx)
Note that the terms related to y/, in 7 contain an odd number of wys. Considering that w is i.i.d. and zero-mean Gaussian, its odd moments are equal to zero, which means these terms will eliminated. This completes our proof.
As we can observe from Proposition A. Â5F is provably independent of the task and only depends on inputs. In order to show that this behavior remains consistent at the initial stages of training, we formulate an experiment similar to Section 3 Specifically, we try to show that even if there's 's nothing to learn from the data, and the task is fully about the "memorization" of the data, Ay still follows the behavior outline in Conjecture ! So we relabel the CIFAR-2 dataset introduced in Section 3 using a uniform random distribution over {-1.1). We then report the correlation between G and GG along with the training accuracy for each epoch. The results are visualized in Figure [ We point out that these results clearly indicate the universality of our theoretical motivations, meaning that the behavior outlined the theorems in Section culminating in the introduction of Conjecture I are independent of the task.
A.5 THE STRUCTURE OF G5
In this section we try to investigate how the architecture affects Gy. First, we will present the following theorem for the case where F corresponds to an MLP or CNN with ReLU non-linearity and without pooling layers or skip-connections:
Theorem A.2. Let F be the family of MLPs or CNNs with ReLU non-linearity, without pooling layers or skip-connections. Then we have Gy = c. I for some constant c. For the proof. please refer to App. A.1.44
As we can observe in Theorem A.2 in the case where FF does not contain any pooling, self-attention, or skip-connections, the matrix G will correspond to the identity matrix, and therefore, is a full rank matrix.
However, this not the case in other types of architecture components. For instance, the following theorem will show that the introduction of an average pooling layer will introduce a structure to G
Theorem A.3. Let F be the family of CNNs with a linear convolution layer and single kernel, followed by a batch normalization layer and a global average pooling layer. Also, let Mj correspond to a binary matrix that simulates the convolutional operation for the jth filter. Assuming we use SSE loss, we have:
GF
31
52
19

---

Published as a conference paper at ICLR 2025
where J1.j2 is diagonal binary matrix corresponding to the overlap of the Ji and 1 patch of the data. For the proof. please refer to App. A.11.5
From Theorem A.3 we observe that G5 will have a diagonal form, with each diagonal element proportional to how many times the corresponding input element has been in the receptive field of the convolution filter. In the case of batch normalization, the structure of G can be potentially even more complex:
Corollary AA. Let F be the famity of CNNs with linear convolution layer and single kernel, followed by a batch normalization layer and a global average pooling layer. Also, let Mj correspond to a binary matrix that simulates the convolutional operation for the jth filter. Assuming that the train data is zero-mean and we SSE loss, then we have:
00T
G
ΣΣM,
M
R.
M
0j (0) (0)
where Varx,, ED (0 MJ mn). Therefore, the terms G corresponding to each pair of indices du is bounded as:
00T
Ji 2 Mj, En
M
J2
Xin N mai
3z)
Xthrx Via
where Smin:Nmee are the smallest and largest eigenvalues of the covariance of the jth patch of the data. For the proof. please refer to App. A.11.6
The Corollary A.4 iidicates that introducing normalization to the model will result in a re-weighting of the influence of the input elements based on their variance onto G5, atop the receptive field-related structure introduced by Theorem A..3 In our experience, in cases where the covariance of the training data has a structure, this will result in the normalization of G w.r.t. each patch of data which depends on the kernel size in CNNs and the patch size VIT.
Therefore, we can see that certain architecture components can introduce structure to Gr, which can reduce the ranking of Gy. This indicates significant narrowing of the geometric inductive biases of the model on the input space, considered by this paper, which as we observed in Figure 4 results in the model being unable to generalize in tasks wherein the decision boundary lies in directions with low correlation with G
A.6 VISUALIZATION OF THE AVERAGE GEOMETRY
In this section, provide visualization of the average geometry at initialization, i.e., Gy for various architectures. As we can observe in Figure an MLP has a full-rank average geometry at initialization, with the matrix G resembling the identity matrix. On the other hand, the eigenvalues of the other architecture have much wider range, clearly indicating a presence of structure in the average geometry.
A.7 GIH AND GENERALIZATION GAP WITH ISOTROPIC DATA
In this section, we try to isolate the effect of GIH over the generalization gap by making the data covariance as close to "isotropic" as possible. Specifically, we design the following experiment involving synthetic data formulated as follows:
x= (I.o((((-uumm) T)
(
where have (0,11, and we consider be the discriminant feature, similar to Section 4.2 Note
that in this setting, for R we can roughly say x~N((0,0r which eliminates the effect of data over the
changes in geometry. Similar to the experiment Section 4.2 we assess the generalization ability of the
model for varying complexity of tasks: linear y= [ u) +b), quadratic as y ==sgn(( (xx uu + b)) and sinusoidal as y ssinn(sin(X b). Note that we can view the ratio e/o as a measure of the margin between the two classes, which controls the difficulty of the problem. In Figure 10] you observe the test accuracy of the models for these tasks, with u selected from the eigenvectors of Gy in descending order of the eigenvalues.
As we can observe, in the absence of a structure in the data covariance, the MLP model will be completely devoid of any geometric inductive bias, achieving generalization in all directions with similar equality.
20

---

Published as a conference paper at ICLR 2025
107
187
20-4
16
20
33
2/6 Spenssssidds
(a) MLP
(h) LeNet
(c) ResNet18
(d) VIT
Figure 9: The visualization of the average geometry at initialization (GG) for (a) MLP, (b) LeNet. (c)
ResNet18, and (d) ViT the upper row, along with their eigenvalues in the lower row. The input space corresponds to single-channel 32 x 32 image. Note that the visualized NAD components (Ortiz-iiéénez et al. 2020) corresponds to the cigenvectors of this matrix. Considering Conjectures | and2 we interpret this matrix how the variation in the data impacts the geometry during training. For instance, MLP we observe that the average geometry is a scaled identity matrix. As a result, the data variation impacts the geometry of each input element separately. On the other hand, in LeNet the average geometry is a matrix with non-zero elements around the diagonal. As a result, we expect the data variation impact the geometry of input elements in proximity each other. Similarly, in ResNet and ViT we observe a periodic structure in the average geometry. As a result, we expect the data variation impact the geometry of input elements a cyclic manner.
Tattouuucy (h)
Deaterr
Ab-4
Aoluocu thd
Siraciile
12
xte-
+12-0
1b-
Unerr
Sinanica
122-
-22-4
42-4
42-4
Ctanrdla trdl.
Spenddas Inda
(a) MLP
LeNet
ResNer18
(d) VIT
Figure 10: The test accuracy for the (a) MLP, (b) LeNet, (c) ResNet18, and (d) VIT on the synthetic data. We set 1 for all experiments and to 0.9 and 2.0 for the linear and quadratic decision boundaries. respectively. For the sinusoidal decision boundary, we set to 2.1 for MLP and LeNet, and 5.0 for ResNet18 and VIT.
On the other hand, in the case of the models with structured average geometry, the performance the model in all tasks is correlated with the eigenvalue of the corresponding discriminant feature. These results provide further support for the GIH, while also confirming that the observation in Section 4.2 is due to the existence of a structure in both the data covariance and the initial geometry.
A.8 THR EFFECT OF DATASIZE ON GIH
GIH is implicitly based on the assumption that the average geometry evolution still follows the pattern introduced in Conjecture [ in the latter stages of training. Therefore, natural question that may arise from this assumption and experimental setting is that: Can GIH be solely attributed to the over vee-paaranterizeed regime? order to answer this question definitely, we designed several experiments to investigate the effect of datasize GIH fully.
21

---

Published as a conference paper at ICLR 2025
10a
10-1
Linear Quadratic
lest Accuracy (%)
100
Quadratic
Test Accuracy (%)
90
10
80
Eigenvalue
Sinusoidal
-10-
Eigenvalus
80
Sinusoidal
Eigenvalue
10
Eigenvalue
10-7
10
60
M
-10-20
60
10
50
10-1
3000
250
500
750
1000
Elgenvalue Index
Eigenvalue Index
(a) CIFAR-10
(b) Gaussian
Figure 11: The test accuracy LeNet for the (a) CIFAR-10 and the (b) synthetic Gaussian datasets introduced in Section 4.2 and App. A.7, respectively. We used the exact same training setting but with the same amount of training samples as model parameters (50k samples).
100
Train Data Size
100
Train Data Size
Train Accuracy (%%
10000
Train Accuracy (%)
10000
20000
20000
40000
40000
60000
60000
80000
75
80000
100000
50
10
20
# Epoch
# Epoch
(a) Train Accuracy
Gyr covariance
(b) Train Accuracy flip G covariance
1.0
1.0
Train Data Size
Train Data Size
10000
10000
20000
20000
40000
40000
60000
60000
G 0.5
80000
0.5
80000
100000
100000
0.0
10
20 # Epoch
Epoch
(c) Velocity
G5 covariance
(d) Velocity flipGG;) covariance
The train accuracy and velocity G ) of the ResNer18 without batch normalization synthetic datasets: G covariance N (0, (0,G>/||/>>ll>)) and flip (Gx) covariance x ~ N (O,flip ( ( with random labels. We perform the experiments for datasizes in (10000,20000,0000)
Figure 12:
100000)
In the first step, we start by looking at the accuracy and the effects GIH on generalization we observed in Section 4.2 and App. A.7 on the verge of under--aramettrizatton in LeNet. You can see the results of this experiment in Figure we had to slightly decrease the size of LeNet by making the fully connected layers slightly smaller. Despite the larger dataset and the fact that the model is no longer in the wwrrppaameterrized regime, we can observe that the performance of the model on the test sample still exactly as the GIH would predict. Specifically, we can still observe a "preference" to learn the features more aligned with the average geometry during training. This observation rules out praamttrtettiiiioonnnn
Note that in order make the under-pal padderrpramtterieeed regime possible,
condition for GIH.
We can also look at the train accuracy and how is affected by datasize. order to do so, we focus on ResNet18 8 and use the experimental settings of Section for Figure 3 So will look the train accuracy and the velocity of the average geometry for datasizes in {10000,20000, 40000000000000000 for data sampled from ((0 and (0.flipp (irn)
100000) You see the experiment
22

---

Published as a conference paper at ICLR 2025
100
Train Data Size
100
Train Data Size
Train Accuracy (%)
10000
Train Accuracy (%)
10000
20000
20000
40000
40000
60000
60000
80000
75
100000
100000
50
50
10
20
10
20
# Epoch
# Epoch
(a) Train Accuracy Gy covariance
(b) Train Accuracy Gl covariance
1.0
1.0
Train
Train
10000
10000
40000
40000
60000
50000
6 0.5
G 0.5
80000
100000
100000
0.0 10
0.0
20
10
20
# Epoch
# Epoch
(c) Velocity
GF covariance
(d) Velocity flip(G=) covariance
Figure 13: The train accuracy and velocity G .) of the ResNer18 without batch normalization on two synthetic datasets: G5 covariance x N' (0))))||>>l))) and flip (G=) covariance x N (0.flip(G)-)// [[ip [L.iip(G=>)|liipGG>>>]) (1))) with random labels. We perform the experiments for datasizes in {10000, 40000, 40000,60000,0000 800000, 10000000100000); Furthermore, we increase the batchsize for each datasize so that the number of mini-batches would remain the same.
results in Figure At first glance, the results may seem to indicate that the effects of GIH are vanishing on the train accuracy as the number of datapoints increases. However, we were able to refute this point. Specifically, based on these results we hypothesize that the reason the model able to generalize on the flip( G)) covariance data is that the number of iterations are also increasing, resulting in the parameter ending up very far away from the initialization point. Therefore, will effectively eliminate the geometric inductive biases of the model caused by initialization, resulting in the model behaving according the geometric inductive biases of some G for some very large t
In order to support our claim, designed another similar experiment this time with dynamic batchsize that increases with the datasize to make sure the number of iterations of SGD on the parameters remains constant. This will ensure that all train data sizes. You can see the experiment results in Figure 13 As evident by the results, in this exper- iment, the model is reverting to adhering the behavior predicted by GIH, which supports our hypothesis for the reason behind the observation in Figure [2 In App. A.9 we will see that while the distance from ini- tialization seems be affecting the predictions GIH for train accuracy, is not the case for test accuracy.
moving similar distance in the parameter space during training for
A.9 THE EFFECT OF LEARNING RATE ON GIH
Given that GIH is concerned with the training dynamics of neural networks, an important relationship explore is how learning rate impacts GIH. Specifically, this section, we are trying understand whether the learning rate amplifies or reduces the effect of GIHH, similar to the results of Appendix A.8 In order to answer this question, we designed the following experiment for the ResNet18 8 model. We follow the experimental settings of App. A.7 for the experiments presented in Figure I0 but for learning rate in (1.0,0.1,0.0 1.001,0.01,00001, You can see the experiment results in Figure 14 Note that in all these experiments, the model converges on the train data and reaches 100% train accuracy.
Note that are abusing our notation make the point that the effects of GIH are still present, hut with some average geometry computed different point during training than initialization
23

---

Published as a conference paper at ICLR 2025
tramligfar
100
20-1
:c-1
22-1
g
3.002
253 507
750
2000
253 500 750
2000
230
300
750
2000
(a) Linear
(b) Quadratic
(c) Sinusoidal
Figure 14: The test accuracy of ResNet18 for the (a) Linear, (b) Quadratic, and (c) Sinusoidal decision boundaries on the synthetic data. We follow the same experimental setting as in App. A.7 except for the learning rate, which is chosen from the following set: [1.0000110001.00000.}.
0
-1
-2
-2
Figure 15: visualization of the eigenvectors for the model introduced in App. A.10 A.10 have non-zero eigenvalues, while the red vector has eigenvalue.
The blue vectors
As we can observe, a very small learning rate has a detrimental effect on the performance the model, while larger learning rates generally generalize better. However, in all these cases we can observe that the model is following the predictions of GIH, and we can observe "preference" to learn the features more aligned with the average geometry during training. Furthermore, these observations also support our hypothesis for the results of the experiment performed in App. App.[A.8, as larger learning rates, and thus longer optimization paths, do not cause the effects of GIH over the test data vanish.
A. AN EXAMPLE
For the sake of clarity, let us consider an example of a 3-dimensional input space, point on which we denote as:
X1
42
24

---

Published as a conference paper at ICLR 2025
88
For our model, us assume we have a convolution layer with single kernel in the form of followed by average pooling layer of size 2. Then, we can write the model in the vector form
[/]
T1
0
82
0
Tx(x)=11/2
1/2
0
01
02
T22
Lig.
ar
[0
+0
Oz]
22
2
[33
So the gradient of the model w.r.t the input will be:
01
Vxfo(x)= = 2
01 +
02
02
So the outer product of the gradient with itself will be:
0f 89 0 0202
09+0,02
0,02
Vxl(( =
(0)+
0,0, 0
0102+03
Now let us assume the parameters are initialized ii.d. 0, 82~N (0.og). Then we can write:
zero-mean Gaussian random variables, i.c.
[ 1
0
G
0
2
2
[
1
1
Therefore, the average geometry at initialization a rank-2 matrix with the following eigenvalues and cigenvectors:
Eigenvector: V1
eigenvalue: Å 33
Eigenvector: V2
eigenvalue: A2=
1
Eigenvector: V3
eigenvalue: Ag =0
You can see a visualization of these eigenvectors in Figure 15 According to GIH the geometry of the model the input space can only change in the direction the blue vectors, which corresponds a 2-dimensional plane. Therefore, the model is not capable of forming decision boundaries in the direction of the red vectors, thus having geometric inductive bias towards the features residing on the plane defined by the blue vectors.
A.11 PROOF OF THEOREMS
A.11.1 LEMMAS
We will start the proofs by first proving four lemmas that we will be using in the rest of this section.
Lemma A.5. Let 8 ~ (O.II be a standard Gaussian variable, and b E RD be two unit vectors.
Then, as becomes larger (D - 20), and assuming a, are non-sparse (i.e., their magnitude concentrated on few elements), we have:
Es 1 T a 4T 52005
2J Ea a Ip,
which becomes accurate the rale of VD.
25

---

Published as a conference paper at ICLR 2025
Proof. We will start by partitioning the variability into three factors: 3=a-a+B-b_+t where bj is the unit vector in the direction of the orthogonal projection of b on a, and 61 (0,Ib aa -bub]) Then we can write:
Es [1 000000000
= (E las0 b. >0a
aa E laza + + bbbb
E Las0 t 200
E [ 1as0-as bu bi 2015
( a +ab)
+ E +y Lp.
Now, will upper-bound the coefficients of the first three terms. For the coefficient of the first term, we can write:
El
1 1a
E 1
Las0
(9)
1
1
+1)))
2
1020
erf
V2
0.C
+1
E
laso
erf
V2 a.e
(10)
1
1
Lasoerf
a.CC
a
E
lasoerf
a.
(11)
2
V2
V2"
1
1 1osaaexp
- V2
a 2
)
(12)
π
where c= b
comes from the definition of indicator function and the error function erf(-).
comes from a being standard Gaussian variable which gives us E Lasoa²] =[[10>0) = _.2 and
12
comes from the following:
a²
E
1
nesf
a.c
2
erf
a.0 /2
e'exp
da
(13)
2
2
too
a.C
a²
erf
dexp
(14)
V
V2
1
0.C
2
a².0²
a²
erf
+
nexp
exp
da
(15)
V2m
V2
VT
V2
2
a-c
2
a²
6222
E
Lasperf
axxoxexp
(16)
V2
22T
V2
with 14
resulting from aexp
2
do= dexp
2
and
15 resulting from integration by parts.
Similarly, for the coefficient of the second term write:
E
[1a20
1 arot 1aree 6 - E[11>00wactpp
(17)
1
00.02
E
1
laxpa-c-exp
+E -ET (18)
2
2
1
a².c2
E
cxxp
(19)
22T
2
where 18
comes from:
1
+00
32
Eg
Pexp
d3
(20)
V2T
2
82
dexp
(21)
22
are
.P.02
+00
82
-a-c-epp
exp
d3
(22)
22T
2
82.02
+ +Eg Laactt220)
(23)
2T
26

---

Published as a conference paper at ICLR 2025
And for the third term, have:
1
1xppa-exp (-
[a2.c2
E
V2T
which comes from:
1
+00
82
B-exp
dg
V2
2
82
dexp
22T
are
1
22.02
exp
2
For 12
we have:
- 1 2.2
1
1ox0xexp
Insoal
V2
π
TV 2T
T
which comes from exp( -ap S1. For (19 note that the maximum value of c-exp( ( (-2222)
happens
at 22 which gives us:
a².c²
]
1
lasuexp (-1)
E
lasua-c-exp
2m
2
22
exp(- 2
2/2
And for A.111 have:
1
a2.02
E
Lap
E 1ax00]
22TT
2
2m
2m
2,22 2
which also comes from exp
1.
So that the coefficients of the first three terms are bounded. Furthermore, we can assume a c' bb for some constant E (+1,-1) since otherwise the result would be trivial:
Es [1 a0olo
| Es 85000007
Es [00
2
2 In
for c'= and:
Es [1 55 650 =0,
otherwise. So as D-+oc given that a, are non-sparse and unit vectors, their magnitude will be stretched over the elements, resulting in each element of a and approaching 0 at the rate of vD. Therefore, the last term will be the dominant term, which completes our proof.
O
Lemma A.6. Let 2N be standard Gaussian variable, and a D be two vectors. Then, assum- ing a (0.Ip), the term E. sTaDo bbo)) will be a uniform random variable war.. a, and we have:
E5 [ a2 1sbbool=
Eg E6 Be
(24)
2'
27

---

Published as a conference paper at ICLR 2025
Proof. Note that since a is standard Gaussian variable, its angle with given vector b is uniformly distributed variable. Furthermore, Eg [5[TTTa>-)) T corresponds to the overlap of the two halfspaces defined by 8 a > and 8 b > 0, which equal the ratio of the angle between a and b in Radian, divided by 2T.. As result, it is uniform variable ~ U[0,1] (i.e., uniformly distributed in [0,1] range). So its expected value and second moment are, respectively, 2 and 3--
Lemma A.7. Let 6~ (0.I() be a standard Gaussian random variable of size D. Then we have:
88 |
Es
D ID.
(25)
Proof. First, note that due to symmetry, the diagonal elements and the off-diagonal elements are equal in 25 Therefore, we can get the diagonal elements by evaluating the trace:
88
Tr(55T)
Tr
Es
EEs
101
|
[1][]]
=1.
This gives us diagonal elements equal to L/D. On the other hand, note that we can write an off-diagonal element in (55 as:
8,0
Eg Es,6,
which is equal to 0 due to the fact that d is zero-mean and symmetric in distribution. This completes the proof.
A.11.2 PROOF FOR THEOREM 3.1
Now that we have proved the two lemmas, we can prove Theorem 3.1
Proof. Let fo(x) be the model described in the theorem:
(x(x) Eui wi LoT x x
where wi and di are i...d random Gaussian variables with zero mean and variance of and o respectively. As such, we have TT = [ Ii i.e., the concatenation of wis and Φis. The derivatives of fo(x) can be written as:
Vxfo(x)= = Σ
We 07 xxx 1=1
0.0 (x) = 8o Tx20007 x,
Ow
0fe (x) = - oJ x>0X,
0o:
0V dVxfo
(x) = 1 x
Ow
8V fxfo(xx
(x) =w 1 1o I
06i
Following Proposition A.1
we can write
A5=A5 r AJ from B with:
m
A,
Exx,
Σ
of x TTx, >0 (10, xn) di + w wāxx))
(26)
(
wh oo x 200 X -0)
Σ
wil الالمرن [o fo0o
/=1
28

---

Published as a conference paper at ICLR 2025
Note that since is vero-mean, the terms related to y eliminated from 26 Similarly, the terms involving j l are also eliminated. So we have:
m
A
B0,x for x of x. of x xns0 (0, ))]()) x ]vo]]
(27)
+ Box (ooT xox x of x oJ x, (0)xx,1 X xx
For the first term in (27. have terms with i= which are equal to:
Ex,x 1 x>0 of x, x. (0) 2 "2eo?"
= E.p
Rxx
1 o x
1 of x (of xx) " a20!
1
Eg
1
Lxxx (o) xx) 610.
2
4 Eg
1
(o) x "200!
x x + x x
And in the case when i we have n² n terms each of which are equal to:
Ex
ToTxool Txx x 20000
x [1o] x xu 200,0
(28)
=
Ex
Egg
xs o7 x
BBg LoTxso x X
(29)
X
(30)
3
where 29) comes from Lemma A.5 and (30) comes from Lemma A.6 For the second term i j=j j have terms in the following form:
when
Ex 1oTxx>u xxx (o x xx
= x,, x, Eo 1 x>o of x,, 2.o
(31)
03 x, X, Excon 1 LoTx x,
(32)
Φ x x
(33)
where 32 follows from Lemma A.5 and (33) follows from Lemma A.6 And finally, the second term (7
n² n terms with i which are equal to:
Ex Es,
Lo xx oo oTxx =0 Es [oT xxxx=0 o x 0o (0, x, x
(34)
o Ex Eess 1o x 1 of x
Esg ['oj x - oj x X
(35)
x,x,
(36)
where 35
comes from Lemmal
A.5 and 36 comes from Lemma A.6
Finally, given the results so far, we can write:
no
00
no?
2
good
noto2
A 2.
+(n -n)
Φ
(n²
x
T -2-
| 11 --2-0
E
2
3
4
VD
=-2n
00
+ (n (n-1) 1)
0 202
w
+
(n-
1)
2004
Σ
m
2no
go
m
x 11
1-1--1
0
2
E
2
3
/D
1
=-0(13)---0 0
& (())
VD
where & is the error corresponding our approximation from Lemma A.5 T
and Eg Σm 1
( & (())))
(AJ)))
So we have
lim
A
lim
(on+))++0
||||||
=1
29

---

Published as a conference paper at ICLR 2025
A.11.3 PROOF FOR THEOREM 3.2
Proof. Let fo(x) be the model described in the theorem:
fx(x)
Wi E EoTM]x
where w; and d, are i.idd random Gaussian variables with zero mean and variances o and o respectively, and M, is binary matrix mimicking the convolution operation. Note that we set j M,, M which is binary diagonal matrix with elements in the receptive field of both ji and j2 set to and otherwise. Furthermore, we have 0 = (and Ii=1 ie., the concatenation of wis and d,s. The derivatives of (x) can be written as:
k
1
Vxfo(x)=
Wi
j=1
0f((x)
(x)
1
Σ
M, x,
Ous
7:
y=1
vV
1
(x)
M 1
- 7: 1=1
Ow,
Ofe
1
(x)
wi
MJ x,
0s.
k
0V
1
(x)
M,.
0Φ:
k
j=1
We first evaluate G>(x)):
))
Σ Mysdrs ))
n
k
b
Gy(x)= = 12
Eg
Σ Wis
Σ My oi
Σ Wis
1==1
Eg M 0:07 M.
k.a
-11=1//1
no
03
k
A
11=1/=1
Now we will try to show that ( (x)==Axe we can write A as:
Gx()))G)(x)) Following Proposition A.1
1 K
A=(x)
B [0 MJ. xp) (0 M. x, M di M
(37)
to Eo
The first term in 37 has terms with i = (substitute i1.02 with i), which can be written as:
nM, E.. [(o MJ, o.0 x,,x,. MMM0o. T =nM, Ea. [or0, M, x,J, M, o.0 !
So using Stein's 's lemma, we have:
nM, jgg Ea, [dr M.. x M (ho = Mj (Bo, [oT M,, M Mb + +MJ. X Mj, Bo, [0a0)]] J]-
30

---

Published as a conference paper at ICLR 2025
The first term is equal to Tr u. M2,). and the second term is equal to ojM x2xx M So the n. terms in (37) with 61 = i2 are equivalent to:
I X laci tno Tr (M,,,, x,xx, M,, Livvs
The first term in 37 also has n² n terms with i1 + i2, which can be written as:
(n²-n) M, Eug,, [di, P M,, Be., Pil M, - (n² -n) jast 1xxTIns
The second tern in 37 has n² which can be written as:
n Ji x Li, 10 0 M =0 1int X jssis.
So we can write:
4
k
n -2 0 TM, SM,,, Iads
AF -
K
11
So we have:
-= O - -on))
where & corresponds to the emor term
>n==1 Epp=1
D ا Tr (MJ,
Ljasts
Tr(Ar(x) (x))),()()
Therefore,
we
can write:
lim n
|Ar(x)) Gy (x)ll
Th (( +00))))))))))))))))))))))))))))) (x(x)) (=00)) (*(x))())))
lim
which completes our proof.
A.11.4 PROOF FOR THROREM A.2
Proof. Let (x) be an MLP or CNN with l layers, no pooling or normalization layers, and a scalar output:
f0(x) wi for (x) tx)).
where @ corresponds to the union of wis and dis, and (.) and for (.) for if share the same parameters up the last layer. The derivative of fx() w.r.t is:
V Vxf((x)
wi (x)20 Vxfa (x).
which means we have:
Es V=
Eg WHwe 1 ( 15,
(x)20 /
v (xlo))(()) (x) V (x))
1==1==1
1 | (x)1(x)000 (x)20
(x)
1=1
So we need prove that:
Es,
1 (0-1 (x =c..
First note that since the weights of the last layer (i.e., the classification layer) are mero-mean and Gaussian, the model w.r.t. these weights symmetric at 0, which means we have:
1 Eg. 1 1 (x)20 VxJa Vx) of - = B..
Vxf pf Vx) (xx)
So in order to prove this theorem, we need to use induction. For the case with we can write the model as:
fx() =
2 wi oJ x x.
31

---

Published as a conference paper at ICLR 2025
where both wis and dys are iid. Gaussian variables with variances equal to o and o respectively. This gives us the following gradient w.r.t the input:
vaje(x)=
Σ w/ xxxxx0000
So we have:
1
Eg Vx
1
Eg Lo xsoRid
(38)
1
(39)
2 1=1
no
I,
(40)
2
where (39 follows from 648 being zero-mean i.i.d. Gaussian variables. The rest of the proof follows from induction.
A.11.5 PROOF FOR THEOREM A.3
Proof. Let fo(x) be the model described:
k + e O x, j=1
fo(x)
1
where is a ii.d. random Gaussian variable with zero mean and variance og. and M, is binary matrix mimicking the convolution operation. The derivative fo(x) wxt. x is:
1 1 1: 3=1 00)
vxfo(x)=
~ M,0.
So can write:
k
G
M Eo M.
j1=1/==1
M,
which completes the proof.
A.11.6 PROOF FOR COROLLARY A.4
Proof. Let fo(x) be the model described:
& Σ 0 MM]x x, j=1
fo(x)
(0)
where is a i.i.d. random Gaussian variable with zero mean and variance oz. and M, is binary matrix mimicking the convolution operation. The derivative of sa(x) wan.. x is:
1 -ー
Vxfe(x)
32

---

Published as a conference paper at ICLR 2025
So can write:
00
G h
Mj,eo Eo
on (0)0z22)))) M
51=1j==1
Note that we can write (f) for some j as:
03(0)
A MJ SM,0
D
1 BTs 5,0.
D
which can be bounded as:
minn 101155 D D- S Vinau |012
This gives us the following bounds:
00
T0
000
Ee
Eg
Eg
ja
100133
J (0) (0)
31
|0013
min mai
And from Lemma A.7
have:
00
E
I.
J1 min min
JJ (0)
Amaaz Amar
which completes our proof.
A.12
EXPERIMENTAL SETTING
A.12.1
HYPERPARAMETERS
All of our experiments are performed on the following models:
MLP: A multi-layer perception with hidden layers of size 100 and ReLU non-linenan -oneeiinarity.
LeNet: We use the model introduced in (LeCun et al. 1998).
ResNet18: We use the model introduced in (He et al.
2016).
VGGII: We use the model introduced (Simonyan & Zisserman 2015).
VIT. We use the architecture introduced in (Dosovitskiy by (Lee et al. 2021 for small-scale data.
2021 with the settings provided
Similar to Jittiz-liménez et al. 2021 we observed much better and accurate value for G when using GELU non-linearity (Hendrycks & Gimpel 2016) compared to ReLU, which we also attribute numerical problems caused by ReLU. In this case, we observed lower variance when computing G with probing function with a smaller standard deviation, which attribute the saturation of the GELU function for standard Gaussian input. Specifically, in our experiments we set P P= (0,00, with dp working best in range of 10-3 and 10-5 in terms of correlation between generalization gap and eigenvalues of Gy in the experiment presented in App.Ar A.7 However, similar (Ortiz-Jiménez et al.) (202), we observe that first-order approximation of G in the form of G0=(0) is also relatively good approximation.
The experiments provided Figure are performed using gradient-descent with the learning rate equal to 0.1 and momentum set to 0.9, and for 100 epochs. The experiments provided in Figure are performed using the Adam variant of gradient descent with default parameters due to slower convergence and for 100 epochs. The weight decay in all these experiments is set to 0.
For the experiments in Figure and Figure 5 we use the experiment settings of (Ortiz-iméénez et al. 2020) for the synthetic data, except for the experiments in Figure @ performed on sinusoidal decision boundary, for which we train all models with 50 epochs.
The experiments in Figure 3 are performed using stochastic gradient-descent with a batch size of 128 and learning rate 0.1. We train the model for 100 epochs for the experiment involving LeNet and 50 epochs for the experiments involving ResNet18 8 and VIT. We set the momentum 0.9 and the weight decay 0.
33

---

Published as a conference paper at ICLR 2025
The experiments in Figure are performed using stochastic paadient-descem with batch size of 128, learning rate of 0.1, and momentum set to 0.9 and 50 epochs. The weight decay all these experiments is set to 0. The experiments in Table [ and Table 2 are performed using stochastic gradient- with a batch size of 128, learning rate of 0.2, and momentum set to 0.9 and 50 epochs. The weight decay in all these experiments is set to 5 10-4.
In all experiments, we estimate F on 10000 models and G'y for t> on 25 models.
A.12.2 STATISTICAL INFORMATION
For the experiments in Figure 2 Figure and Figure we only report the average of test/train accuracy since mainly aim to show the trend of the changes in these values. However, the experiments in Figure[4] Figure Figure 6 Table | and Table all performed 5 times. We report the average along with the 68% confidence interval for all these experiments.
A.1 HARDWARE AND FRAMEWORK
All experiments are performed on single Nvidia GTX 1080 Tr, and implemented on Python using PyTorch 2.1.1. We base our implementation on the code provided by (Ortiz-Jiménez et 2020
A.13 LIMITATIONS
We consider two main limitations our work: theoretical limitations and experimental limitations. In the case of the theoretical limitation, we note that our theoretical results are mainly focused on simple models (i.e., models with one or two hidden layers, and ReLU or linear activation functions) and based on approximation (i.e., correct for large input space size and model width). The reason for this approach is that in theoretical results, we're mainly concerned with insight as opposed completeness. Consequently, we omit providing proofs with minimum assumptions and more realistic models in order to keep the message of the paper focused and the content concise.
In the case of experimental limitation, we note that our experiments are mainly concerned with over-parameterrized settings and small-scale image classification datasets (i.e., models with a few million parameters and datasets with about 5000 samples for each class). This decision is mainly due our lack of computational resources. As a result, we do not make any claims about more complex settings involving. for instance, foundation models or large-scale scale datasets. Furthermore, we do not extend results to other aspects of machine learning, such as regression or reinforcement learning.
34


Reviews:
[{"review_id": "TG1i4iKnFY", "reviewer": "Reviewer_wQbe", "summary": "The paper studies how the geometry of neural network predictors evolves in the input space during training. It proposes the Geometric Invariance Hypothesis (GIH), which posits that this evolution occurs within a constrained subspace of the input space determined by the network architecture. The GIH is supported through experiments with MLPs, CNNs, and ViTs on a subset of CIFAR with binarized labels. The paper also studies the link between the GIH and generalization.", "questions": "9. From Figure 1, it seems that simple (and standard) early stopping would solve the generalization gap problem and actually result in a lower test loss for dataset $D_B$ compared to dataset $D_A$. Can you comment on this? How general is this observation? \n\n10. What’s the underlying distribution over which you are taking the expectation at L227? It should be made explicit.\n\n11. When you write “momentums” (L244, L246) do you mean “moments” instead?\n\n12. Is the fact that $\\Delta_F$ becomes label-independent purely due to the fact that you assume $\\mathbb{E}_\\theta [\\theta]=0$ at initialization?\n\n13. Could the phenomena observed in the paper, particularly when testing conjecture 2 about the GIH, be due to the fact that networks in all experiments are strongly overparameterized (even with CIFAR, the paper considers only a small fraction of the full dataset) and training in the “lazy” or NTK regime where weights stay close to their initial values? Did the authors measure the weight evolution throughout training? Would you expect the same results when training with more data, e.g., the full CIFAR10 dataset, and/or with tasks requiring learning latent features of the data that might be less dependent on the exact geometry and statistics in input space, e.g., more complex image datasets such as ImageNet?\n\n14. Could the authors comment on Conjecture 2 in the case of linear regression? Intuitively, it seems it should hold in that simpler setting.\n\n15. What is the motivation behind adding label noise (L406)? Would the same results or trends be observed in its absence?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The introduction of the Geometric Invariance Hypothesis appears novel and extends findings of Neural Anisotropy Directions (Ortiz-Jimenez et al., 2021) to non-linear decision boundaries. This hypothesis has the potential to provide insights into the relationship between neural network architecture and the structure of data, contributing to our understanding of inductive biases in deep learning.\n\n2. The experiments and the theoretical analysis are generally fair, although several imprecisions are present, and some clarifications are needed (see detailed points below).\n\n3. The theoretical findings are tested using real networks (e.g., ResNets, ViTs) and datasets (CIFAR). These experiments appear to align with and support the GIH. Nevertheless, it is worth noting that the considered settings are highly overparameterized and relatively simple, which raises concerns regarding the broader validity and general significance of the results (see detailed points below).", "weaknesses": "The paper is quite dense. There are multiple points of confusion and imprecisions affecting both clarity and soundness. Specifically:\n\n4. The introduction and main text lack a comprehensive overview of the field and references to related work. Only a few broad papers are cited, despite the extended page limit of this year's edition. I strongly encourage the authors to move much of the discussion from Appendix A.1 into the main text to better place the work in context. In particular, NADs introduced in Ortiz-Jimenez et al. (2021) should be explicitly discussed in the main text, given how strongly related they are to this work.\n\n5. Lines L058+ seem to confuse expressivity with generalization, i.e., approximation vs. statistical properties. The fact that an MLP is universal (as some deep and wide CNNs are) does not imply that it can generalize effectively. \n\n6. Assuming i.i.d. parameters is not enough to conclude that $G_F \\propto I$. A Gaussian distribution with a non-zero mean is a counterexample.\n\n7. The claim that MLPs have a lack of inductive bias (L210, L224, L249) is wrong. Even in the NTK regime, MLPs display a strong spectral bias for low modes. It may be more accurate to state that MLPs at initialization have no preferred directions in the input space, i.e., they have an isotropic prior in the input space.\n\n8. Multiple terms are left undefined or used imprecisely: e.g., “SSE loss” and $y_{\\mu}$. Some phrases are vague: What is the “input-space curvature”, and how can it depend on the training process (L011)? What is the “input geometry” of a neural network (L076, L521)? What does it mean for a dimension to be noise (L053)? What are “all the possible values of $\\theta$” ( L145)? How are they distributed? Given that you consider gradient-flow in your analysis, how do you define the “first step” in L286?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7iby3qvIPW", "reviewer": "Reviewer_xFyb", "summary": "In their paper, the authors argue for the link between input-space geometry and the ability of models to generalize. They propose the geometric invariance hypothesis (GIH) which states that, depending on the architecture of the model, the curvature of the input-space remains invariant under transformations in certain directions. They then attempt to prove and study this phenomenon by separating the changes in the input-space geometry during training into architecture-dependent changes and data-dependent changes. This resulted in finding the special cases where the invariance occurs, and they link this to the generalization ability of the model and where it collapses to reliance on noise and memory.", "questions": "Just as a small note, in line 211 you mention the interaction between two factors, specifically the data-dependent factor and another. I'm guessing this is the model's geometric inductive bias? If so, I think it could be written a bit clearer. \n\nAdditionally, would you think this approach would be useful for architecture-based optimal dataset subsampling methodology?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is able to gradually build up to the main hypothesis being proposed while maintaining a clear chain of reasoning. The authors also provide extensive mathematical proofs for each step in the build-up and mention what assumptions are made and any limitations on what can be shown or derived. Finally, they are able to provide some insight into the effect of this hypothesis on an architecture's generalization ability while addressing any possible ideas with empirical results.", "weaknesses": "While the \"performance\" gains of the paper do seem marginal, I see these experiments as more of a proof of concept of the ideas and the proposed hypothesis. However, it would have been nice to see these experiments on multiple datasets to verify if the claims still hold, especially given the simplicity of the current model choices as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LEh86TpMbm", "reviewer": "Reviewer_tWDw", "summary": "This paper introduces the geometric invariance hypothesis (GIH), which suggests that during the training of neural networks, the curvature of the input space remains invariant under transformations along specific directions determined by the network’s architecture. The authors begin by examining a non-linear binary classification problem situated on a hyperplane in a high-dimensional space. They observe that a multilayer perceptron (MLP) can solve this problem regardless of the hyperplane’s orientation, whereas a residual network (ResNet) cannot. Motivated by this finding, they define two architecture-dependent maps: the average geometry, which provides a compact summary of the network’s input-space geometry, and the average geometry evolution, which describes how this geometry changes during training. Through both theoretical analysis and empirical investigations, the paper demonstrates that GIH arises because the average geometry evolution closely approximates the projection of the data covariance onto the average geometry. This results in an invariance property, especially when the average geometry is low-rank. Finally, the authors present extensive experimental results to explore the consequences of GIH and its relationship to generalization in neural networks.", "questions": "I am interested in how your observations are affected by different training dynamics, specifically when using stochastic gradient descent (SGD) with large learning rates—a common practice for achieving high prediction accuracy. Have you explored how the geometric invariance hypothesis (GIH) and the behavior of average geometry and its evolution manifest under such training conditions?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The introduction of the geometric invariance hypothesis (GIH) offers a fresh and innovative perspective on the interplay between neural network architectures and the geometry of the input space during training. By proposing the concepts of average geometry and average geometry evolution, the authors provide novel tools for quantifying how different architectures influence learning dynamics. This approach moves beyond traditional analyses by directly linking architectural properties to geometric transformations in the input space, which is a significant conceptual advancement in understanding deep learning models.", "weaknesses": "While the paper makes significant contributions, there are areas that could be improved:\n\n- Lack of Intuitive Explanation: It is challenging to develop an intuition for why ResNets behave differently from MLPs. Providing more intuitive explanations or illustrative examples before introducing the mathematical formalism would help readers grasp the core concepts and follow the subsequent analysis more effectively.\n- Limited Architectural Comparison: The focus on ResNets without discussing other architectures like AlexNet leaves some gaps. Clarifying whether the observed behaviors are due to specific features like skip connections or are common across different architectures would strengthen the generality of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d1t5RT08T7", "reviewer": "Reviewer_sttW", "summary": "The authors study the input-space geometry of neural networks, i.e., the decision boundaries and loss surface at each point in the input space. They give mathematical characterizations of the expected geometry and dynamics of the geometry under gradient flow at initialization. They make some intuitive conjectures about how the geometry evolves under training, one of which is the Geometric Invariance Hypothesis, which says that the geometry of the model remains invariant in \"un-important\" directions of the input space while training. They then experimentally verify parts of their conjectures using some representative architectures on both synthetic data and smaller-scale real data.", "questions": "- The experiments remove normalization from the neural networks. Is this crucial? Does the trend totally deteriorate with normalization? Is there any hypothesis made by the theory as to why?\n- Regarding the Conjectures 1 and 2 about dynamics: Is there an infinite- or large-width assumption being implicitly made, for instance? Should readers interpret the Conjectures as holding in appropriate scaling limits or should they hold at reasonable scales?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The examination of input-space geometry of neural networks is a more novel approach: other more mainstream approaches such as NTK and AGOP consider the parameter-space geometry.\n- The conclusions are interesting, and to some degree, sensible: if there are directions in the input space which do not support the training data distribution, then the training dynamics may not move the input-space geometry in these directions.\n- The paper does a good job motivating a very theoretical research direction with plausible practical implications in Section 5.\n- The paper is reasonably well-written; despite the material being technically dense, it mostly makes sense after a thorough read.", "weaknesses": "- The definitions of geometry in this work are based on expected second moments of the data. This captures the smallest supporting subspace of the data. But data can have very large linear dimension while having small intrinsic dimension, cf \"manifold hypothesis\". The theory here does not seem to capture this more involved geometric aspect, but this assumption has been hypothesized to be more realistic for real data. The lack of understanding here makes the experiments less likely to track with the theory and may preclude more prescriptive insights, beyond those generated by the preliminary experiments in Section 5.\n- More minor: There is not much intuition provided about why the predictions made at initialization should transfer reasonably well to training, and thus (in my opinion) not too much motivation for the Conjectures (which are about training dynamics).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)