Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture

## Abstract

Abstract In this paper, we propose the geometric invariance hypothesis (GIH) , which argues that when training a neural network, the input space curvature remains invariant under transformation in certain directions determined by its architecture. Starting with a simple non-linear binary classification problem residing on a plane in a high dimensional space, we observe that while an MLP can solve this problem regardless of the orientation of the plane, this is not the case for a ResNet. Motivated by this example, we define two maps that provide a compact architecture-dependent summary of the input space geometry of a neural network and its evolution during training, which we dub the average geometry and average geometry evolution , respectively. By investigating average geometry evolution at initialization, we discover that the geometry of a neural network evolves according to the projection of data covariance onto average geometry. As a result, in cases where the average geometry is low-rank (such as in a ResNet), the geometry only changes in a subset of the input space. This causes an architecture-dependent invariance property in input-space curvature, which we dub GIH. Finally, we present extensive experimental results to observe the consequences of GIH and how it relates to generalization in neural networks.

## 1 Introduction

Figure: Figure 1: Geometric Invariance Hypothesis in ResNet. A simple binary classification problem, where the two classes correspond to two circles sharing the same center, residing on the plane determined by two orthogonal directions $\mathbf{u}_{1},\mathbf{u}_{2}\in\mathbb{R}^{D}$. The subspace orthogonal to $\mathbf{u}_{1},\mathbf{u}_{2}$ contains uncorrelated noise. The green dotted curves demonstrate the decision boundary of the model for different initializations and mini-batches. In experiment A, the model is trained on $\mathcal{D}_{A}$ with $\mathbf{u}_{1}=\mathbf{v},\mathbf{u}_{2}=\mathbf{v}_{A}$ and $\mathbf{v}\perp\mathbf{v}_{A}$, and in experiment B on $\mathcal{D}_{B}$ with $\mathbf{u}_{1}=\mathbf{v},\mathbf{u}_{2}=\mathbf{v}_{B}$ and $\mathbf{v}\perp\mathbf{v}_{B}$. The bottom-left corner of each figure shows the projection of the average geometry onto $\mathbf{u}_{1}$ and $\mathbf{u}_{2}$ ($\mathbf{u}^{\top}\mathbf{G}_{\mathcal{F}}^{t}\mathbf{u}$, with $\mathbf{G}_{\mathcal{F}}^{t}$ defined in Section [2](#S2)).
Refer to caption: /html/2410.12025/assets/x1.png

Ever since the advent of deep learning, the importance of architecture and its impact on performance, especially through introducing inductive biases, have been well-known and established Battaglia et al. (); Tay et al. (; ); White et al. (). However, the modern machine learning toolbox still lacks a unified tool that can provide an interpretable and tangible connection between architecture, data, and inductive biases. This has fragmented the approaches to architecture design to - among other things - heuristic-based modifications He et al. (); Ioffe & Szegedy (); Hochreiter & Schmidhuber (), biology-inspired design Bahdanau et al. (); Vaswani et al. (); LeCun et al. (), or under the sole interest of efficiency and stability Dosovitskiy et al. (); Gu & Dao (). Given that we can understand many consequential characteristics of a deep neural network in the input space - such as decision boundaries, smoothness, and the types of features the model relies on for prediction - by focusing on its geometry in the input space, we believe a focus on this particular topic can provide such a model for the inductive biases of a neural architecture.

Consider a simple $D$-dimensional non-linear binary classification problem, where the discriminative features of the two classes are two $2$-dimensional concentric circles (Figure [1](#S1.F1)) occupying a plane determined by the two orthogonal directions $(\mathbf{u}_{1},\mathbf{u_{2}})\in\mathbb{R}^{D}$. The other $D-2$ dimensions are uncorrelated noise. Solving such a problem with a nonlinear decision boundary was one of the motivations behind the exodus of the machine learning community from linear models (logistic regression, SVMs, etc.) to kernel methods and then neural networks Bishop (); Goodfellow et al. (). Conventional wisdom tells us that no matter on which directions the circles reside on, a modern neural network is complex enough to solve this problem with ease Zhang et al. (; ). For an MLP, this is indeed the case: any model that is wide enough and uses non-linearity with at least a single hidden layer can solve this problem for any randomly selected orthogonal $\mathbf{u}_{1},\mathbf{u}_{2}$. However, for a standard ResNet18 He et al. (), despite the elaborate architecture design and much larger size, for a randomly selected $\mathbf{u}_{1},\mathbf{u}_{2}$ the model is seldom capable of reaching more than $50\%$ accuracy on test samples. This observation indicates the existence of an architecture dependent geometric inductive bias in the input space, which brings us to our central question: How does the geometry of a neural network in the input space evolve during training, and what is the role of architecture and data in it?

In our quest to answer this question, we discover a phenomenon, which we dub the Geometric Invariance Hypothesis (GIH). GIH claims that for a family of neural networks $\mathcal{F}$, there exists an architecture-dependent subspace $\mathbb{S}_{\mathcal{F}}\subseteq\mathbb{R}^{D}$, where the geometry of neural networks in $\mathcal{F}$ can only change in directions $\mathbf{u}\in\mathbb{S}_{\mathcal{F}}$ during training. In the context of our example, let us assume two datasets $\mathcal{D}_{A}$ and $\mathcal{D}_{B}$ sharing x-axis $\mathbf{u}_{1}=\mathbf{v}\in\mathbb{S}_{\mathcal{F}}$, but have different y-axes $\mathbf{u}_{2}=\mathbf{v}_{A}\in\mathbb{S}_{\mathcal{F}}$ for $\mathcal{D}_{A}$ and $\mathbf{u}_{2}=\mathbf{v}_{B}\notin\mathbb{S}_{\mathcal{F}}$ for $\mathcal{D}_{B}$, with $\mathcal{F}$ corresponding to the ResNet18 architecture. As evident by Figure [1](#S1.F1), in this case the neural network is able to achieve generalization on $\mathcal{D}_{A}$, but not on $\mathcal{D}_{B}$. More specifically, the decision-boundaries (green dotted curves) show us that in $\mathcal{D}_{A}$ the model is able to find the optimal, circular-shaped decision boundary regardless of the stochasticity introduced by initialization or mini-batching. However, in $\mathcal{D}_{B}$, the model is missing the signal on the directions corresponding to $\mathbf{u}_{2}$, relying on noise on other dimensions $\in\mathbb{S}_{\mathcal{F}}$ in order to reduce the train loss, and thus failing to generalize.

In order to formalize a notion of “geometry in the input space” for a neural architecture and how it “evolves” during training, we define two distinct but related maps for a neural network with a scalar output, which we dub the average geometry ($\mathbf{G}_{\mathcal{F}}^{t}$ at time $t$ during training and for the family of models $\mathcal{F}$) and the average geometry evolution ($\Delta_{\mathcal{F}}^{t}$ at time $t$ during training), respectively. These two maps correspond to the covariance of the gradient of a neural network w.r.t. the input and its derivative w.r.t. time according to the gradient flow model Barrett & Dherin (). We find these functions to be related to decision boundaries through a first-order Taylor approximation, and also to the Hessian of the loss w.r.t. the input. Through investigating $\Delta_{\mathcal{F}}^{t}$ at initialization ($t=0$) for a simple MLP and CNN architecture, we observe that the changes in the input geometry of a neural network at initialization correspond to the projection of the train data covariance onto its geometry at initialization. Hence, we hypothesize a similar behavior for the remainder of training, which brings us to our definition of GIH, whereupon the role of $\mathbb{S}_{\mathcal{F}}$ is taken by the subspace spanned by the eigenvectors of $\mathbf{G}_{\mathcal{F}}^{0}$. We confirm GIH through experimentation on synthetic data and CIFAR-10.Later, we are able to reconcile GIH with another hypothesis aimed at investigating the inductive biases of neural networks, namely the simplicity bias hypothesis Pérez et al. (); Shah et al. (). Finally, we also discuss the connection between GIH and generalization, showing its usefulness for detecting the effect of features and samples on the performance of a neural network on CIFAR-10. Our contributions can be summarized as follows:

- •
We first define a function summarizing the input space geometry of a model and its evolution. We then factorize the contributors to the evolution of geometry during training into a data-dependent term and a model-dependent term. We investigate each element and their interaction theoretically, from which we conjecture a general form. Then, the results are confirmed empirically.
- •
Following these insights, we propose The Geometric Invariance Hypothesis (GIH), which states that the geometry of deep learning models in the input space becomes invariant in certain directions determined by the architecture.
- •
We provide extensive experimental results aimed at establishing the validity of GIH and how it relates to generalization in neural networks. Specifically, we show that through GIH, we can determine the location of the decision boundaries a neural network can learn. Furthermore, we observe that GIH can help us detect the features and train samples the model relies on for generalization.

For details on the experimental settings and a discussion on the limitations of our work, please refer to App. [A.9](#A1.SS9) and App. [A.10](#A1.SS10), respectively. For a summary of the related works, please refer to App. [A.1](#A1.SS1).

## 2 The Geometry of the Input Space

In this section, we will first provide the notations we use in our paper. Then, we provide preliminary definitions and results which we will use to formalize and motivate our claims in the rest of the paper.

### 2.1 Notations

We denote the pre-activation output of the classification layer of the model, which for simplicity we assume to be scalar (i.e., dealing with binary classification problems) as $f_{\theta}(\mathbf{x})$, parameterized by $\theta$ with $\mathbf{x}\in\mathbb{R}^{D}$. We assume the model to be a neural network, trained on $\mathcal{D}_{T}=\{\mathbf{x}_{\mu}\}_{\mu=1}^{m}$. We denote a family of models (i.e., architecture) as $\mathcal{F}$, with $f_{\theta}(\cdot)\in\mathcal{F}$. We denote the loss function as $\mathcal{L}(\theta)=\sum_{\mu=1}^{m}\ell\left(f_{\theta}\left(\mathbf{x}_{\mu}\right),y_{\mu}\right)$. Furthermore, we denote the gradient w.r.t. the input and the parameters by $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ and $\nabla_{\theta}f_{\theta}(\mathbf{x})$, respectively, and the second-order derivative w.r.t. the parameters and data as $\nabla_{\mathbf{x},\theta}^{2}f_{\theta}(\mathbf{x})$. For ease of analysis, we rely entirely on the gradient flow model for modeling the training procedure: i.e., we denote $t\in\mathbb{R}_{0}^{+}$ as the $t\geq 0$ moment in training and assume the optimization process to be continuous. So to investigate the changes in a particular function $g_{\mathcal{F}}(\mathbf{x},\theta)$, which is a function of input $\mathbf{x}$ and parameters $\theta$ for the family of model $\mathcal{F}$, we usually rely on $\frac{\text{d}g_{\mathcal{F}}}{\text{d}t}=\nabla_{\theta}g_{\mathcal{F}}(\mathbf{x},\theta)^{\top}\dot{\theta}$, where $\dot{\theta}=\frac{\text{d}\theta}{\text{d}t}$.

We also use a probing distribution over the input to get the average behavior of a particular function $g_{\mathcal{F}}(\mathbf{x},\theta)$, which we denote as $\mathbf{x}\sim\mathcal{P}$. The probing distribution can be any distribution of choice that covers the area of interest in the input space. For instance, we can use train or validation samples, or a standard Gaussian distribution, which is our choice for $\mathcal{P}$ in this paper. We also define $\mathbf{A}\xrightarrow[]{m_{1},m_{2},\dots,m_{k}}\mathbf{B}$ as $\left|\textbf{Corr}\left(\mathbf{A},\mathbf{B}\right)\right|$ approaching $1$ at the limit $\forall i~{}m_{i}\to\infty$, where $\mathbf{A}$ and $\mathbf{B}$ are both matrices of the same dimension, $\mathbf{Corr}(\cdot,\cdot)$ is the Frobenius cosine similarity, $|\cdot|$ is the absolute value operator, and $\mathbf{A}$ is a function of scalars $\{m_{i}\}_{i=1}^{k}$. More informally, we use $\mathbf{A}\propto\mathbf{B}$ when we claim $\left|\mathbf{Corr}(\mathbf{A},\mathbf{B})\right|$ for the two matrices $\mathbf{A}$ and $\mathbf{B}$ to be much larger than the magnitude of the cosine similarity between two randomly generated vectors of the same dimension to the point of one being able to approximate the other.

### 2.2 Definitions

In order to facilitate answering our central question, we will try to define two functions that help us understand the geometry of a neural network architecture in the input space, and how it changes during training. Given that our focus in this paper is on classification tasks, a good starting point would be the gradient of the model w.r.t. the input $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$. From robustness literature, we know that $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ can inform us about the decision-boundaries in the immediate vicinity of $\mathbf{x}$ Madry et al. (); Moosavi-Dezfooli et al. (). This can be easily shown through a first-order Taylor expansion of the output of the model. Specifically, let $\delta$ be a vector with a small L2 norm. Then we can write $f_{\theta}(\mathbf{x}+\delta)-f_{\theta}(\mathbf{x})\approx\delta^{\top}\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ using Taylor expansion. Therefore, $\delta$ in the direction of $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ maximizes the change in the output of the model, potentially flipping its prediction. Consequently, by maximizing $\delta^{\top}\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ we obtain valuable information about the local changes in $f_{\theta}(\mathbf{x})$ through $\delta$.

However, a $\delta$ computed based on $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ for a single value of $\theta$ only informs us about a decision-boundary for a single parameterization of the model, subjecting it to noise caused by stochastic factors such as mini-batching and initialization. Therefore, we propose to average the gradient over all the possible values of $\theta$: $\mathbb{E}_{\theta}\left[\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\right]$. While trying to maximize $\delta^{\top}\mathbb{E}_{\theta}\left[\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\right]$ can be a way to find a direction that can potentially flip the prediction of $f_{\theta}(\mathbf{x})$, it only contains first-order information about how the distribution of $\theta$ affects the local changes in $f_{\theta}(\cdot)$ at $\mathbf{x}$, causing potential loss of information. For instance, let us assume a case where $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ is zero-mean w.r.t. $\theta$, i.e., $\mathbf{x}$ corresponds to a local extremum. In this case, $\mathbb{E}_{\theta}\left[\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\right]$ does not contain any information about the local changes in $f_{\theta}(\mathbf{x})$. On the other hand, one can instead look at the covariance of $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ w.r.t. $\theta$ to obtain information about the directions towards which the decision boundaries have a significant presence. Therefore, a much more informative function about the local changes in $f_{\theta}(\mathbf{x})$ in the input space would be $\mathbb{E}_{\theta}\left[\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})^{\top}\right]$. Finally, since we rarely care about the behavior of a model for a single input, we also use a probing distribution $\mathcal{P}$ to sample $\mathbf{x}$ from. With this motivation, we introduce the following definition for a function informing us about the geometry of a neural network architecture in the input space:

###### Definition 2.1 .

In App. [A.3](#A1.SS3), we report that $\mathbf{G}_{\mathcal{F}}^{t}(\mathbf{x})$ is indeed directly related to the curvature of the loss w.r.t. input, making the connection between the function we propose and the geometry of a neural network architecture in the input space clear.

We can capture the geometry of a family of models in the input through $\mathbf{G}_{\mathcal{F}}^{t}(\mathbf{x})$ at time $t$ during training. However, this function does not tell us how the model geometry changes at time $t$, thus only providing half of the picture. So we need another quantity that can inform us about the evolution of the geometry in the input space. Given our line of reasoning that resulted in the definition of average geometry, we start with the function $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})^{\top}$. Following the gradient-flow model, we can capture the evolution of this function by getting its derivative w.r.t. time, which gives us: $\nabla_{\mathbf{x},\theta}^{2}f_{\theta}(\mathbf{x})\dot{\theta}\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})^{\top}+\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\dot{\theta}^{\top}\nabla_{\mathbf{x},\theta}^{2}f_{\theta}(\mathbf{x})^{\top}$. With a similar motivation, we get the expectation of this quantity w.r.t. the parameters $\theta$ and $\mathbf{x}$ sampled from a probing distribution, resulting in the following definition:

###### Definition 2.2 .

In Appendix [A.3](#A1.SS3), we will see that $\Delta_{\mathcal{F}}^{t}(\mathbf{x})$ is indeed directly related to the evolution of the curvature of the loss w.r.t. input, making the connection between the function and the evolution of geometry of a neural network architecture in the input space clear. For the SSE loss we have:

$$ $\Delta_{\mathcal{F}}^{t}(\mathbf{x})=-\sum_{\mu=1}^{m}\mathbb{E}_{\theta_{t}}\left[\left(\nabla_{\mathbf{x},\theta}^{2}f_{\theta_{t}}(\mathbf{x})\nabla_{\theta}f_{\theta_{t}}(\mathbf{x}_{\mu})\nabla_{\mathbf{x}}f_{\theta_{t}}(\mathbf{x})^{\top}+\nabla_{\mathbf{x}}f_{\theta_{t}}(\mathbf{x})\nabla_{\theta}f_{\theta_{t}}(\mathbf{x}_{\mu})^{\top}\nabla_{\mathbf{x},\theta}^{2}f_{\theta_{t}}(\mathbf{x})^{\top}\right)\cdot\left(f_{\theta_{t}}(\mathbf{x}_{\mu})-y_{\mu})\right)\right].$ (3) $$

Following ([3](#S2.E3)), we can argue that the changes in the local geometry are the result of the interaction between two main components: 1) the data, and 2) the current average geometry. The first component affects the geometry through the changes in the parameter ($\dot{\theta}$) by gradient-descent, while the second component affects the geometry through the mixed-derivative $\nabla_{\mathbf{x},\theta}^{2}f_{\theta}(\mathbf{x})$ and the gradient $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$. While such an interaction may seem extremely complex at first, we will see that by probing the average geometry and average geometry evolution with an appropriate probing distribution $\mathcal{P}$, we observe (through theoretical and empirical means) that the expected value of this interaction is actually surprisingly simple and akin to a linear projection, leading to interesting results on the behavior of the average geometry $\mathbf{G}_{\mathcal{F},\mathcal{P}}^{t}$ and generalization in a neural network.

For simplicity, we refer to both of the average geometry and the average geometry evolution of a standard Gaussian probing distribution and at initialization (i.e., $t=0$) as $\mathbf{G}_{\mathcal{F}}$ and $\Delta_{\mathcal{F}}$, respectively. For $t>0$, we denote these two functions as $\mathbf{G}_{\mathcal{F}}^{t}$ and $\Delta_{\mathcal{F}}^{t}$. Furthermore, we set $\mathbf{S}=\sum_{\mu=1}^{m}x_{\mu}x_{\mu}^{\top}$ as the unnormalized covariance of the data.

###### Definition 2.1 .

###### Definition 2.2 .

## 3 Changes in the Average Geometry During Training

In this section, we study the factors that affect the changes in the geometry of the model in the input space through theoretical and empirical means. More specifically, we will start by theoretically detecting the data-dependent factor in $\Delta_{\mathcal{F}}$ by investigating it in an “isotropic” model. The term “isotropic,” as will become clear in this section, indicates a lack of geometric inductive bias in the model. We then theoretically look at the interaction between the two factors in a “non-isotropic” model, which we set to be a convolutional neural network with a pooling layer. Motivated by these theoretical results, we will conjecture that at the initial stage of training, the changes in the average geometry of the model follow a linear dynamic driven by an interaction between the initial geometry ($\mathbf{G}_{\mathcal{F}}$) due to the model and the covariance of the data. In Appendix [A.4](#A1.SS4), we provide theoretical and empirical results supporting the claim that this dynamic is independent of the labels, and can be considered task-independent.

### 3.1 Isotropic model

In this subsection, we will provide theoretical results for the average geometry evolution $\Delta_{\mathcal{F},\mathcal{P}}$ at initialization according to the gradient flow model, over the probing distribution $\mathcal{P}=\mathcal{N}\left(0,\sigma_{x}^{2}\mathbf{I}\right)$(^1^11Given the homogeneous quality of the architectures used in our theoretical analysis, we will set $\sigma_{x}=1$ for the probing distribution.). We do not make assumptions about the data, and instead focus on models that can be considered “isotropic” Battaglia et al. ().

First, let us start with a simple example of a linear regression (i.e., $\mathcal{F}$ corresponding to linear regression models), which we write as $f_{\theta}(\mathbf{x})=\theta^{\top}\mathbf{x}$. Since $\nabla_{\mathbf{x}}f_{\theta}(x)=\theta$, using any i.i.d. initialization for $\theta$, the average geometry at initialization for this model will be $\mathbf{G}_{\mathcal{F}}=\mathbb{E}_{\theta_{0}}\left[\theta_{0}\theta_{0}^{\top}\right]\propto\mathbf{I}$. This property indicates a “lack of geometric inductive biases” at initialization, which as shown in Appendix [A.5](#A1.SS5), is also shared with MLPs of any depth with ReLU non-linearities. In Section [3.2](#S3.SS2), we will elaborate on the significance of this property. Following ([3](#S2.E3)), and given that $\nabla_{\mathbf{x},\theta}^{2}f_{\theta}(\mathbf{x})=\mathbf{I}$, we have $\Delta_{\mathcal{F}}=-\sum_{\mu=1}^{m}\mathbb{E}\left[\left(\theta^{\top}\mathbf{x}_{\mu}-y_{\mu}\right)\cdot\mathbf{x}_{\mu}\theta^{\top}\right]=-\sigma_{\theta}\mathbf{S}$ for an i.i.d. Gaussian initialization of $\theta$ with standard deviation $\sigma_{\theta}$, following Stein’s lemma. Therefore, we can see that in the absence of non-linearity and more layers, the average geometry evolution of an isotropic model at initialization corresponds to the covariance of the data. When introducing these two elements, in the form of an MLP with a single hidden layer and ReLU non-linearity, we have the following theorem that shows a similar behavior:

###### Theorem 3.1 .

Let $\mathcal{F}$ be the family of MLPs with a single hidden layer of size $n$ and ReLU non-linearity.
Assuming that we use the SSE loss, then as the input dimension $D$ and the model width $n$ become larger, the average geometry at initialization $\Delta_{\mathcal{F}}$ approaches the data covariance $\mathbf{S}$ up to a constant, i.e., $\Delta_{\mathcal{F}}\xrightarrow{n,D}\mathbf{S}$. The convergence rate is $\mathcal{O}\left(\frac{1}{\sqrt{D}\cdot n}\right)$.

We provide a sketch of the proof and encourage the reader to read the whole proof in Appendix [A.8.2](#A1.SS8.SSS2). The average geometry evolution of this model at initialization corresponds to $n^{2}$ terms of the second and fourth momentum of the parameters of the first layer $\phi$ over the space shared by halfspaces defined by the train data and the inputs sampled from the probing distribution, i.e. $\mathbb{E}_{\phi}\left[\mathbf{1}_{\phi^{\top}\mathbf{x}>0}\mathbf{1}_{\phi^{\top}\mathbf{x}_{\mu}>0}\phi\phi^{\top}\right]\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}$ and $\mathbb{E}_{\phi}\left[\mathbf{1}_{\phi^{\top}\mathbf{x}>0}\mathbf{1}_{\phi^{\top}\mathbf{x}_{\mu}>0}\left(\phi^{\top}\mathbf{x}_{\mu}\right)^{2}\phi\phi^{\top}\right]$. As the input dimension becomes large enough, these two halfspaces will almost surely become orthogonal, resulting in the second and the fourth momentums of the first-layer parameters over the intersection of these two halfspaces becoming equal to the second and the fourth momentums up to a constant. The second momentums yield $n^{2}-n$ terms corresponding to $\mathbf{S}$, while the fourth momentums yield $n^{2}-n$ terms corresponding to $\mathbf{S}$ and $n$ terms corresponding to $\mathbb{E}_{\mathbf{x}_{\mu}\in\mathcal{D}_{T}}\left[\|x_{\mu}\|_{2}^{2}\right]\mathbf{I}$, which vanish in favor of the $\mathbf{S}$ terms as the depth of the first layer becomes large enough.

Theorem [3.1](#S3.Thmtheorem1) gives us an interesting insight into our factorization of $\Delta_{\mathcal{F}}$ underscoring that in the absence of inductive biases introduced by the model, the geometry adapts according to the data covariance. Therefore, the data-dependent component in ([3](#S2.E3)) is equivalent to the empirical covariance of the data estimated on the train samples.

###### Theorem 3.1 .

### 3.2 The linear relationship between geometry, data, and architecture

We now shift our focus to examining the effect of the model on the geometry and the interaction between the data-dependent and the model-dependent components in $\Delta_{\mathcal{F}}$. Specifically, we aim to introduce architecture components, such as convolution operations and pooling layers, that as noted in Appendix [A.5](#A1.SS5), introduce geometric inductive biases to the model.
We start our investigation by looking at a neural network with a linear activation function:

###### Theorem 3.2 .

Let $\mathcal{F}$ be the family of convolutional neural networks with a single hidden layer, linear activation function, and a global average pooling layer on each channel. Assuming that we use the SSE loss, then as the model width $n$ becomes larger, the average geometry evolution at initialization $\Delta_{\mathcal{F}}(\mathbf{x})$ approaches $\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})$ up to a constant, i.e., $\Delta_{\mathcal{F}}(\mathbf{x})\xrightarrow{n}\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})$. The convergence rate is $\mathcal{O}\left(1/n\right)$. For proof, please refer to Appendix [A.8.3](#A1.SS8.SSS3).

From Theorem [3.2](#S3.Thmtheorem2), we observe that in the initial stages of training, $\mathbf{G}_{\mathcal{F}}^{t}$ will change towards the linear projection of $\mathbf{S}$ onto $\mathbf{G}_{\mathcal{F}}$ in a linear convolutional neural network with pooling. Considering the results of Theorem [3.1](#S3.Thmtheorem1) and Theorem [3.2](#S3.Thmtheorem2), we can now better understand the effect of the two components present in ([3](#S2.E3)) causing the shift in geometry at the beginning of training: the data in the form of covariance, and the architecture in the form of average geometry. We provide empirical confirmation for the two theorems in Figure [7](#A1.F7).

Given the results of Theorem [3.1](#S3.Thmtheorem1) and Theorem [3.2](#S3.Thmtheorem2), we will conjecture about $\Delta_{\mathcal{F}}$ for a neural network without making assumptions about the architecture. From Theorem [3.1](#S3.Thmtheorem1), we know that in the first step of gradient-descent, the changes in the geometry in the presence of an isotropic model (more explanation in Appendix [A.5](#A1.SS5)) corresponds to the covariance of the data. From Theorem [3.2](#S3.Thmtheorem2), we understand that the data covariance is linearly projected on the average geometry at initialization $\mathbf{G}_{\mathcal{F}}$ for the non-isotropic model used in the theorem. Then, assuming that this is the case in all neural networks trained via gradient-descent, without any assumptions about their architecture, we have the following:

###### Conjecture 1 .

So the changes in the geometry are caused by the projection of the covariance of the data onto the initial average geometry. As a result, in the case of the existence of a structure in the initial average geometry $\mathbf{G}_{\mathcal{F}}$, one can fully expect the geometry to be invariant (or numerically, changing extremely slowly) in certain directions with low correlation to the initial average geometry.

Considering that the average geometry evolution and our analysis of $\Delta_{\mathcal{F}}^{t}$ is based on the gradient-flow model, we cannot confirm Conjecture [1](#Thmconjecture1) directly in a discrete training regime. Therefore, in order to present empirical evidence confirming our results so far, we instead rely on looking at the average geometry $\mathbf{G}_{\mathcal{F}}^{t}$ for the entirety of the training period using gradient descent. In this case, an increasing correlation between $\mathbf{G}_{\mathcal{F}}^{t}$ and $\mathbf{G}_{\mathcal{F}}\mathbf{S}\mathbf{G}_{\mathcal{F}}$ (or $\mathbf{S}$ for an MLP) near the initialization point in training will support our theoretical analysis. We provide the experimental results supporting Conjecture [1](#Thmconjecture1) in Figure [2](#S3.F2). In this experiment, we plot the correlation between average geometry at time $t$ ($\mathbf{G}_{\mathcal{F}}^{t}$) and the projection of data onto the initial average geometry of the model, as proposed in Conjecture [1](#Thmconjecture1). We will use the normalized Frobenius dot product to measure the correlation between two matrices. For our experiments, we use the CIFAR-10 data. We construct a binary variant of CIFAR-10 in which the task is to distinguish animal from non-animal inputs, which we dub CIFAR-2. For each class of CIFAR-2, we sample 5000 data points randomly(^2^22Note that per the results of Corollary [A.4](#A1.Thmtheorem4), in case of normalization, $\mathbf{G}_{\mathcal{F}}$ will depend on the variance of each individual patch of the input. In our experiments, we found that in practice it is difficult to exactly compute this “normalized” variant of $\mathbf{G}_{\mathcal{F}}$ due to its dependence on, among other things, the kernel size of the convolution operations or the patch size of the ViT. For this reason, and for the sake of simplicity, we omit the normalization layers from the architectures we experiment on for non-synthetic data.). Furthermore, we report the results for an MLP with 2 hidden layers of size 100, LeNet, ResNet18 without batch normalization, and ViT without layer normalization. More details on the implementation can be found in the appendix.

Figure: (a) MLP
Refer to caption: /html/2410.12025/assets/x4.png

As we can observe in Figure [2](#S3.F2), there is clear evidence for the changes in the average geometry being highly correlated with $\mathbf{S}$ and $\mathbf{G}_{\mathcal{F}}\mathbf{S}\mathbf{G}_{\mathcal{F}}$ at the beginning of training, which is what we claim in Conjecture [1](#Thmconjecture1). Another interesting observation to note is that the values of $\mathbf{S}$ and $\mathbf{G}_{\mathcal{F}}\mathbf{S}\mathbf{G}_{\mathcal{F}}$ are virtually identical in the case of MLPs, which confirms our view of MLPs being isotropic, or in other words, having no geometric inductive biases, which manifests itself in the form of $\mathbf{G}_{\mathcal{F}}\approx\mathbf{I}$. On the other hand, this is not the case in the other models, with the correlation between $\mathbf{G}^{t}_{\mathcal{F}}$ and $\mathbf{G}_{\mathcal{F}}\mathbf{S}\mathbf{G}_{\mathcal{F}}$ being significantly larger than MLP during the linear stage compared to $\mathbf{S}$. This difference in correlation indicates the existence of a structure in $\mathbf{G}_{\mathcal{F}}$ for these models, which we consider to be the geometric inductive biases of the model.

###### Theorem 3.2 .

###### Conjecture 1 .

## 4 The Geometric Invariance Hypothesis

In this section, we will introduce the Geometric Invariance Hypothesis (GIH), providing empirical evidence supporting it, and investigating its implications on the generalization ability of deep networks. Notably, we study the relationship between the complexity of the decision boundary and GIH, establishing a connection between GIH and the simplicity bias hypothesis (SBH) Pérez et al. ().

### 4.1 The projection of discriminant features onto the initial geometry

In the previous section, we observed an intriguing phenomenon wherein at the initial stages of training, the average geometry of the model changes according to the projection of the covariance of the data $\mathbf{S}$ onto the initial average geometry $\mathbf{G}_{\mathcal{F}}$.
Note that as we get further into the training process, we expect the set of features used by the model to narrow down to “discriminant features,” which can be viewed as the features upon which the model can discriminate data. Assuming that the changes in the geometry are still caused by the projection of a subspace of the data support (relating to the discriminant features) onto the initial average geometry, and the initial average geometry itself is structured, then the input geometry will remain invariant in the directions with low correlation with $\mathbf{G}_{\mathcal{F}}$(^3^33Note that while we use the term invariance to refer to the geometry of the model having small changes in certain directions, in practice this “invariance” comes in the form of a large condition number in $\mathbf{G}_{\mathcal{F}}$. As a result, the invariance happens on a spectrum, and the changes in $\mathbf{G}_{\mathcal{F}}^{t}$ in a specific direction $\mathbf{u}$ gradually decreases as the correlation between $\mathbf{u}$ and $\mathbf{G}_{\mathcal{F}}$ becomes smaller.). In Appendix [A.5](#A1.SS5), we consider the choices in the architecture that can result in a structure in $\mathbf{G}_{\mathcal{F}}$. Furthermore, in Appendix [A.6](#A1.SS6) we will observe the structure in $\mathbf{G}_{\mathcal{F}}$ for the neural networks used in this paper.

This observation indicates the possibility that the initial geometry also plays a role in the later stages of training. More specifically, we want to know whether the model’s geometry in the input space will remain invariant during later stages of training in directions with low correlation with $\mathbf{G}_{\mathcal{F}}$ as well. We formalize this concept in the form of the following hypothesis, which we dub the “ Geometric Invariance Hypothesis” (GIH):

###### Conjecture 2 .

Figure: (a) MLP
Refer to caption: /html/2410.12025/assets/x8.png

Figure: (a) LeNet
Refer to caption: /html/2410.12025/assets/x12.png

In order to support Conjecture [2](#Thmconjecture2), we first define a measure to monitor the changes in the average geometry, which we call the geometric velocity: $\dot{\mathbf{G}}_{t}=1-\textbf{Corr}\left(\mathbf{G}_{\mathcal{F}}^{t},\mathbf{G}_{\mathcal{F}}^{t-1}\right)$, where $\textbf{Corr}(\cdot,\cdot)$ is the normalized Frobenius dot product. For this experiment, we will look at $\dot{\mathbf{G}}_{t}$ and training accuracy when the data is sampled from the Gaussian distribution $\mathcal{N}\left(0,\mathbf{G}_{\mathcal{F}}/\|\mathbf{G}_{\mathcal{F}}\|_{2}\right)$, where $\|\mathbf{G}_{\mathcal{F}}\|_{2}$ is the spectral norm of $\mathbf{G}_{\mathcal{F}}$. We compare training accuracy and the velocity of this dataset to another dataset where the data is sampled from another Gaussian distribution $\mathcal{N}\left(0,\text{flip}(\mathbf{G}_{\mathcal{F}})/\|\text{flip}(\mathbf{G}_{\mathcal{F}})\|_{2}\right)$, with $\text{flip}(\mathbf{G}_{\mathcal{F}})$ being $\mathbf{G}_{\mathcal{F}}$ with its eigenvalues flipped (i.e., in the reverse order). The experimental results are observable in Figure [3](#S4.F3).

We observe very little change in the average geometry of the model when trained on the data with $\text{flip}(\mathbf{G}_{\mathcal{F}})$ covariance compared to the data with $\mathbf{G}_{\mathcal{F}}$ covariance. This larger shift in the geometry supports Conjecture [2](#Thmconjecture2), showing that the geometry of the model will change very little in directions with a low correlation with $\mathbf{G}_{\mathcal{F}}$. In Section [4.2](#S4.SS2), we will see that neural networks trained using gradient descent may fail to generalize when the normal to the decision boundary aligns with directions exhibiting geometric invariance, as defined by Conjecture [2](#Thmconjecture2).

###### Conjecture 2 .

### 4.2 The geometric invariance hypothesis and the generalization gap

In this section, we try to investigate the relationship between GIH and generalization.
In particular, we define three types of labeling for the CIFAR-10 dataset with varying complexity: linear as $y=\textbf{sgn}((\mathbf{x}^{\top}\mathbf{u})+\mathbf{b})$, quadratic as $y=\textbf{sgn}((\mathbf{x}^{\top}\mathbf{u})^{2}+\mathbf{b})$, and sinusoidal as $y=\textbf{sgn}(\sin(\bar{\mathbf{x}}^{\top}\mathbf{u})+\mathbf{b})$
, where $\bar{\mathbf{x}}$ corresponds to a standard normalized input(^4^44We added a normalization component to the sinusoidal labeling in order to control the complexity of the problem by controlling the variance of $\mathbf{x}$, which is set to $1.0$ on each channel separately.), and $\mathbf{u}$ corresponds to the discriminant feature. We train and evaluate a model on each labeling function for $\mathbf{u}$ set to the eigenvectors of $\mathbf{G}_{\mathcal{F}}\mathbf{S}\mathbf{G}_{\mathcal{F}}$ in descending order, which can be seen in Figure [4](#S4.F4). In order to make sure that our observation is not caused by a lack of separability for certain selections of $\mathbf{u}$, we set the learning rate and number of epochs in a way to ensure all models will reach $100\%$ train accuracy. Also, in order to simulate the effect of label noise, we add a small amount of noise to the input when calculating the labels (but not to the input itself during training)(^5^55We sample i.i.d. Gaussian noise scaled on each channel separately, setting the variance to $0.2$ times the variance of the channel.).

As we can observe in Figure [4](#S4.F4), there is a clear correlation between the eigenvalues corresponding to $\mathbf{u}$ and the test accuracy of the model on all labeling functions, with datasets labeled based on a $\mathbf{u}$ with a smaller eigenvalue having larger generalization gap. These results further extend Ortiz-Jiménez et al. (), which mainly focused on linear decision boundaries. Note that as we saw in the previous subsection, for datasets with a low correlation between the discriminant feature and $\mathbf{G}_{\mathcal{F}}$, the model is incapable of finding the actual decision boundary. As a result, it will instead rely on features in other directions to “memorize” the train samples, resulting in a lack of generalization. Consequently, the GIH explains this phenomenon, which was first observed in Ortiz-Jiménez et al. (). Another interesting observation to make is that for more complex decision boundaries, GIH will have a “sharper” impact on generalization, meaning that the decision boundary needs to reside in $\mathbf{u}$s with larger eigenvalues in order for the model to detect the decision boundary. This observation appears to be separate from GIH, and best explained by the SBH. However, as we will see in the next subsection, the geometric inductive biases need to be considered when talking about SBH. In Appendix [A.7](#A1.SS7), you can observe the relationship between GIH and generalization in the presence of “isotropic” data.

### 4.3 Rethinking the simplicity bias hypothesis

Figure: (a) LeNet
Refer to caption: /html/2410.12025/assets/x15.png

The simplicity bias hypothesis (SBH) argues that neural networks prefer to learn features with simpler relationships with the predictive variable. However, as we observed in the previous subsection, depending on the direction of these features, the model may not be capable of learning them at all. So in order to reconcile the SBH with GIH, we design the following experiment involving synthetic data formulated as $\mathbf{x}=\epsilon\mathbf{u}_{1}y+\mathbf{z}_{y}^{\mathbf{u}_{2},\mathbf{u}_{3}}+\omega$, with Gaussian noise $\omega\sim\mathcal{N}\left(0,\sigma_{\omega}^{2}\left(\mathbf{I}-\mathbf{u}_{1}\mathbf{u}_{1}^{\top}-\mathbf{u}_{2}\mathbf{u}_{2}^{\top}-\mathbf{u}_{3}\mathbf{u}_{3}^{\top}\right)\right)$
for orthogonal $\mathbf{u}_{1},\mathbf{u}_{2},\mathbf{u}_{3}$. We define $\mathbf{z}_{y}$ as $\mathbf{z}_{y}^{\mathbf{u}_{2},\mathbf{u}_{3}}=r_{y}\cdot\frac{\alpha\cdot\mathbf{u}_{2}+\beta\cdot\mathbf{u}_{3}}{\|\alpha\cdot\mathbf{u}_{2}+\beta\cdot\mathbf{u}_{3}\|_{2}}$,
with $\alpha$ and $\beta$ being standard normal variables, and $r_{1}\neq r_{-1}$ for uniformly distributed $y\in\{-1,1\}$.

Therefore, in this experiment we have a linear discriminant and a non-linear discriminant feature in the form of $\mathbf{u}_{1}$ and $\mathbf{z}_{y}$. We select $\mathbf{u}_{2}$ and $\mathbf{u}_{3}$ from eigenvectors of $\mathbf{G}_{\mathcal{F}}$ with large eigenvalues, while changing $\mathbf{u}_{1}$ according to the eigenvectors of $\mathbf{G}_{\mathcal{F}}$. Figure [5](#S4.F5) reports the test accuracy for the linear and non-linear components. In agreement with GIH, we can see that when $\mathbf{u}_{1}$ has a high correlation with $\mathbf{G}_{\mathcal{F}}$, the model always selects the linear component since the linear component has perfect accuracy while the non-linear component does not do better than a random guess. However, as the correlation between $\mathbf{u}_{1}$ and $\mathbf{G}_{\mathcal{F}}$ decreases, we observe that the model becomes gradually more reliant on the non-linear component. Therefore, while SBH holds true when both components are on the same scale in terms of correlation with $\mathbf{G}_{\mathcal{F}}$ when the linear component has low correlation with $\mathbf{G}_{\mathcal{F}}$, the model learns the non-linear component contrary to what SBH would predict.

## 5 What Can Geometry Tell us about the Samples

In this section, we provide an application-based approach to the implications of the GIH. Specifically, we focus on the implications of the GIH on the feature space and the distribution of the samples, we try to see if the distribution of features and data can be related to the generalization ability of the model based on GIH.

Figure: (a) LeNet
Refer to caption: /html/2410.12025/assets/x19.png

**Table 1: The test accuracy of LeNet, ResNet18 without batch normalization, and ViT without layer normalization for the data distribution experiment. We report results for CIFAR-10, with datapoints eliminated randomly (R) or based on their score value (S). We report the mean performance along with a $68\%$ confidence interval for each setting. The best performance is boldface.**
| Model | Percentage of Removed Data (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | $10$ | $30$ | $50$ | $70$ | $90$ |
| LeNet-R | $65.0\pm 0.4$ | $61.7\pm 0.4$ | $56.6\pm 0.3$ | $52.4\pm 0.3$ | $45.1\pm 0.3$ |
| LeNet-S | $\mathbf{65.9\pm 0.5}$ | $\mathbf{62.5\pm 0.2}$ | $\mathbf{58.5\pm 0.4}$ | $\mathbf{53.5\pm 0.3}$ | $\mathbf{45.9\pm 0.2}$ |
| ResNet18-R | $78.6\pm 0.2$ | $75.8\pm 0.6$ | $71.8\pm 0.4$ | $65.1\pm 0.3$ | $52.4\pm 0.4$ |
| ResNet18-S | $\mathbf{79.6\pm 0.2}$ | $\mathbf{77.0\pm 0.2}$ | $\mathbf{73.2\pm 0.4}$ | $\mathbf{66.2\pm 0.3}$ | $\mathbf{54.4\pm 0.2}$ |
| ViT-R | $62.0\pm 0.1$ | $57.5\pm 0.1$ | $51.5\pm 0.1$ | $45.5\pm 0.1$ | $35.6\pm 0.1$ |
| ViT-S | $\mathbf{62.9\pm 0.1}$ | $\mathbf{58.4\pm 0.1}$ | $\mathbf{52.2\pm 0.2}$ | $\mathbf{46.5\pm 0.1}$ | $\mathbf{36.2\pm 0.1}$ |

Feature importance. A natural question that arises from the geometric invariance hypothesis is: will eliminating features with the lowest possible correlation with $\mathbf{G}_{\mathcal{F}}$ from the data have an impact on the performance? In order to answer this question, we design the following experiment. We find the directions with the lowest correlation with $\mathbf{G}_{\mathcal{F}}$ and the highest correlation with $\mathbf{S}$ by solving the generalized eigenvalue decomposition problem over $\mathbf{G}_{\mathcal{F}}$ and $\mathbf{S}$. Using the resulting eigenvectors(^6^66Note that the generalized eigenvalue decomposition is not necessarily numerically stable. As a result, the lower eigenvectors (i.e., with smaller eigenvalues) usually become less orthogonal. Therefore, we also perform a Gram-Schmidt orthogonalization procedure on the resulting eigenvectors to ensure they can span the input space.), we perform a simple orthogonal projection on the data to eliminate the features residing in those directions. More concretely, for a sample $\mathbf{x}$ and the set of spanning vectors $\mathbb{V}_{\text{GIH}}=\{\mathbf{v}_{1}^{n},\mathbf{v}_{2}^{n},\dots,\mathbf{v}_{D}^{n}\}$, we set $\mathbf{x}^{\perp\text{GIH}}_{k}=\mathbf{Q}_{k}^{\text{GIH}}\mathbf{x}$, where $\mathbf{Q}_{k}^{\text{GIH}}=\mathbf{I}_{D}-\sum_{i=1}^{k}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}$. We perform the orthogonalization on both the train and the test data, essentially eliminating the features from the data. As a baseline method, we also sample a set of spanning vectors $\mathbb{V}_{\text{Rand}}$ from a random Gaussian distribution orthogonalized using the Gram-Schmidt procedure. However, for a fair comparison, for a given $\mathbf{Q}_{k}^{\text{GIH}}$, we first find an index $k^{\prime}$ where the explained variance by $\{\mathbf{v}_{1}^{r},\mathbf{v}_{2}^{r},\dots,\mathbf{v}_{k^{\prime}}^{r}\}$ on $\mathbf{S}$ becomes no more than the explained variance by $\{\mathbf{v}_{1}^{n},\mathbf{v}_{2}^{n},\dots,\mathbf{v}_{k^{\prime}}^{n}\}$, and then set $\mathbf{Q}_{k}^{\text{Rand}}=\mathbf{I}_{D}-\sum_{i=1}^{k^{\prime}}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}$. We then train and evaluate the model on the orthogonalized dataset. The results from CIFAR-2 are reported in Figure [6](#S5.F6).

From the experiment results we can observe that eliminating features based on their correlation with the geometry at initialization causes a lower drop in performance than eliminating them randomly. Therefore, we can safely claim that the geometry of the model in the input space is to some degree invariant to some directions, and the model will not use them for prediction.

Sample importance. In this experiment, we try to understand whether there is a relationship between the contribution of a datapoint to the performance of the model and the amount of variation it has in the invariant directions of the geometry. Specifically, given a datapoint $\mathbf{x}$, we measure its correlation with the initial geometry $\mathbf{G}_{\mathcal{F}}$ as: $\textbf{Score}(\mathbf{x})=\left(\mathbf{x}/\|\mathbf{x}\|_{2}\right)^{\top}\mathbf{G}_{\mathcal{F}}\left(\mathbf{x}/\|\mathbf{x}\|_{2}\right)$. Then, we sort the datapoints according to their $\textbf{Score}(\cdot)$ value and eliminate the ones with the smallest score. We compare the results to a baseline wherein we randomly eliminate the datapoints. You can see the results of this experiment in Table [1](#S5.T1). For a more detailed experiment please refer to Table [2](#A1.T2) in the Appendix.

As evident by the experimental results, there is a clear relationship between the contribution of a datapoint to the performance and its $\textbf{Score}(\cdot)$ value. While the difference in performance between the random baseline and our geometry-based score is somewhat modest (at most about $2\%$), we note two factors are not considered in our $\textbf{Score}(\cdot)$: 1) whether the features present in a datapoint with high correlation with the initial geometry are actually discriminant or not, 2) whether by eliminating a datapoint, we lose a rare feature that is not frequent in the other train samples. Considering these factors may improve the performance further, which is out of the scope of this paper.

## 6 Conclusion and Future Work

In this paper, we have provided theoretical and empirical results that investigate the changes in the input geometry of a neural network during training. Based on these results, we proposed the Geometric Invariance Hypothesis, which argues that depending on the architecture, the changes in the input geometry of the model can be limited to a small subspace of the input space. After empirically supporting this hypothesis, we provided several experimental pieces of evidence to show the impact of geometric invariance on the generalization of deep neural networks. Considering the practical impacts of GIH, as discussed in Section [5](#S5), we expect our paper to provide valuable information to the study of inductive biases in neural networks, the role of data and its conditioning on generalization, and architecture design.

For future work, we point to several possible extensions of our results. Firstly, we note that the theoretical results provided can be applied to more complex architectures. Secondly, other optimization methods and their implications for GIH can be considered. Thirdly, we note the possible impact of GIH on other subfields of machine learning, such as AutoML and neural architecture search, the student-teacher framework, and foundation models as another important issue to consider.

## Acknowledgment

We would like to thank Hadi Daneshmand for the helpful discussions and comments. We also thank Mostafa Dehghani for his contributions to this work in its earlier phase. Antonio Orvieto acknowledges the financial support of the Hector Foundation.

## Appendix A Appendix

**Table 2: The test accuracy of LeNet, ResNet18 without batch normalization, and ViT without layer normalization for the data distribution experiment. We report results for CIFAR-10, with datapoints eliminated randomly (R) or based on their score value (S). We report the mean performance along with a $68\%$ confidence interval for each setting. The best performance is boldface.**
| Model | Percentage of Removed Data (%) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $10$ | $20$ | $30$ | $40$ | $50$ | $60$ | $70$ | $80$ | $90$ |
| LeNet-R | $65.0\pm 0.4$ | $63.95\pm 0.3$ | $61.7\pm 0.4$ | $58.9\pm 0.4$ | $56.6\pm 0.3$ | $54.8\pm 0.3$ | $52.4\pm 0.3$ | $49.5\pm 0.2$ | $45.1\pm 0.3$ |
| LeNet-S | $\mathbf{65.9\pm 0.5}$ | $\mathbf{65.1\pm 0.6}$ | $\mathbf{62.5\pm 0.2}$ | $\mathbf{60.9\pm 0.46}$ | $\mathbf{58.5\pm 0.4}$ | $\mathbf{56.4\pm 0.1}$ | $\mathbf{53.5\pm 0.3}$ | $\mathbf{50.8\pm 0.2}$ | $\mathbf{45.9\pm 0.2}$ |
| ResNet18-R | $78.6\pm 0.2$ | $76.8\pm 0.3$ | $75.8\pm 0.6$ | $73.9\pm 0.3$ | $71.8\pm 0.4$ | $68.8\pm 0.1$ | $65.1\pm 0.3$ | $60.3\pm 0.5$ | $52.4\pm 0.4$ |
| ResNet18-S | $\mathbf{79.6\pm 0.2}$ | $\mathbf{78.4\pm 0.3}$ | $\mathbf{77.0\pm 0.2}$ | $\mathbf{75.9\pm 0.3}$ | $\mathbf{73.2\pm 0.4}$ | $\mathbf{70.0\pm 0.3}$ | $\mathbf{66.2\pm 0.3}$ | $\mathbf{61.5\pm 0.4}$ | $\mathbf{54.4\pm 0.2}$ |
| ViT-R | $62.0\pm 0.1$ | $60.0\pm 0.2$ | $57.5\pm 0.1$ | $54.6\pm 0.1$ | $51.5\pm 0.1$ | $48.7\pm 0.0$ | $45.5\pm 0.1$ | $41.3\pm 0.1$ | $35.6\pm 0.1$ |
| ViT-S | $\mathbf{62.9\pm 0.1}$ | $\mathbf{60.6\pm 0.1}$ | $\mathbf{58.4\pm 0.1}$ | $\mathbf{55.4\pm 0.1}$ | $\mathbf{52.2\pm 0.2}$ | $\mathbf{49.7\pm 0.1}$ | $\mathbf{46.5\pm 0.1}$ | $\mathbf{42.1\pm 0.0}$ | $\mathbf{36.2\pm 0.1}$ |

Figure: (a) Theorem [3.1](#S3.Thmtheorem1) - MLP
Refer to caption: /html/2410.12025/assets/x22.png

### A.1 Related Works

Since the seminal work of Zhang et al. (), there has been a lot of research aimed at reconciling the classical notion of bias-variance hypothesis and the uniform convergence theory with the success of deep neural networks Neyshabur et al. (); Nagarajan & Kolter (); Belkin et al. (); Kalimeris et al. (); Bachmann et al. (); Chiang et al. (). Regarding this issue, there are two schools of thoughts that either attribute this observation to an inductive bias introduced by the gradient-based optimization methods used in deep learning Neyshabur et al. (); Barrett & Dherin (); Smith et al. (), or the “volume hypothesis” arguing for the inherently larger volume of “good” solutions that generalize well Chiang et al. (). However, there is one certainty with regards to this issue: the existence of “inductive biases” in these networks that result in a large decrease in the hypothesis class of solutions provided by the model, causing generalization in practice.

In search of these inductive biases, some research has been focused on the optimization of neural networks Neyshabur et al. (); Arora et al. (); Guille-Escuret et al. (), which inevitably deals with the geometry of these models in the parameter space Arpit et al. (). On the other hand, the introduction of neural tangent kernels allowed a link to be established between the rich literature of kernel methods Bartlett & Mendelson () and the asymptotic behavior of neural networks at infinite width Jacot et al. (). This also gave rise to some research concerned with inductive biases of deep models in the kernel space, which is also mainly focused on the geometry of parameter space Jacot et al. (); Bietti & Mairal (); Ortiz-Jiménez et al. (). However, there are also some research on the spectral inductive biases of neural networks, which provide interesting but intangible results for the type of frequencies used by these models in practice Basri et al. (); Wen & Jacot ().

The limited research on the geometry of deep neural networks in the input space is our main motivation for this work. The closest line of research to our paper are the simplicity bias hypothesis (SBH) Pérez et al. (), and the neural anisotropy directions Ortiz-Jiménez et al. (). The SBH Arpit et al. (); Pérez et al. (); Kalimeris et al. () has become a cornerstone of deep learning theory, with implication for generalization and robustness of deep neural networks Kalimeris et al. (); Shah et al. (). Specifically, SBH argues that neural networks learn functions of increasing complexity from the data. Therefore, in the presence of multiple solutions to the problem with various complexities, deep neural networks opt for the simplest one. In this paper, we see that while SBH is still a valid hypothesis for the inductive biases of neural networks, the geometry of the model in the input space - which is determined by its architecture - also plays a role in the“preference” of deep neural networks for simpler functions. Meaning that, depending on the architecture, there are more complex solutions that may be learned by the model before simpler ones. This behavior is due to the geometry of the model remaining invariant in certain directions of the input space.
We dub this behavior geometric invariance, and our hypothesis that introduces this behavior as a geometric inductive bias of neural networks the Geometric Invariance Hypothesis (GIH).

The neural anisotropy directions (NADs) as defined in Ortiz-Jiménez et al. () touch on the concept of geometric invariance for training on linearly separable datasets. Specifically, Ortiz-Jiménez et al. () observe that in the case of linearly separable data in specific direction $\mathbf{u}$, depending on the architecture, neural networks are incapable of generalizing when $\mathbf{u}$ has a low correlation with the covariance of the gradient of the model w.r.t. input at initialization. This observation can be seen as a specific version of GIH limited to the linearly separable data. However, unlike our paper which motivates GIH from the perspective of investigating the geometry of a neural network, Ortiz-Jiménez et al. () explain their observations regarding GIH through the concept of discriminative dipoles, which corresponds to a pair of data points residing on the opposite sides of a linear decision boundary. Therefore, given that our paper does not make assumptions about the task, GIH can be seen as a generalization of the concept of NADs.

### A.2 Sum of squared error loss and its dynamics

Following related works Jacot et al. (); Lee et al. (); Arora et al. () we use the sum of squared error (SSE) loss between the labels and the prediction of the model for the simplicity of analysis. However, as we see in the experiments, in practice our analysis holds for cross-entropy loss as well. The SSE loss can be written as:

$$ $\mathcal{L}(\theta)=\frac{1}{2}\sum_{\mu=1}^{m}\left(f_{\theta}(\mathbf{x}_{\mu})-y_{\mu}\right)^{2}.$ (4) $$

In this case, we can write the dynamics of the parameter as:

$$ $\dot{\theta}=-\sum_{\mu=1}^{m}\nabla_{\theta}f_{\theta}(\mathbf{x}_{\mu})(f_{\theta}(\mathbf{x}_{\mu})-y_{\mu}).$ (5) $$

### A.3 Loss curvature in the input space

An important source of information about the geometry of a neural network in the input space is the Hessian of the loss w.r.t. the input, which contains information about the curvature. Let $\nabla_{\mathbf{x}}^{2}$ be the Hessian operator. Using the chain rule on $\ell(f_{\theta}(\mathbf{x}),y)$, we can write:

$$ $\nabla_{\mathbf{x}}^{2}\ell(f_{\theta}(\mathbf{x}),y)=\frac{\partial\ell}{\partial f}\left(f_{\theta}\left(\mathbf{x}\right),y\right)\cdot\nabla_{\mathbf{x}}^{2}f_{\theta}(\mathbf{x})+\frac{\partial^{2}\ell}{\partial f^{2}}(f_{\theta}(\mathbf{x}),y)\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})^{\top}.$ $$

We can factorize the two components determining the curvature of the loss in the input space in two: 1) global model curvature (Hessian, for scalar $f_{\theta}(\mathbf{x})$), and 2) local model curvature (gradient outer-product). Assuming that we’re in the over-parameterized setting with $f_{\theta}(\cdot)$ belonging to a family of neural networks, we can expect the magnitude of the first derivative of the loss function w.r.t. the model to reach near 0 0 at some point during the training procedure. Therefore, starting at some point in the training, we continue to have:

$$ $\nabla_{\mathbf{x}}^{2}\ell\left(f_{\theta}(\mathbf{x}),y\right)\approx\frac{\partial^{2}\ell}{\partial f^{2}}(f_{\theta}(\mathbf{x}),y)\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})^{\top}.$ (6) $$

So we can look at the gradient outer-product of the model as a low-rank approximation of the geometry of the model in the input space(^7^77Note that this is very similar to the Gauss-Newton approximation of the loss Hessian w.r.t. the parameters, which is extensively used in optimization Martens (). However, here we are using this approximation solely to understand the geometry of the model in the input space.). In order to isolate the effect of architecture and other non-stochastic components in the model (e.g. the optimization method, initialization method, etc.) that influence the “geometric inductive biases” of the model, we can marginalize the effect of the parameters out.
Assuming that the second-derivative of the loss function w.r.t. the model output has a small variation over $\mathcal{T}_{t}$(^8^88This can happen, for instance, when we have the SSE loss where the second-derivative is constant.), then we can write:

$$ $\mathbb{E}_{\theta\sim\mathcal{T}_{t}}\left[\nabla_{\mathbf{x}}^{2}\ell\left(f_{\theta}(\mathbf{x}),y\right)\right]\propto\mathbf{G}_{\mathcal{F}}^{t}(x).$ $$

Therefore, by looking at the average geometry at point $\mathbf{x}$ and time $t$ through $\mathbf{G}_{\mathcal{F}}^{t}(\mathbf{x})$, we will have an increasingly accurate approximation of the loss curvature in the input at point $\mathbf{x}$ and time $t$ up to a constant. Given a probing distribution $\mathcal{P}$, we can consider $\mathbb{E}_{\mathbf{x}\sim\mathcal{P},\theta\sim\mathcal{T}_{t}}\left[\nabla_{\mathbf{x}}^{2}\ell\left(f_{\theta}(\mathbf{x}),y\right)\right]$ as the average geometry, which we estimate by $\mathbf{G}_{\mathcal{F},\mathcal{P}}^{t}=\mathbb{E}_{x\sim\mathcal{P}}\left[\mathbf{G}_{\mathcal{F}}^{t}(\mathbf{x})\right]$.

Based on this approximation of the geometry in the input space, we will formulate a function informing us about the changes in the geometry as well. Let us start with the derivative of ([6](#A1.E6)) w.r.t. time in order to get the changes in the loss curvature through the gradient flow model:

$$ $\frac{\text{d}\nabla_{\mathbf{x}}^{2}\ell}{\text{d}t}\left(f_{\theta}(\mathbf{x}),y\right)\approx\frac{\partial^{3}\ell}{\partial f^{3}}(f_{\theta}(\mathbf{x}),y)\cdot\left(\nabla_{\theta}f_{\theta}(\mathbf{x})^{\top}\dot{\theta}\right)\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})^{\top}+\frac{\partial^{2}\ell}{\partial f^{2}}(f_{\theta}(\mathbf{x}),y)\nabla_{\mathbf{x},\theta}^{2}f_{\theta}(\mathbf{x})\dot{\theta}\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})^{\top}+\frac{\partial^{2}\ell}{\partial f^{2}}(f_{\theta}(\mathbf{x}),y)\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})\dot{\theta}^{\top}\nabla_{\mathbf{x},\theta}f_{\theta}(\mathbf{x})^{\top}.$ $$

Assuming that the third-derivative of the loss function w.r.t. the model output is close to zero(^9^99For instance, in the cross entropy loss the third derivative will be equal to the second derivative of the softmax function, which is bounded by $1/8$, while it is equal to zero in the SSE loss.), and assuming the independence of $\mathcal{T}_{t}$ and the second derivative of the loss, we can get the expected value of the two sides w.r.t. the parameters to isolate the effect of inductive biases, which gives us:

$$ $\mathbb{E}_{\theta\sim\mathcal{T}_{t}}\left[\frac{\text{d}\nabla_{\mathbf{x}}^{2}\ell}{\text{d}t}\left(f_{\theta}(\mathbf{x}),y\right)\right]\propto\Delta_{\mathcal{F}}^{t}(\mathbf{x}).$ $$

Therefore, based on the loss curvature evolution function and given a probing distribution $\mathcal{P}$, we define the average geometry evolution as $\mathbb{E}_{\mathbf{x}\sim\mathcal{P},\theta\sim\mathcal{T}_{t}}\left[\frac{\text{d}\nabla_{\mathbf{x}}^{2}\ell}{\text{d}t}\left(f_{\theta}(\mathbf{x}),y\right)\right]$, which we estimate as $\Delta_{\mathcal{F},\mathcal{P}}^{t}$ up to a constant.

### A.4 Independence of Δ ℱ ​ ( 𝐱 ) subscript Δ ℱ 𝐱 \Delta_{\mathcal{F}}(\mathbf{x}) from labels

Figure: (a) MLP
Refer to caption: /html/2410.12025/assets/x24.png

In this section, we try to investigate the dependence between the average geometry evolution at initialization $\Delta_{\mathcal{F}}$ and the train labels $y_{\mu}$, i.e., the relationship between the changes in the geometry at initialization and the task. Firstly, we provide the following proposition to show that at the first gradient descent step, this independence can be proved provided that the model has a single-layer perceptron for its classification layer and is trained via SSE loss.

###### Proposition A.1 .

Let $\mathcal{F}$ be a family of neural networks with a single layer perceptron initialized i.i.d. from a zero-mean Gaussian as the classification layer, trained via SSE. Then $\Delta_{\mathcal{F}}(\mathbf{x})$ is independent of train labels.

Proof. Let $f_{\theta}(\mathbf{x})$ be the model defined by the proposition:

$$ $f_{\theta}(\mathbf{x})=\sum_{i=1}^{n}\omega_{i}h_{\phi}^{i}(\mathbf{x}),$ $$

where $h_{\phi}(\mathbf{x})$ is the hidden representation in the penultimate layer of the model, $h_{\phi}^{i}(\mathbf{x})$ is its $i^{th}$ element, and $\omega$ corresponds to the weights of the classification layer. Therefore, $\theta$ corresponds to the concatenation of $\omega$ and $\phi$. The required derivatives are:

$$ $\displaystyle\nabla_{x}f_{\theta}(\mathbf{x})$ $\displaystyle=\sum_{i=1}^{n}\omega_{i}\nabla_{x}h_{\phi}^{i}(\mathbf{x}),$ $\displaystyle\frac{\partial f_{\theta}}{\partial\omega_{i}}(\mathbf{x})$ $\displaystyle=h_{\phi}^{i}(\mathbf{x}),$ $\displaystyle\frac{\partial\nabla_{\mathbf{x}}f_{\theta}}{\partial\omega_{i}}(\mathbf{x})$ $\displaystyle=\nabla_{\mathbf{x}}h_{\phi}^{i}(\mathbf{x}),$ $\displaystyle\nabla_{\phi}f_{\theta}(\mathbf{x})$ $\displaystyle=\sum_{i=1}^{n}\omega_{i}\nabla_{\phi}h_{\phi}^{i}(\mathbf{x}),$ $\displaystyle\nabla_{\mathbf{x},\phi}f_{\theta}(\mathbf{x})$ $\displaystyle=\sum_{i=1}^{n}\omega_{i}\nabla_{\mathbf{x},\phi}h_{\phi}^{i}(\mathbf{x}).$ $$

So we can write $\Delta_{\mathcal{F}}(\mathbf{x})=\mathbf{A}_{\mathcal{F}}(\mathbf{x})+\mathbf{A}_{\mathcal{F}}(\mathbf{x})^{\top}$, where we have:

$$ $\mathbf{A}_{\mathcal{F}}(\mathbf{x})=-\sum_{\mu=1}^{m}\left(\sum_{i_{1}=1}^{n}\sum_{i_{3}=1}^{n}\sum_{i_{4}=1}^{n}\mathbb{E}_{\theta}\left[\omega_{i_{3}}h_{\phi}^{i_{1}}(\mathbf{x}_{\mu})\left(\omega_{i_{4}}h_{\phi}^{i_{4}}(\mathbf{x}_{\mu})-y_{\mu}\right)\nabla_{\mathbf{x}}h_{\phi}^{i_{1}}(\mathbf{x})\nabla_{\mathbf{x}}h_{\phi}^{i_{3}}(\mathbf{x})^{\top}\right]+\mathbb{E}_{\theta}\left[\omega_{i_{1}}\omega_{i_{2}}\omega_{i_{3}}\left(\omega_{i_{4}}h_{\phi}^{i_{4}}(\mathbf{x}_{\mu})-y_{\mu}\right)\nabla_{\mathbf{x},\phi}h_{\phi}^{i_{1}}(\mathbf{x})\nabla_{\phi}h_{\phi}^{i_{2}}(\mathbf{x}_{\mu})\nabla_{\mathbf{x}}h_{\phi}^{i_{3}}(\mathbf{x})^{\top}\right]\right).$ (7) $$

Note that the terms related to $y_{\mu}$ in ([7](#A1.E7)) contain an odd number of $\omega_{i}$s. Considering that $\omega$ is i.i.d. and zero-mean Gaussian, its odd momentums are equal to zero, which means these terms will be eliminated. This completes our proof.∎

As we can observe from Proposition [A.1](#A1.Thmtheorem1), $\Delta_{\mathcal{F}}$ is provably independent of the task and only depends on inputs. In order to show that this behavior remains consistent at the initial stages of training, we formulate an experiment similar to Section [3](#S3). Specifically, we try to show that even if there’s nothing to learn from the data, and the task is fully about the “memorization” of the data, $\Delta_{\mathcal{F}}$ still follows the behavior outline in Conjecture [1](#Thmconjecture1). So we relabel the CIFAR-2 dataset introduced in Section [3](#S3) using a uniform random distribution over $\{-1,1\}$. We then report the correlation between $\mathbf{G}_{\mathcal{F}}^{t}$ and $\mathbf{G}_{\mathcal{F}}\mathbf{S}\mathbf{G}_{\mathcal{F}}$ along with the training accuracy for each epoch. The results are visualized in Figure [8](#A1.F8). We point out that these results clearly indicate the universality of our theoretical motivations, meaning that the behavior outlined in the theorems in Section [3](#S3) culminating in the introduction of Conjecture [1](#Thmconjecture1) are independent of the task.

###### Proposition A.1 .

### A.5 The structure of 𝐆 ℱ subscript 𝐆 ℱ \mathbf{G}_{\mathcal{F}}

In this section we try to investigate how the architecture affects $\mathbf{G}_{\mathcal{F}}$. First, we will present the following theorem for the case where $\mathcal{F}$ corresponds to an MLP or CNN with ReLU non-linearity and without pooling layers or skip-connections:

###### Theorem A.2 .

Let $\mathcal{F}$ be the family of MLPs or CNNs with ReLU non-linearity, without pooling layers or skip-connections. Then we have $\mathbf{G}_{\mathcal{F}}(\mathbf{x})=c\cdot\mathbf{I}$ for some constant $c$. For the proof, please refer to Appendix [A.8.4](#A1.SS8.SSS4).

As we can observe in Theorem [A.2](#A1.Thmtheorem2), in the case where $\mathcal{F}$ does not contain any pooling, self-attention, or skip-connections, the matrix $\mathbf{G}_{\mathcal{F}}$ will correspond to the identity matrix, and therefore, is a full rank matrix.

However, this is not the case in other types of architecture components. For instance, the following theorem will show that the introduction of an average pooling layer will introduce a structure to $\mathbf{G}_{\mathcal{F}}$:

###### Theorem A.3 .

Let $\mathcal{F}$ be the family of CNNs with a linear convolution layer and a single kernel, followed by a batch normalization layer and a global average pooling layer. Also, let $\mathbf{M}_{j}$ correspond to a binary matrix that simulates the convolutional operation for the $j^{th}$ filter. Assuming we use SSE loss, we have:

$$ $\mathbf{G}_{\mathcal{F}}\propto\sum_{j_{1}}\sum_{j_{2}}\mathbf{I}_{j_{1},j_{2}},$ $$

where $\mathbf{I}_{j_{1},j_{2}}$ is a diagonal binary matrix corresponding to the overlap of the $j_{1}^{th}$ and $j_{2}^{th}$ patch of the data. For the proof, please refer to Appendix [A.8.5](#A1.SS8.SSS5).

From Theorem [A.3](#A1.Thmtheorem3) we observe that $\mathbf{G}_{\mathcal{F}}$ will have a diagonal form, with each diagonal element proportional to how many times the corresponding input element has been in the receptive field of the convolution filter. In the case of batch normalization, the structure of $\mathbf{G}_{\mathcal{F}}$ can be potentially even more complex:

###### Corollary A.4 .

Let $\mathcal{F}$ be the family of CNNs with a linear convolution layer and a single kernel, followed by a batch normalization layer and a global average pooling layer. Also, let $\mathbf{M}_{j}$ correspond to a binary matrix that simulates the convolutional operation for the $j^{th}$ filter. Assuming that the train data is zero-mean and we use SSE loss, then we have:

$$ $\mathbf{G}_{\mathcal{F}}\propto\sum_{j_{1}}\sum_{j_{2}}\mathbf{M}_{j_{1}}\mathbb{E}_{\theta}\left[\frac{\theta\theta^{\top}}{\sigma_{j_{1}}\left(\theta\right)\cdot\sigma_{j_{2}}\left(\theta\right)}\right]\mathbf{M}_{j_{2}}^{\top},$ $$

where $\sigma_{j}\left(\theta\right)=\mathbb{V}\mathbbm{ar}_{\mathbf{x}_{\mu}\in\mathcal{D}_{T}}(\theta^{\top}\mathbf{M}_{j}^{\top}x_{\mu})$. Therefore, the terms in $\mathbf{G}_{\mathcal{F}}$ corresponding to each pair of indices $j_{1},j_{2}$ is bounded as:

$$ $\frac{1}{\sqrt{\lambda_{min}^{j_{1}}\lambda_{min}^{j_{2}}}}\mathbf{I}_{j_{1},j_{2}}\succeq\mathbf{M}_{j_{1}}\mathbb{E}_{\theta}\left[\frac{\theta\theta^{\top}}{\sigma_{j_{1}}\left(\theta\right)\cdot\sigma_{j_{2}}\left(\theta\right)}\right]\mathbf{M}_{j_{2}}^{\top}\succeq\frac{1}{\sqrt{\lambda_{max}^{j_{1}}\lambda_{max}^{j_{2}}}}\mathbf{I}_{j_{1},j_{2}},$ $$

where $\lambda_{min}^{j},\lambda_{max}^{j}$ are the smallest and largest eigenvalues of the covariance of the $j^{th}$ patch of the data. For the proof, please refer to Appendix [A.8.6](#A1.SS8.SSS6).

The Corollary [A.4](#A1.Thmtheorem4) indicates that introducing normalization to the model will result in a re-weighting of the influence of the input elements based on their variance onto $\mathbf{G}_{\mathcal{F}}$, atop the receptive field-related structure introduced by Theorem [A.3](#A1.Thmtheorem3). In our experience, in cases where the covariance of the training data has a structure, this will result in the normalization of $\mathbf{G}_{\mathcal{F}}$ w.r.t. each patch of data which depends on the kernel size in CNNs and the patch size in ViT.

Therefore, we can see that certain architecture components can introduce structure to $\mathbf{G}_{\mathcal{F}}$, which can reduce the ranking of $\mathbf{G}_{\mathcal{F}}$. This indicates a significant narrowing of the geometric inductive biases of the model on the input space, as considered by this paper, which as we observed in Figure [4](#S4.F4), results in the model being unable to generalize in tasks wherein the decision boundary lies in directions with low correlation with $\mathbf{G}_{\mathcal{F}}$.

###### Theorem A.2 .

###### Theorem A.3 .

###### Corollary A.4 .

### A.6 Visualization of the average geometry

Figure: (a) MLP
Refer to caption: /html/2410.12025/assets/figures/G_mlp.jpg

In this section, we provide a visualization of the average geometry at initialization, i.e., $\mathbf{G}_{\mathcal{F}}$ for various architectures. As we can observe in Figure [9](#A1.F9), an MLP has a full-rank average geometry at initialization, with the matrix $\mathbf{G}_{\mathcal{F}}$ resembling the identity matrix. On the other hand, the eigenvalues of the other architecture have a much wider range, clearly indicating a presence of structure in the average geometry.

### A.7 GIH and generalization gap with isotropic data

Figure: (a) MLP
Refer to caption: /html/2410.12025/assets/x28.png

In this section, we try to isolate the effect of GIH over the generalization gap by making the data covariance as close to “isotropic” as possible. Specifically, we design the following experiment involving synthetic data formulated as follows:

$$ $\mathbf{x}=\epsilon\alpha\mathbf{u}+\omega,\quad\omega\sim\mathcal{N}\left(0,\sigma^{2}\left(\mathbf{I}-\mathbf{u}\mathbf{u}^{\top}\right)\right),$ (8) $$

where we have $\alpha\sim\mathcal{N}\left(0,1\right)$, and we consider $\mathbf{u}$ to be the discriminant feature, similar to Section [4.2](#S4.SS2). Note that in this setting, for $\epsilon\approx\sigma$ we can roughly say $\mathbf{x}\sim\mathcal{N}\left(0,\sigma^{2}\mathbf{I}\right)$, which eliminates the effect of data over the changes in geometry. Similar to the experiment in Section [4.2](#S4.SS2), we assess the generalization ability of the model for varying complexity of tasks: linear as $y=\textbf{sgn}((\mathbf{x}^{\top}\mathbf{u})+\mathbf{b})$, quadratic as $y=\textbf{sgn}((\mathbf{x}^{\top}\mathbf{u})^{2}+\mathbf{b})$, and sinusoidal as $y=\textbf{sgn}(\sin(\bar{\mathbf{x}}^{\top}\mathbf{u})+\mathbf{b})$. Note that we can view the ratio $\epsilon/\sigma$ as a measure of the margin between the two classes, which controls the difficulty of the problem. In Figure [10](#A1.F10), you can observe the test accuracy of the models for these tasks, with $\mathbf{u}$ selected from the eigenvectors of $\mathbf{G}_{\mathcal{F}}$ in descending order of the eigenvalues.

As we can observe, in the absence of a structure in the data covariance, the MLP model will be completely devoid of any geometric inductive bias, achieving generalization in all directions with similar equality. On the other hand, in the case of the models with structured initial geometry, the performance of the model in all tasks in correlated with the eigenvalue of the corresponding discriminant feature. These results provide further support for the GIH, while also confirming that the observation in Section [4.2](#S4.SS2) is due to the existence of a structure in both the data covariance and the initial geometry.

### A.8 Proof of theorems

#### A.8.1 Lemmas

We will start the proofs by first proving four lemmas that we will be using in our proofs.

###### Lemma A.5 .

Let $\delta\sim\mathcal{N}(0,\mathbf{I})$ be a standard Gaussian variable, and $\mathbf{a},\mathbf{b}\in\mathbb{R}^{D}$ be two unit vectors. Then, as $D$ becomes larger ($D\to\infty$), and assuming $\mathbf{a},\mathbf{b}$ are non-sparse (i.e., their magnitude is not concentrated on a few elements), we have:

$$ $\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\delta\delta^{\top}\right]\approx\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\right]\cdot\mathbf{I}_{D},$ $$

which becomes accurate at the rate of $\sqrt{D}$.

Proof. We will start by partitioning the variability into three factors: $\delta=\alpha\cdot\mathbf{a}+\beta\cdot\mathbf{b}_{\perp}+\delta_{\perp}$, where $\mathbf{b}_{\perp}$ is the unit vector in the direction of the orthogonal projection of $\mathbf{b}$ on $\mathbf{a}$, and $\delta_{\perp}\sim\mathcal{N}\left(0,\mathbf{I}_{D}-\mathbf{a}\mathbf{a}^{\top}-\mathbf{b}_{\perp}\mathbf{b}_{\perp}^{\top}\right)$. Then we can write:

$$ $\displaystyle\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\delta\delta^{\top}\right]$ $\displaystyle=\left(\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot\mathbf{a}^{\top}\mathbf{b}_{\perp}+\beta\cdot\mathbf{b}^{\top}\mathbf{b}_{\perp}>0}\alpha^{2}\right]-\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot\mathbf{a}^{\top}\mathbf{b}_{\perp}+\beta\cdot\mathbf{b}^{\top}b_{\perp}>0}\right]\right)\mathbf{a}\mathbf{a}^{\top}$ $\displaystyle+\left(\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot\mathbf{a}^{\top}\mathbf{b}_{\perp}+\beta\cdot\mathbf{b}^{\top}\mathbf{b}_{\perp}>0}\beta^{2}\right]-\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot\mathbf{a}^{\top}\mathbf{b}_{\perp}+\beta\cdot\mathbf{b}^{\top}\mathbf{b}_{\perp}>0}\right]\right)\mathbf{b}_{\perp}\mathbf{b}_{\perp}^{\top}$ $\displaystyle+\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot\mathbf{a}^{\top}\mathbf{b}_{\perp}+\beta\cdot\mathbf{b}^{\top}\mathbf{b}_{\perp}>0}\alpha\beta\right]\left(\mathbf{b}_{\perp}\mathbf{a}^{\top}+\mathbf{a}\mathbf{b}_{\perp}^{\top}\right)$ $\displaystyle+\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot\mathbf{a}^{\top}\mathbf{b}_{\perp}+\beta\cdot\mathbf{b}^{\top}\mathbf{b}_{\perp}>0}\right]\mathbf{I}_{D}.$ $$

Now, we will upper-bound the coefficients of the first three terms. For the coefficient of the first term, we can write:

$$ $\displaystyle=\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot c+\beta>0}\alpha^{2}\right]-\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot c+\beta>0}\right]$ (9) $\displaystyle=\frac{1}{2}\cdot\left(\mathbb{E}\left[\mathbf{1}_{\alpha>0}\left(\text{erf}\left(\frac{1}{\sqrt{2}}\alpha\cdot c\right)+1\right)\alpha^{2}\right]-\mathbb{E}\left[\mathbf{1}_{\alpha>0}\left(\text{erf}\left(\frac{1}{\sqrt{2}}\alpha\cdot c\right)+1\right)\right]\right)$ (10) $\displaystyle=\frac{1}{2}\cdot\left(\mathbb{E}\left[\mathbf{1}_{\alpha>0}\text{erf}\left(\frac{1}{\sqrt{2}}\alpha\cdot c\right)\alpha^{2}\right]-\mathbb{E}\left[\mathbf{1}_{\alpha>0}\text{erf}\left(\frac{1}{\sqrt{2}}\alpha\cdot c\right)\right]\right)$ (11) $\displaystyle=\frac{1}{\pi}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\exp\left(-\frac{1}{\sqrt{2}}\alpha^{2}\cdot c^{2}\right)\right],$ (12) $$

where $c=\frac{\mathbf{a}^{\top}\mathbf{b}_{\perp}}{\mathbf{b}^{\top}\mathbf{b}_{\perp}}$, ([10](#A1.E10)) comes from the definition of indicator function and the error function $\text{erf}(\cdot)$, ([11](#A1.E11)) comes from $\alpha$ being a standard Gaussian variable which gives us $\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha^{2}\right]=\mathbb{E}\left[\mathbf{1}_{\alpha>0}\right]=\frac{1}{2}$, and ([12](#A1.E12)) comes from the following:

$$ $\displaystyle\mathbb{E}\left[\mathbf{1}_{\alpha>0}\text{erf}\left(\frac{\alpha\cdot c}{\sqrt{2}}\right)\alpha^{2}\right]$ $\displaystyle=\frac{1}{\sqrt{2\pi}}\int_{0}^{+\infty}\text{erf}\left(\frac{\alpha\cdot c}{\sqrt{2}}\right)\alpha^{2}\exp\left(-\frac{\alpha^{2}}{2}\right)~{}\text{d}\alpha$ (13) $\displaystyle=-\frac{1}{\sqrt{2\pi}}\int_{0}^{+\infty}\text{erf}\left(\frac{\alpha\cdot c}{\sqrt{2}}\right)\alpha~{}\text{d}\exp\left(-\frac{\alpha^{2}}{2}\right)$ (14) $\displaystyle=\frac{1}{\sqrt{2\pi}}\int_{0}^{+\infty}\left(\text{erf}\left(\frac{\alpha\cdot c}{\sqrt{2}}\right)+\frac{2}{\sqrt{\pi}}\alpha\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{\sqrt{2}}\right)\right)\exp\left(-\frac{\alpha^{2}}{2}\right)~{}\text{d}\alpha$ (15) $\displaystyle=\mathbb{E}\left[\mathbf{1}_{\alpha>0}\text{erf}\left(\frac{\alpha\cdot c}{\sqrt{2}}\right)\right]+\frac{2}{\sqrt{2\pi}}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{\sqrt{2}}\right)\right],$ (16) $$

with ([14](#A1.E14)) resulting from $\alpha\exp\left(-\frac{\alpha^{2}}{2}\right)~{}\text{d}\alpha=-\text{d}\exp\left(-\frac{\alpha^{2}}{2}\right)$, and ([15](#A1.E15)) resulting from integration by parts.

Similarly, for the coefficient of the second term we can write:

$$ $\displaystyle=\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot c+\beta>0}\beta^{2}\right]-\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot c+\beta>0}\right]$ (17) $\displaystyle=\frac{1}{\sqrt{2\pi}}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\cdot c\cdot\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{2}\right)\right]+\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot c+\beta>0}\right]-\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot c+\beta>0}\right]$ (18) $\displaystyle=\frac{1}{\sqrt{2\pi}}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\cdot c\cdot\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{2}\right)\right],$ (19) $$

where ([18](#A1.E18)) comes from:

$$ $\displaystyle\mathbb{E}_{\beta}\left[\mathbf{1}_{\alpha\cdot c+\beta>0}\beta^{2}\right]$ $\displaystyle=\frac{1}{\sqrt{2\pi}}\int_{-\alpha\cdot c}^{+\infty}\beta^{2}\exp\left(-\frac{\beta^{2}}{2}\right)~{}\text{d}\beta$ (20) $\displaystyle=-\frac{1}{\sqrt{2\pi}}\int_{-\alpha\cdot c}^{+\infty}\beta~{}\text{d}\exp\left(-\frac{\beta^{2}}{2}\right)$ (21) $\displaystyle=-\frac{1}{\sqrt{2\pi}}\left(-\alpha\cdot c\cdot\exp\left(-\frac{\beta^{2}\cdot c^{2}}{2}\right)-\int_{-\alpha\cdot c}^{+\infty}\exp\left(-\frac{\beta^{2}}{2}\right)~{}\text{d}\beta\right)$ (22) $\displaystyle=\frac{1}{\sqrt{2\pi}}\alpha\cdot c\cdot\exp\left(-\frac{\beta^{2}\cdot c^{2}}{2}\right)+\mathbb{E}_{\beta}\left[\mathbf{1}_{\alpha\cdot c+\beta>0}\right].$ (23) $$

And for the third term, we have:

$$ $\mathbb{E}\left[\mathbf{1}_{\alpha>0}\mathbf{1}_{\alpha\cdot c+\beta>0}\alpha\beta\right]=\frac{1}{\sqrt{2\pi}}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\cdot\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{2}\right)\right],$ $$

which comes from:

$$ $\displaystyle\mathbb{E}_{\beta}\left[\mathbf{1}_{\alpha\cdot c+\beta>0}\beta\right]$ $\displaystyle=\frac{1}{\sqrt{2\pi}}\int_{-\alpha\cdot c}^{+\infty}\beta\cdot\exp\left(-\frac{\beta^{2}}{2}\right)~{}\text{d}\beta$ $\displaystyle=-\frac{1}{\sqrt{2\pi}}\int_{-\alpha\cdot c}^{+\infty}~{}\text{d}\exp\left(-\frac{\beta^{2}}{2}\right)$ $\displaystyle=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{2}\right).$ $$

For ([12](#A1.E12)), we have:

$$ $\displaystyle\frac{1}{\pi}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\exp\left(-\frac{1}{\sqrt{2}}\alpha^{2}\cdot c^{2}\right)\right]$ $\displaystyle\leq\frac{1}{\pi}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\right]$ $\displaystyle=\frac{1}{\pi\sqrt{2\pi}},$ $$

which comes from $\exp\left(-\alpha^{2}\cdot c^{2}\right)\leq 1$. For ([19](#A1.E19)), note that the maximum value of $c\cdot\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{2}\right)$ happens at $\alpha^{2}\cdot c^{2}=1$, which gives us:

$$ $\displaystyle\frac{1}{\sqrt{2\pi}}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\cdot c\cdot\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{2}\right)\right]$ $\displaystyle\leq\frac{1}{\sqrt{2\pi}}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\exp\left(-\frac{1}{2}\right)\right]$ $\displaystyle=\frac{\exp\left(-\frac{1}{2}\right)}{2\sqrt{2\pi}}.$ $$

And for ([A.8.1](#A1.Ex19)), we have:

$$ $\displaystyle\frac{1}{\sqrt{2\pi}}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\cdot\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{2}\right)\right]$ $\displaystyle\leq\frac{1}{\sqrt{2\pi}}\mathbb{E}\left[\mathbf{1}_{\alpha>0}\alpha\right]$ $\displaystyle=\frac{1}{2\pi},$ $$

which also comes from $\exp\left(-\frac{\alpha^{2}\cdot c^{2}}{2}\right)\leq 1$.

So we can see that the coefficients of the first three terms are bounded. Furthermore, we can assume $\mathbf{a}\neq c^{\prime}\cdot\mathbf{b}$ for some constant $c^{\prime}\in\{+1,-1\}$, since otherwise the result would be trivial:

$$ $\displaystyle\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\delta\delta^{\top}\right]$ $\displaystyle=\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\delta\delta^{\top}\right]$ $\displaystyle=\frac{1}{2}\mathbb{E}_{\delta}\left[\delta\delta^{\top}\right]$ $\displaystyle=\frac{1}{2}\mathbf{I}_{D},$ $$

for $c^{\prime}=1$, and:

$$ $\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\delta\delta^{\top}\right]=0,$ $$

otherwise. So as $D\to\infty$, given that $\mathbf{a},\mathbf{b}$ are non-sparse and unit vectors, their magnitude will be stretched over the elements, resulting in each element of $\mathbf{a}$ and $\mathbf{b}$ approaching 0 0 at the rate of $\sqrt{D}$. Therefore, the last term will be the dominant term, which completes our proof.∎

###### Lemma A.6 .

Let $\delta\sim\mathcal{N}(0,\mathbf{I})$ be a standard Gaussian variable, and $\mathbf{a},\mathbf{b}\in\mathbb{R}^{D}$ be two vectors. Then, assuming $a\sim\mathcal{N}\left(0,\mathbf{I}_{D}\right)$, the term $\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\right]$ will be a uniform random variable w.r.t. $\mathbf{a}$, and we have:

$$ $\mathbb{E}_{\delta,\mathbf{a}}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\right]=\frac{1}{2},\quad\mathbb{E}_{\mathbf{a}}\left[\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\right]^{2}\right]=\frac{1}{3}.$ (24) $$

Proof. Note that since $\mathbf{a}$ is a standard Gaussian variable, its angle with a given vector $\mathbf{b}$ is a uniformly distributed variable. Furthermore, $\mathbb{E}_{\delta}\left[\mathbf{1}_{\delta^{\top}\mathbf{a}>0}\mathbf{1}_{\delta^{\top}\mathbf{b}>0}\right]$ corresponds to the overlap of the two halfspaces defined by $\delta^{\top}\mathbf{a}>0$ and $\delta^{\top}\mathbf{b}>0$, which is equal to the ratio of the angle between $\mathbf{a}$ and $\mathbf{b}$ in Radian, divided by $2\pi$. As a result, it is a uniform variable $\sim\text{U}\left[0,1\right]$ (i.e., uniformly distributed in $[0,1]$ range). So its expected value and second momentum are, respectively, $\frac{1}{2}$ and $\frac{1}{3}$.∎

###### Lemma A.7 .

Let $\delta\sim\mathcal{N}\left(0,\mathbf{I}_{D}\right)$ be a standard Gaussian random variable of size $D$. Then we have:

$$ $\mathbb{E}_{\delta}\left[\frac{\delta\delta^{\top}}{\|\delta\|_{2}^{2}}\right]=\frac{1}{D}\mathbf{I}_{D}.$ (25) $$

Proof. First, note that due to symmetry, the diagonal elements and the off-diagonal elements are equal in ([25](#A1.E25)). Therefore, we can get the diagonal elements by evaluating the trace:

$$ $\displaystyle\mathbf{Tr}\left(\mathbb{E}_{\delta}\left[\frac{\delta\delta^{\top}}{\|\delta\|_{2}^{2}}\right]\right)$ $\displaystyle=\mathbb{E}_{\delta}\left[\frac{\mathbf{Tr}\left(\delta\delta^{\top}\right)}{\|\delta\|_{2}^{2}}\right]$ $\displaystyle=\mathbb{E}_{\delta}\left[1\right]$ $\displaystyle=1.$ $$

This gives us diagonal elements equal to $1/D$. On the other hand, note that we can write an off-diagonal element in ([25](#A1.E25)) as:

$$ $\mathbb{E}_{\delta_{i},\delta_{j}}\left[\frac{\delta_{i}\delta_{j}}{\delta_{i}^{2}+\delta_{j}^{2}+D-2}\right],$ $$

which is equal to 0 0 due to the fact that $\delta$ is zero-mean and symmetric in distribution. This completes the proof.∎

###### Lemma A.5 .

###### Lemma A.6 .

###### Lemma A.7 .

#### A.8.2 Proof for Theorem 3.1

Now that we have proved the two lemmas, we can prove Theorem [3.1](#S3.Thmtheorem1).

Proof. Let $f_{\theta}(\mathbf{x})$ be the model described in the theorem:

$$ $f_{\theta}(\mathbf{x})=\sum_{i=1}^{n}\omega_{i}\cdot\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\phi_{i}^{\top}x,$ $$

where $\omega_{i}$ and $\phi_{i}$ are i.i.d. random Gaussian variables with zero mean and variance $\sigma_{\omega}^{2}$ and $\sigma_{\phi}^{2}$, respectively. As such, we have $\theta^{\top}=[\omega_{i};\phi_{i}^{\top}]_{i=1}^{n}$, i.e., the concatenation of $\omega_{i}$s and $\phi_{i}$s. The derivatives of $f_{\theta}(\mathbf{x})$ can be written as:

$$ $\displaystyle\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ $\displaystyle=\sum_{i=1}^{n}\omega_{i}\cdot\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}}\phi_{i},$ $\displaystyle\frac{\partial f_{\theta}}{\partial\omega_{i}}(\mathbf{x})$ $\displaystyle=\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\phi_{i}^{\top}\mathbf{x},$ $\displaystyle\frac{\partial f_{\theta}}{\partial\phi_{i}}(\mathbf{x})$ $\displaystyle=\omega_{i}\cdot\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{x},$ $\displaystyle\frac{\partial\nabla_{\mathbf{x}}f_{\theta}}{\partial\omega_{i}}(\mathbf{x})$ $\displaystyle=\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\phi_{i},$ $\displaystyle\frac{\partial\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})}{\partial\phi_{i}}(\mathbf{x})$ $\displaystyle=\omega_{i}\cdot\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{I}.$ $$

Following Proposition [A.1](#A1.Thmtheorem1), we can write $\Delta_{\mathcal{F}}=\mathbf{A}_{\mathcal{F}}+\mathbf{A}_{\mathcal{F}}^{\top}$ from ([3](#S2.E3)) with:

$$ $\mathbf{A}_{\mathcal{F}}=-\sum_{\mu=1}^{m}\mathbb{E}_{\mathbf{x},\theta}\left[\left(\sum_{i=1}^{n}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\left(\left(\phi_{i}^{\top}\mathbf{x}_{\mu}\right)\phi_{i}+\omega_{i}^{2}\mathbf{x}_{\mu}\right)\right)\\ \left(\sum_{j=1}^{n}\omega_{j}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}>0}\phi_{j}^{\top}\right)\left(\sum_{k=1}^{n}\omega_{k}\mathbf{1}_{\phi_{k}^{\top}\mathbf{x}_{\mu}>0}\phi_{k}^{\top}\mathbf{x}_{\mu}-y\right)\right].$ (26) $$

Note that since $\omega$ is zero-mean, the terms related to $y$ are eliminated from ([26](#A1.E26)). Similarly, the terms involving $j\neq k$ are also eliminated. So we have:

$$ $\mathbf{A}_{\mathcal{F}}=-\sum_{\mu=1}^{m}\sum_{i=1}^{n}\sum_{j=1}^{n}\sigma_{\omega}^{2}\left[\mathbb{E}_{\theta,\mathbf{x}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}_{\mu}>0}\left(\phi_{i}^{\top}\mathbf{x}_{\mu}\right)\left(\phi_{j}^{\top}\mathbf{x}_{\mu}\right)\phi_{i}\phi_{j}^{\top}\right]+\sigma_{\omega}^{4}\mathbb{E}_{\theta,\mathbf{x}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}_{\mu}>0}\left(\phi_{j}^{\top}\mathbf{x}_{\mu}\right)\mathbf{x}_{\mu}\phi_{j}^{\top}\right]\right].$ (27) $$

For the first term in ([27](#A1.E27)), we have $n$ terms with $i=j$, which are equal to:

$$ $\displaystyle\mathbb{E}_{\theta,\mathbf{x}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\left(\phi_{i}^{\top}\mathbf{x}_{\mu}\right)^{2}\phi_{i}\phi_{i}^{\top}\right]$ $\displaystyle=\mathbb{E}_{\theta}\left[\mathbb{E}_{\mathbf{x}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\right]\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\left(\phi_{i}^{\top}\mathbf{x}_{\mu}\right)^{2}\phi_{i}\phi_{i}^{\top}\right]$ $\displaystyle=\frac{1}{2}\mathbb{E}_{\theta}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\left(\phi_{i}^{\top}\mathbf{x}_{\mu}\right)^{2}\phi_{i}\phi_{i}^{\top}\right]$ $\displaystyle=\frac{1}{4}\mathbb{E}_{\theta}\left[\left(\phi_{i}^{\top}\mathbf{x}_{\mu}\right)^{2}\phi_{i}\phi_{i}^{\top}\right]$ $\displaystyle=\frac{\sigma_{\phi}^{4}}{4}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}+\frac{\sigma_{\phi}^{4}}{4}\mathbf{x}_{\mu}^{\top}\mathbf{x}_{\mu}\mathbf{I},$ $$

And in the case when $i\neq j$, we have $n^{2}-n$ terms each of which are equal to:

$$ $\displaystyle\mathbb{E}_{\mathbf{x}}\left[\mathbb{E}_{\phi_{i}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\phi_{i}\phi_{i}^{\top}\right]\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbb{E}_{\phi_{j}}\left[\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}_{\mu}>0}\phi_{j}\phi_{j}^{\top}\right]\right]$ (28) $\displaystyle\approx\sigma_{\phi}^{4}\mathbb{E}_{\mathbf{x}}\left[\mathbb{E}_{\phi_{i}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\right]\cdot\mathbb{E}_{\phi_{j}}\left[\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}_{\mu}>0}\right]\right]\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}$ (29) $\displaystyle=\frac{\sigma_{\phi}^{4}}{3}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top},$ (30) $$

where ([29](#A1.E29)) comes from Lemma [A.5](#A1.Thmtheorem5), and ([30](#A1.E30)) comes from Lemma [A.6](#A1.Thmtheorem6). For the second term in ([27](#A1.E27)), when $i=j$ we have $n$ terms in the following form:

$$ $\displaystyle\mathbb{E}_{\theta,\mathbf{x}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\left(\phi_{i}^{\top}\mathbf{x}_{\mu}\right)\mathbf{x}_{\mu}\phi_{i}^{\top}\right]$ $\displaystyle=\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbb{E}_{\theta,\mathbf{x}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}}\phi_{i}\phi_{i}^{\top}\right]$ (31) $\displaystyle=\sigma_{\phi}^{2}\cdot\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbb{E}_{\mathbf{x},\phi_{i}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\right]$ (32) $\displaystyle=\frac{\sigma_{\phi}^{2}}{2}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top},$ (33) $$

where ([32](#A1.E32)) follows from Lemma [A.5](#A1.Thmtheorem5) and ([33](#A1.E33)) follows from Lemma [A.6](#A1.Thmtheorem6). And finally, the second term in ([27](#A1.E27)) has $n^{2}-n$ terms with $i\neq j$, which are equal to:

$$ $\displaystyle\mathbb{E}_{\mathbf{x}}\left[\mathbb{E}_{\phi_{i}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\right]\mathbb{E}_{\phi_{j}}\left[\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}_{\mu}>0}\left(\phi_{j}^{\top}\mathbf{x}_{\mu}\right)\mathbf{x}_{\mu}\phi_{j}^{\top}\right]\right]$ (34) $\displaystyle=\sigma_{\phi}^{2}\mathbb{E}_{\mathbf{x}}\left[\mathbb{E}_{\phi_{i}}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}_{\mu}>0}\right]\mathbb{E}_{\phi_{j}}\left[\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}>0}\mathbf{1}_{\phi_{j}^{\top}\mathbf{x}_{\mu}>0}\right]\right]\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}$ (35) $\displaystyle=\frac{\sigma_{\phi}^{2}}{3}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top},$ (36) $$

where ([35](#A1.E35)) comes from Lemma [A.5](#A1.Thmtheorem5) and ([36](#A1.E36)) comes from Lemma [A.6](#A1.Thmtheorem6).

Finally, given the results so far, we can write:

$$ $\displaystyle\Delta_{\mathcal{F}}$ $\displaystyle=-2\cdot\sum_{\mu=1}^{m}\left(\frac{n\sigma_{\phi}^{4}\sigma_{\omega}^{2}}{4}+\left(n^{2}-n\right)\frac{\sigma_{\phi}^{4}\sigma_{\omega}^{2}}{3}-\frac{n\sigma_{\phi}^{2}\sigma_{\omega}^{4}}{2}-\left(n^{2}-n\right)\frac{\sigma_{\phi}^{2}\sigma_{\omega}^{4}}{3}\right)\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}-2\cdot\frac{n\sigma_{\phi}^{4}\sigma_{\omega}^{2}}{4}\|\mathbf{x}_{\mu}\|_{2}^{2}\mathbf{I}-2\cdot\mathcal{O}(\frac{1}{\sqrt{D}})\mathcal{E}_{1}$ $\displaystyle=-2n\left(\frac{\sigma_{\phi}^{4}\sigma_{\omega}^{2}}{4}+\left(n-1\right)\frac{\sigma_{\phi}^{4}\sigma_{\omega}^{2}}{3}+\frac{\sigma_{\phi}^{2}\sigma_{\omega}^{4}}{2}+\left(n-1\right)\frac{\sigma_{\phi}^{2}\sigma_{\omega}^{4}}{3}\right)\sum_{\mu=1}^{m}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}-\frac{2n\sigma_{\phi}^{4}\sigma_{\omega}^{2}}{4}\sum_{\mu=1}^{m}\|\mathbf{x}_{\mu}\|_{2}^{2}\mathbf{I}-\mathcal{O}(\frac{2}{\sqrt{D}})\mathcal{E}_{1}$ $\displaystyle=-\mathcal{O}(n^{2})\cdot\mathbf{S}-\mathcal{O}(\frac{1}{\sqrt{D}})\cdot\mathcal{E}_{1}-\mathcal{O}(n)\cdot\mathcal{E}_{2},$ $$

where $\mathcal{E}_{1}$ is the error corresponding to our approximation from Lemma [A.5](#A1.Thmtheorem5), and $\mathcal{E}_{2}=\sum_{x_{\mu}=1}^{m}\|x_{\mu}\|_{2}^{2}\mathbf{I}$. So we have $\underset{{n,D\to\infty}}{\text{lim}}\left|\frac{\mathbf{Tr}\left(\Delta_{\mathcal{F}}^{\top}\mathbf{S}\right)}{\|\Delta_{\mathcal{F}}\|_{F}\cdot\|\mathbf{S}\|_{F}}\right|=\underset{{n,D\to\infty}}{\text{lim}}\left|\frac{\mathbf{Tr}\left(\left(\mathcal{O}(n^{2})\mathbf{S}+\mathcal{O}\left(\frac{1}{\sqrt{D}}\right)\mathcal{E}_{1}+\mathcal{O}(n)\mathcal{E}_{2}\right)^{\top}\mathbf{S}\right)}{\left\|\left(\mathcal{O}(n^{2})\mathbf{S}+\mathcal{O}\left(\frac{1}{\sqrt{D}}\right)\mathcal{E}_{1}+\mathcal{O}(n)\mathcal{E}_{2}\right)^{\top}\mathbf{S}\right\|_{F}\cdot\|\mathbf{S}\|_{F}}\right|=1$∎

#### A.8.3 Proof for Theorem 3.2

Proof. Let $f_{\theta}(\mathbf{x})$ be the model described in the theorem:

$$ $f_{\theta}(\mathbf{x})=\frac{1}{k}\sum_{i=1}^{n}\omega_{i}\left(\sum_{j=1}^{k}\phi_{i}^{\top}\mathbf{M}_{j}^{\top}\mathbf{x}\right),$ $$

where $\omega_{i}$ and $\phi_{i}$ are i.i.d. random Gaussian variables with zero mean and variances $\sigma_{\omega}^{2}$ and $\sigma_{\phi}^{2}$, respectively, and $\mathbf{M}_{j}$ is a binary matrix mimicking the convolution operation. Note that we set $\mathbf{I}_{j_{1},j_{2}}=\mathbf{M}_{j_{1}}\mathbf{M}_{j_{2}}^{T}$, which is a binary diagonal matrix with elements in the receptive field of both $j_{1}$ and $j_{2}$ set to $1$ and 0 0 otherwise. Furthermore, we have $\theta^{\top}=[\omega_{i};\phi_{i}^{\top}]_{i=1}^{n}$, i.e., the concatenation of $\omega_{i}$s and $\phi_{i}$s. The derivatives of $f_{\theta}(\mathbf{x})$ can be written as:

$$ $\displaystyle\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})$ $\displaystyle=\frac{1}{k}\sum_{i=1}^{n}\omega_{i}\left(\sum_{j=1}^{k}\mathbf{M}_{j}\phi_{i}\right),$ $\displaystyle\frac{\partial f_{\theta}(\mathbf{x})}{\partial\omega_{i}}(\mathbf{x})$ $\displaystyle=\frac{1}{k}\sum_{j=1}^{k}\phi_{i}^{\top}\mathbf{M}_{j}^{\top}\mathbf{x},$ $\displaystyle\frac{\partial\nabla_{\mathbf{x}}f_{\theta}}{\partial\omega_{i}}(\mathbf{x})$ $\displaystyle=\frac{1}{k}\sum_{j=1}^{k}\mathbf{M}_{j}\phi_{i},$ $\displaystyle\frac{\partial f_{\theta}}{\partial\phi_{i}}(\mathbf{x})$ $\displaystyle=\frac{1}{k}\omega_{i}\sum_{j=1}^{k}\mathbf{M}_{j}^{\top}\mathbf{x},$ $\displaystyle\frac{\partial\nabla_{\mathbf{x}}f_{\theta}}{\partial\phi_{i}}(\mathbf{x})$ $\displaystyle=\frac{1}{k}\omega_{i}\sum_{j=1}^{k}\mathbf{M}_{j}.$ $$

We first evaluate $\mathbf{G}_{\mathcal{F}}(\mathbf{x})$:

$$ $\displaystyle\mathbf{G}_{\mathcal{F}}(\mathbf{x})$ $\displaystyle=\frac{1}{k^{2}}\mathbb{E}_{\theta}\left[\left(\sum_{i_{1}=1}^{n}\omega_{i_{1}}\left(\sum_{j_{1}=1}^{k}\mathbf{M}_{j_{1}}\phi_{i_{1}}\right)\right)\left(\sum_{i_{2}=1}^{n}\omega_{i_{2}}\left(\sum_{j_{2}=1}^{k}\mathbf{M}_{j_{2}}\phi_{i_{2}}\right)\right)^{\top}\right]$ $\displaystyle=\frac{\sigma_{\omega}^{2}}{k^{2}}\sum_{i=1}^{n}\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\mathbb{E}_{\theta}\left[\mathbf{M}_{j_{1}}\phi_{i}\phi_{i}^{T}\mathbf{M}_{j_{2}}^{\top}\right]$ $\displaystyle=\frac{n\sigma_{\omega}^{2}\sigma_{\phi}^{2}}{k^{2}}\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\mathbf{I}_{j_{1},j_{2}}.$ $$

Now we will try to show that $\Delta_{\mathcal{F}}(\mathbf{x})=\mathbf{A}_{\mathcal{F}}(\mathbf{x})+\mathbf{A}_{\mathcal{F}}(\mathbf{x})^{\top}=\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})$. Following Proposition [A.1](#A1.Thmtheorem1), we can write $\Delta_{\mathcal{F}}$ as:

$$ $\mathbf{A}_{\mathcal{F}}(\mathbf{x})=-\frac{1}{k^{4}}\sum_{\mu=1}^{m}\sum_{i_{1}=1}^{n}\sum_{i_{2}=1}^{n}\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\sum_{j_{3}=1}^{k}\sum_{j_{4}=1}^{k}\left(\sigma_{\omega}^{2}\mathbb{E}_{\theta}\left[\left(\phi_{i_{1}}^{\top}\mathbf{M}_{j_{1}}^{\top}\mathbf{x}_{\mu}\right)\left(\phi_{i_{1}}^{\top}\mathbf{M}_{j_{4}}^{\top}\mathbf{x}_{\mu}\right)\mathbf{M}_{j_{2}}\phi_{i_{1}}\phi_{i_{2}}^{\top}\mathbf{M}_{j_{3}}^{\top}\right]+\sigma_{\omega}^{4}\mathbb{E}_{\theta}\left[\left(\phi_{i_{2}}^{\top}\mathbf{M}_{j_{4}}^{\top}\mathbf{x}_{\mu}\right)\mathbf{M}_{j_{1}}\mathbf{M}_{j_{2}}^{\top}\mathbf{x}_{\mu}\phi_{i_{2}}^{\top}\mathbf{M}_{j_{3}}^{\top}\right]\right).$ (37) $$

The first term in ([37](#A1.E37)) has $n$ terms with $i_{1}=i_{2}$ (substitute $i_{1},i_{2}$ with $i$), which can be written as:

$$ $n\mathbf{M}_{j_{2}}\mathbb{E}_{\phi_{i}}\left[\left(\phi_{i}^{\top}\mathbf{M}_{j_{1}}^{\top}\mathbf{x}_{\mu}\right)\phi_{i}\phi_{i}^{\top}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}\phi_{i}\phi_{i}^{\top}\right]=n\mathbf{M}_{j_{2}}\mathbb{E}_{\phi_{i}}\left[\phi_{i}\phi_{i}^{\top}\mathbf{M}_{j_{1}}^{\top}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}\phi_{i}\phi_{i}^{\top}\right]$ $$

So using Stein’s lemma, we have:

$$ $n\mathbf{M}_{j_{2}}\mathbb{E}_{\phi_{i}}\left[\phi_{i}\phi_{i}^{\top}\mathbf{M}_{j_{1}}^{\top}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}\phi_{i}\phi_{i}^{\top}\right]=n\sigma_{\phi}^{2}\mathbf{M}_{j_{2}}\left(\mathbb{E}_{\phi_{i}}\left[\phi_{i}^{\top}\mathbf{M}_{j_{1}}^{\top}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}\phi_{i}\right]+\mathbf{M}_{j_{1}}^{\top}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}\mathbb{E}_{\phi_{i}}\left[\phi_{i}\phi_{i}^{\top}\right]\right).$ $$

The first term is equal to $\sigma_{\phi}^{2}\mathbf{Tr}\left(\mathbf{M}_{j_{1}}^{\top}x_{\mu}x_{\mu}^{\top}\mathbf{M}_{j_{4}}\right)$, and the second term is equal to $\sigma_{\phi}^{2}\mathbf{M}_{j_{1}}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}$. So the $n$ terms in ([37](#A1.E37)) with $i_{1}=i_{2}$ are equivalent to:

$$ $n\sigma_{\phi}^{4}\mathbf{I}_{j_{2},j_{1}}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{I}_{j_{3},j_{4}}+n\sigma_{\phi}^{4}\mathbf{Tr}\left(\mathbf{M}_{j_{1}}^{\top}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}\right)\mathbf{I}_{j_{2},j_{3}}.$ $$

The first term in ([37](#A1.E37)) also has $n^{2}-n$ terms with $i_{1}\neq i_{2}$, which can be written as:

$$ $\left(n^{2}-n\right)\mathbf{M}_{j_{2}}\mathbb{E}_{\phi_{i_{1}}}\left[\phi_{i_{1}}\phi_{i_{1}}^{\top}\right]\mathbf{M}_{j_{1}}^{\top}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}\mathbb{E}_{\phi_{i_{2}}}\left[\phi_{i_{2}}\phi_{i_{2}}^{\top}\right]\mathbf{M}_{j_{3}}=\left(n^{2}-n\right)\sigma_{\phi}^{4}\mathbf{I}_{j_{2},j_{1}}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{T}\mathbf{I}_{j_{3},j_{4}}.$ $$

The second term in ([37](#A1.E37)) has $n^{2}$ which can be written as:

$$ $n^{2}\mathbf{I}_{j_{1},j_{2}}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{\top}\mathbf{M}_{j_{4}}\mathbb{E}_{\phi_{i_{2}}}\left[\phi_{i_{2}}\phi_{i_{2}}^{\top}\right]\mathbf{M}_{j_{3}}=\sigma_{\phi}^{2}n^{2}\mathbf{I}_{j_{1},j_{2}}\mathbf{x}_{\mu}\mathbf{x}_{\mu}^{T}\mathbf{I}_{j_{3},j_{4}}.$ $$

So we can write:

$$ $\Delta_{\mathcal{F}}(\mathbf{x})=-2\cdot\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\sum_{j_{3}=1}^{k}\sum_{j_{4}=1}^{k}\frac{n^{2}}{k^{4}}\left(\sigma_{\omega}^{2}\sigma_{\phi}^{4}+\sigma_{\omega}^{4}\sigma_{\phi}^{2}\right)\mathbf{I}_{j_{1},j_{2}}\mathbf{S}\mathbf{I}_{j_{3},j_{4}}-2\cdot\frac{n}{k^{4}}\sigma_{\omega}^{2}\sigma_{\phi}^{4}\mathbf{Tr}\left(\mathbf{M}_{j_{1}}^{\top}\mathbf{S}\mathbf{M}_{j_{4}}\right)\mathbf{I}_{j_{2},j_{3}}.$ $$

So we have:

$$ $\Delta_{\mathcal{F}}(\mathbf{x})=-\mathcal{O}\left(n^{2}\right)\cdot\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})^{\top}-\mathcal{O}\left(n\right)\cdot\mathcal{E},$ $$

where $\mathcal{E}$ corresponds to the error term $\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\sum_{j_{3}=1}^{k}\sum_{j_{4}=1}^{k}\mathbf{Tr}\left(\mathbf{M}_{j_{1}}^{\top}\mathbf{S}\mathbf{M}_{j_{4}}\right)\mathbf{I}_{j_{2},j_{3}}$. Therefore, we can write: $\underset{n\to\infty}{\text{lim}}\left|\frac{\mathbf{Tr}\left(\Delta_{\mathcal{F}}(\mathbf{x})^{\top}\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})\right)}{\left\|\Delta_{\mathcal{F}}(\mathbf{x})\right\|_{F}\cdot\left\|\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})\right\|_{F}}\right|=\underset{n\to\infty}{\text{lim}}\left|\frac{\mathbf{Tr}\left(\left(\mathcal{O}\left(n^{2}\right)\cdot\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})^{\top}+\mathcal{O}\left(n\right)\cdot\mathcal{E}\right)^{\top}\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})\right)}{\left\|\left(\mathcal{O}\left(n^{2}\right)\cdot\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})^{\top}+\mathcal{O}\left(n\right)\cdot\mathcal{E}\right)\right\|_{F}\cdot\left\|\mathbf{G}_{\mathcal{F}}(\mathbf{x})\mathbf{S}\mathbf{G}_{\mathcal{F}}(\mathbf{x})\right\|_{F}}\right|=1$ which completes our proof.∎

#### A.8.4 Proof for Theorem A.2

Proof. Let $f_{\theta}^{\ell}(\mathbf{x})$ be an MLP or CNN with $\ell$ layers, no pooling or normalization layers, and a scalar output:

$$ $f_{\theta}^{\ell}(\mathbf{x})=\sum_{i=1}^{n}\omega_{i}\mathbf{1}_{f_{\phi_{i}}^{\ell-1}(\mathbf{x})>0}f_{\phi_{i}}^{\ell-1}(\mathbf{x}),$ $$

where $\theta$ corresponds to the union of $\omega_{i}$s and $\phi_{i}$s, and $f_{\phi_{i}}(\cdot)$ and $f_{\phi_{j}}(\cdot)$ for $i\neq j$ share the same parameters up to the last layer. The derivative of $f_{\theta}^{\ell}(\mathbf{x})$ w.r.t. $\mathbf{x}$ is:

$$ $\nabla_{\mathbf{x}}f_{\theta}^{\ell}(\mathbf{x})=\sum_{i=1}^{n}\omega_{i}\mathbf{1}_{f_{\phi_{i}}^{\ell-1}(\mathbf{x})>0}\nabla_{\mathbf{x}}f_{\phi_{i}}^{\ell-1}(\mathbf{x}),$ $$

which means we have:

$$ $\displaystyle\mathbb{E}_{\theta}\left[\nabla_{\mathbf{x}}f_{\theta}^{\ell}(\mathbf{x})\nabla_{\mathbf{x}}f_{\theta}^{\ell}(\mathbf{x})^{\top}\right]$ $\displaystyle=\sum_{i_{1}=1}^{n}\sum_{i_{2}=1}^{n}\mathbb{E}_{\theta}\left[\omega_{i}\omega_{j}\mathbf{1}_{f_{\phi_{i}}^{\ell-1}(\mathbf{x})>0}\mathbf{1}_{f_{\phi_{j}}^{\ell-1}(\mathbf{x})>0}\nabla_{\mathbf{x}}f_{\phi_{i_{1}}}^{\ell-1}(\mathbf{x})\nabla_{x}f_{\phi_{i_{2}}}^{\ell-1}(\mathbf{x})^{\top}\right]$ $\displaystyle=\sigma_{\omega}^{2}\sum_{i=1}^{n}\mathbb{E}_{\phi_{i}}\left[\mathbf{1}_{f_{\phi_{i}}^{\ell-1}(\mathbf{x})>0}\nabla_{\mathbf{x}}f_{\phi_{i}}^{\ell-1}(\mathbf{x})\nabla_{\mathbf{x}}f_{\phi_{i}}^{\ell-1}(\mathbf{x})^{\top}\right].$ $$

So we need to prove that:

$$ $\mathbb{E}_{\phi_{i}}\left[\mathbf{1}_{f_{\phi_{i}}^{\ell-1}(\mathbf{x})>0}\nabla_{\mathbf{x}}f_{\phi_{i}}^{\ell-1}(\mathbf{x})\nabla_{x}f_{\phi_{i}}^{\ell-1}(\mathbf{x})^{\top}\right]=c\cdot\mathbf{I}.$ $$

First note that since the weights of the last layer (i.e., the classification layer) are zero-mean and Gaussian, the model w.r.t. these weights is symmetric at 0 0, which means we have:

$$ $\mathbb{E}_{\phi_{i}}\left[\mathbf{1}_{f_{\phi_{i}}^{\ell-1}(\mathbf{x})>0}\nabla_{\mathbf{x}}f_{\phi_{i}}^{\ell-1}(\mathbf{x})\nabla_{\mathbf{x}}f_{\phi_{i}}^{\ell-1}(\mathbf{x})^{\top}\right]=\frac{1}{2}\mathbb{E}_{\phi_{i}}\left[\nabla_{\mathbf{x}}f_{\phi_{i}}^{\ell-1}(\mathbf{x})\nabla_{\mathbf{x}}f_{\phi_{i}}^{\ell-1}(\mathbf{x})^{\top}\right]$ $$

So in order to prove this theorem, we need to use induction. For the case with $\ell=2$, we can write the model as:

$$ $f_{\theta}^{2}(\mathbf{x})=\sum_{i=1}^{n}\omega_{i}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\phi_{i}^{\top}\mathbf{x},$ $$

where both $\omega_{i}$s and $\phi_{i}$s are i.i.d. Gaussian variables with variances equal to $\sigma_{\omega}^{2}$ and $\sigma_{\phi}^{2}$, respectively. This gives us the following gradient w.r.t. the input:

$$ $\nabla_{x}f_{\theta}^{2}(\mathbf{x})=\sum_{i=1}^{n}\omega_{i}\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\phi_{i}.$ $$

So we have:

$$ $\displaystyle\mathbb{E}_{\theta}\left[\nabla_{\mathbf{x}}f_{\theta}^{2}(\mathbf{x})\nabla_{\mathbf{x}}f_{\theta}^{2}(\mathbf{x})^{\top}\right]$ $\displaystyle=\sigma_{\omega}^{2}\sum_{i=1}^{n}\mathbb{E}_{\theta}\left[\mathbf{1}_{\phi_{i}^{\top}\mathbf{x}>0}\phi_{i}\phi_{i}^{\top}\right]$ (38) $\displaystyle=\frac{1}{2}\sum_{i=1}^{n}\mathbb{E}_{\phi_{i}}\left[\phi_{i}\phi_{i}^{\top}\right]$ (39) $\displaystyle=\frac{n\sigma_{\phi}^{2}}{2}\mathbf{I},$ (40) $$

where ([39](#A1.E39)) follows from $\phi_{i}$s being zero-mean i.i.d. Gaussian variables. The rest of the proof follows from induction.∎

#### A.8.5 Proof for Theorem A.3

Proof. Let $f_{\theta}(\mathbf{x})$ be the model described:

$$ $f_{\theta}(\mathbf{x})=\frac{1}{k}\sum_{j=1}^{k}\theta^{\top}\mathbf{M}_{j}^{\top}\mathbf{x},$ $$

where $\theta$ is a i.i.d. random Gaussian variable with zero mean and variance $\sigma_{\theta}^{2}$, and $\mathbf{M}_{j}$ is a binary matrix mimicking the convolution operation. The derivative of $f_{\theta}(x)$ w.r.t. $x$ is:

$$ $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})=\frac{1}{k}\sum_{j=1}^{k}\frac{1}{\sigma_{j}(\theta)}\mathbf{M}_{j}\theta.$ $$

So we can write:

$$ $\displaystyle\mathbf{G}_{\mathcal{F}}$ $\displaystyle=\frac{1}{k^{2}}\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\mathbf{M}_{j_{1}}\mathbb{E}_{\theta}\left[\theta\theta^{\top}\right]\mathbf{M}_{j_{2}}^{\top}$ $\displaystyle=\frac{\sigma_{\theta}^{2}}{k^{2}}\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\mathbf{M}_{j_{1}}\mathbf{M}_{j_{2}}^{\top}$ $\displaystyle=\frac{\sigma_{\theta}^{2}}{k^{2}}\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\mathbf{I}_{j_{1},j_{2}},$ $$

which completes the proof.∎

#### A.8.6 Proof for Corollary A.4

Proof. Let $f_{\theta}(\mathbf{x})$ be the model described:

$$ $f_{\theta}(\mathbf{x})=\frac{1}{k}\sum_{j=1}^{k}\frac{1}{\sigma_{j}(\theta)}\theta^{\top}\mathbf{M}_{j}^{\top}\mathbf{x},$ $$

where $\theta$ is a i.i.d. random Gaussian variable with zero mean and variance $\sigma_{\theta}^{2}$, and $\mathbf{M}_{j}$ is a binary matrix mimicking the convolution operation. The derivative of $f_{\theta}(x)$ w.r.t. $x$ is:

$$ $\nabla_{\mathbf{x}}f_{\theta}(\mathbf{x})=\frac{1}{k}\sum_{j=1}^{k}\frac{1}{\sigma_{j}(\theta)}\mathbf{M}_{j}\theta.$ $$

So we can write:

$$ $\mathbf{G}_{\mathcal{F}}=\frac{1}{k^{2}}\sum_{j_{1}=1}^{k}\sum_{j_{2}=1}^{k}\mathbf{M}_{j_{1}}\mathbb{E}_{\theta}\left[\frac{\theta\theta^{\top}}{\sigma_{j_{1}}(\theta)\sigma_{j_{2}}(\theta)}\right]\mathbf{M}_{j_{2}}^{\top}.$ $$

Note that we can write $\sigma_{j}(\theta)$ for some $j$ as:

$$ $\displaystyle\sigma_{j}(\theta)$ $\displaystyle=\frac{1}{D}\theta^{\top}\mathbf{M}_{j}^{\top}\mathbf{S}\mathbf{M}_{j}\theta$ $\displaystyle=\frac{1}{D}\theta^{\top}\mathbf{S}_{j}\theta,$ $$

which can be bounded as:

$$ $\lambda_{min}^{j}\|\theta\|_{2}^{2}\leq D\cdot\sigma_{j}(\theta)\leq\lambda_{max}^{j}\|\theta\|_{2}^{2}.$ $$

This gives us the following bounds:

$$ $\frac{1}{\sqrt{\lambda_{min}^{j_{1}}\lambda_{min}^{j_{2}}}}\mathbb{E}_{\theta}\left[\frac{\theta\theta^{\top}}{\|\theta\|_{2}^{2}}\right]\succeq\mathbb{E}_{\theta}\left[\frac{\theta\theta^{\top}}{\sigma_{j_{1}}(\theta)\sigma_{j_{2}}(\theta)}\right]\succeq\frac{1}{\sqrt{\lambda_{max}^{j_{1}}\lambda_{max}^{j_{2}}}}\mathbb{E}_{\theta}\left[\frac{\theta\theta^{\top}}{\|\theta\|_{2}^{2}}\right].$ $$

And from Lemma [A.7](#A1.Thmtheorem7), we have:

$$ $\frac{1}{\sqrt{\lambda_{min}^{j_{1}}\lambda_{min}^{j_{2}}}}\mathbf{I}\succeq\mathbb{E}_{\theta}\left[\frac{\theta\theta^{\top}}{\sigma_{j_{1}}(\theta)\sigma_{j_{2}}(\theta)}\right]\succeq\frac{1}{\sqrt{\lambda_{max}^{j_{1}}\lambda_{max}^{j_{2}}}}\mathbf{I},$ $$

which completes our proof.∎

### A.9 Experimental setting

#### A.9.1 Hyperparameters

All of our experiments are performed on the following models:

- •
MLP: A multi-layer perception with 2 hidden layers of size 100 and ReLU non-linearity.
- •
LeNet: We use the model introduced in LeCun et al. ().
- •
ResNet18: We use the model introduced in He et al. ().
- •
VGG11: We use the model introduced in Simonyan & Zisserman ().
- •
ViT. We use the architecture introduced in Dosovitskiy et al. () with the settings provided by Lee et al. () for small-scale data.

Similar to Ortiz-Jiménez et al. (), we observed much better and accurate value for $\mathbf{G}_{\mathcal{F}}$ when using GELU non-linearity Hendrycks & Gimpel () compared to ReLU, which we also attribute to numerical problems caused by ReLU. In this case, we observed lower variance when computing $\mathbf{G}_{\mathcal{F}}$ with a probing function with a smaller standard deviation, which we attribute to the saturation of the GELU function for standard Gaussian input. Specifically, in our experiments we set $\mathcal{P}=\mathcal{N}\left(0,\sigma_{\mathcal{P}}^{2}\mathbf{I}\right)$ with $\sigma_{\mathcal{P}}$ working best in range of $10^{-3}$ and $10^{-5}$ in terms of correlation between generalization gap and eigenvalues of $\mathbf{G}_{\mathcal{F}}$ in the experiment presented in Appendix [A.7](#A1.SS7). However, similar to Ortiz-Jiménez et al. (), we observe that a first-order approximation of $\mathbf{G}_{\mathcal{F}}$ in the form of $\mathbf{G}_{\mathcal{F}}(0)$ is also a relatively good approximation.

The experiments provided in Figure [2](#S3.F2) are performed using gradient-descent with the learning rate equal to $0.1$ and momentum set to $0.9$, and for $100$ epochs. The experiments provided in Figure [8](#A1.F8) are performed using the Adam variant of gradient descent with default parameters due to slower convergence and for $100$ epochs. The weight decay in all these experiments is set to 0 0.

For the experiments in Figure [4](#S4.F4) and Figure [5](#S4.F5), we use the experiment settings of Ortiz-Jiménez et al. () for the synthetic data, except for the experiments in Figure [4](#S4.F4) performed on a sinusoidal decision boundary, for which we train all models with $50$ epochs.

The experiments in Figure [3](#S4.F3) are performed using stochastic gradient-descent with a batch size of $128$ and learning rate of $0.1$. We train the model for $100$ epochs for the experiment involving LeNet and $50$ epochs for the experiments involving ResNet18 and ViT. We set the momentum to $0.9$ and the weight decay to 0 0.

The experiments in Figure [6](#S5.F6) are performed using stochastic gradient-descent with a batch size of $128$, learning rate of $0.1$, and momentum set to $0.9$ and $50$ epochs. The weight decay in all these experiments is set to 0 0. The experiments in Table [1](#S5.T1) and Table [2](#A1.T2) are performed using stochastic gradient-descent with a batch size of $128$, learning rate of $0.2$, and momentum set to $0.9$ and $50$ epochs. The weight decay in all these experiments is set to $5\times 10^{-4}$.

In all experiments, we estimate $\mathbf{G}_{\mathcal{F}}$ on $10000$ models and $\mathbf{G}_{\mathcal{F}}^{t}$ for $t>0$ on $25$ models.

#### A.9.2 Statistical information

For the experiments in Figure [2](#S3.F2), Figure [8](#A1.F8), and Figure [3](#S4.F3) we only report the average of test/train accuracy since we mainly aim to show the trend of the changes in these values. However, the experiments in Figure [4](#S4.F4), Figure [5](#S4.F5), Figure [6](#S5.F6), Table [1](#S5.T1), and Table [2](#A1.T2) are all performed $5$ times. We report the average along with the $68\%$ confidence interval for all these experiments.

#### A.9.3 Hardware and framework

All experiments are performed on a single Nvidia GTX 1080 Ti, and implemented on Python using PyTorch 2.1.1. We base our implementation on the code provided by Ortiz-Jiménez et al. ().

### A.10 Limitations

We consider two main limitations to our work: theoretical limitations and experimental limitations. In the case of the theoretical limitation, we note that our theoretical results are mainly focused on simple models (i.e., models with one or two hidden layers, and ReLU or linear activation functions) and based on approximation (i.e., correct for large input space size and model width). The reason for this approach is that in theoretical results, we’re mainly concerned with insight as opposed to completeness. Consequently, we omit providing proofs with minimum assumptions and more realistic models in order to keep the message of the paper focused and the content concise.

In the case of experimental limitation, we note that our experiments are mainly concerned with over-parameterized settings and small-scale image classification datasets (i.e., models with a few million parameters and datasets with about 5000 samples for each class). This decision is mainly due to our lack of computational resources. As a result, we do not make any claims about more complex settings involving, for instance, foundation models or large-scale datasets. Furthermore, we do not extend our results to other aspects of machine learning, such as regression or reinforcement learning.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 9. From Figure 1, it seems that simple (and standard) early stopping would solve the generalization gap problem and actually result in a lower test loss for dataset $D_B$ compared to dataset $D_A$. Can you comment on this? How general is this observation? 

10. What’s the underlying distribution over which you are taking the expectation at L227? It should be made explicit.

11. When you write “momentums” (L244, L246) do you mean “moments” instead?

12. Is the fact that $\Delta_F$ becomes label-independent purely due to the fact that you assume $\mathbb{E}_\theta [\theta]=0$ at initialization?

13. Could the phenomena observed in the paper, particularly when testing conjecture 2 about the GIH, be due to the fact that networks in all experiments are strongly overparameterized (even with CIFAR, the paper considers only a small fraction of the full dataset) and training in the “lazy” or NTK regime where weights stay close to their initial values? Did the authors measure the weight evolution throughout training? Would you expect the same results when training with more data, e.g., the full CIFAR10 dataset, and/or with tasks requiring learning latent features of the data that might be less dependent on the exact geometry and statistics in input space, e.g., more complex image datasets such as ImageNet?

14. Could the authors comment on Conjecture 2 in the case of linear regression? Intuitively, it seems it should hold in that simpler setting.

15. What is the motivation behind adding label noise (L406)? Would the same results or trends be observed in its absence?
soundness: 2
strengths: 1. The introduction of the Geometric Invariance Hypothesis appears novel and extends findings of Neural Anisotropy Directions (Ortiz-Jimenez et al., 2021) to non-linear decision boundaries. This hypothesis has the potential to provide insights into the relationship between neural network architecture and the structure of data, contributing to our understanding of inductive biases in deep learning.

2. The experiments and the theoretical analysis are generally fair, although several imprecisions are present, and some clarifications are needed (see detailed points below).

3. The theoretical findings are tested using real networks (e.g., ResNets, ViTs) and datasets (CIFAR). These experiments appear to align with and support the GIH. Nevertheless, it is worth noting that the considered settings are highly overparameterized and relatively simple, which raises concerns regarding the broader validity and general significance of the results (see detailed points below).
summary: The paper studies how the geometry of neural network predictors evolves in the input space during training. It proposes the Geometric Invariance Hypothesis (GIH), which posits that this evolution occurs within a constrained subspace of the input space determined by the network architecture. The GIH is supported through experiments with MLPs, CNNs, and ViTs on a subset of CIFAR with binarized labels. The paper also studies the link between the GIH and generalization.
weaknesses: The paper is quite dense. There are multiple points of confusion and imprecisions affecting both clarity and soundness. Specifically:

4. The introduction and main text lack a comprehensive overview of the field and references to related work. Only a few broad papers are cited, despite the extended page limit of this year's edition. I strongly encourage the authors to move much of the discussion from Appendix A.1 into the main text to better place the work in context. In particular, NADs introduced in Ortiz-Jimenez et al. (2021) should be explicitly discussed in the main text, given how strongly related they are to this work.

5. Lines L058+ seem to confuse expressivity with generalization, i.e., approximation vs. statistical properties. The fact that an MLP is universal (as some deep and wide CNNs are) does not imply that it can generalize effectively. 

6. Assuming i.i.d. parameters is not enough to conclude that $G_F \propto I$. A Gaussian distribution with a non-zero mean is a counterexample.

7. The claim that MLPs have a lack of inductive bias (L210, L224, L249) is wrong. Even in the NTK regime, MLPs display a strong spectral bias for low modes. It may be more accurate to state that MLPs at initialization have no preferred directions in the input space, i.e., they have an isotropic prior in the input space.

8. Multiple terms are left undefined or used imprecisely: e.g., “SSE loss” and $y_{\mu}$. Some phrases are vague: What is the “input-space curvature”, and how can it depend on the training process (L011)? What is the “input geometry” of a neural network (L076, L521)? What does it mean for a dimension to be noise (L053)? What are “all the possible values of $\theta$” ( L145)? How are they distributed? Given that you consider gradient-flow in your analysis, how do you define the “first step” in L286?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Just as a small note, in line 211 you mention the interaction between two factors, specifically the data-dependent factor and another. I'm guessing this is the model's geometric inductive bias? If so, I think it could be written a bit clearer. 

Additionally, would you think this approach would be useful for architecture-based optimal dataset subsampling methodology?
soundness: 3
strengths: The paper is able to gradually build up to the main hypothesis being proposed while maintaining a clear chain of reasoning. The authors also provide extensive mathematical proofs for each step in the build-up and mention what assumptions are made and any limitations on what can be shown or derived. Finally, they are able to provide some insight into the effect of this hypothesis on an architecture's generalization ability while addressing any possible ideas with empirical results.
summary: In their paper, the authors argue for the link between input-space geometry and the ability of models to generalize. They propose the geometric invariance hypothesis (GIH) which states that, depending on the architecture of the model, the curvature of the input-space remains invariant under transformations in certain directions. They then attempt to prove and study this phenomenon by separating the changes in the input-space geometry during training into architecture-dependent changes and data-dependent changes. This resulted in finding the special cases where the invariance occurs, and they link this to the generalization ability of the model and where it collapses to reliance on noise and memory.
weaknesses: While the "performance" gains of the paper do seem marginal, I see these experiments as more of a proof of concept of the ideas and the proposed hypothesis. However, it would have been nice to see these experiments on multiple datasets to verify if the claims still hold, especially given the simplicity of the current model choices as well.

---

confidence: 2
contribution: 4
initial_rating: 8
presentation: 4
questions: I am interested in how your observations are affected by different training dynamics, specifically when using stochastic gradient descent (SGD) with large learning rates—a common practice for achieving high prediction accuracy. Have you explored how the geometric invariance hypothesis (GIH) and the behavior of average geometry and its evolution manifest under such training conditions?
soundness: 4
strengths: The introduction of the geometric invariance hypothesis (GIH) offers a fresh and innovative perspective on the interplay between neural network architectures and the geometry of the input space during training. By proposing the concepts of average geometry and average geometry evolution, the authors provide novel tools for quantifying how different architectures influence learning dynamics. This approach moves beyond traditional analyses by directly linking architectural properties to geometric transformations in the input space, which is a significant conceptual advancement in understanding deep learning models.
summary: This paper introduces the geometric invariance hypothesis (GIH), which suggests that during the training of neural networks, the curvature of the input space remains invariant under transformations along specific directions determined by the network’s architecture. The authors begin by examining a non-linear binary classification problem situated on a hyperplane in a high-dimensional space. They observe that a multilayer perceptron (MLP) can solve this problem regardless of the hyperplane’s orientation, whereas a residual network (ResNet) cannot. Motivated by this finding, they define two architecture-dependent maps: the average geometry, which provides a compact summary of the network’s input-space geometry, and the average geometry evolution, which describes how this geometry changes during training. Through both theoretical analysis and empirical investigations, the paper demonstrates that GIH arises because the average geometry evolution closely approximates the projection of the data covariance onto the average geometry. This results in an invariance property, especially when the average geometry is low-rank. Finally, the authors present extensive experimental results to explore the consequences of GIH and its relationship to generalization in neural networks.
weaknesses: While the paper makes significant contributions, there are areas that could be improved:

- Lack of Intuitive Explanation: It is challenging to develop an intuition for why ResNets behave differently from MLPs. Providing more intuitive explanations or illustrative examples before introducing the mathematical formalism would help readers grasp the core concepts and follow the subsequent analysis more effectively.
- Limited Architectural Comparison: The focus on ResNets without discussing other architectures like AlexNet leaves some gaps. Clarifying whether the observed behaviors are due to specific features like skip connections or are common across different architectures would strengthen the generality of the findings.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 2
questions: - The experiments remove normalization from the neural networks. Is this crucial? Does the trend totally deteriorate with normalization? Is there any hypothesis made by the theory as to why?
- Regarding the Conjectures 1 and 2 about dynamics: Is there an infinite- or large-width assumption being implicitly made, for instance? Should readers interpret the Conjectures as holding in appropriate scaling limits or should they hold at reasonable scales?
soundness: 4
strengths: - The examination of input-space geometry of neural networks is a more novel approach: other more mainstream approaches such as NTK and AGOP consider the parameter-space geometry.
- The conclusions are interesting, and to some degree, sensible: if there are directions in the input space which do not support the training data distribution, then the training dynamics may not move the input-space geometry in these directions.
- The paper does a good job motivating a very theoretical research direction with plausible practical implications in Section 5.
- The paper is reasonably well-written; despite the material being technically dense, it mostly makes sense after a thorough read.
summary: The authors study the input-space geometry of neural networks, i.e., the decision boundaries and loss surface at each point in the input space. They give mathematical characterizations of the expected geometry and dynamics of the geometry under gradient flow at initialization. They make some intuitive conjectures about how the geometry evolves under training, one of which is the Geometric Invariance Hypothesis, which says that the geometry of the model remains invariant in "un-important" directions of the input space while training. They then experimentally verify parts of their conjectures using some representative architectures on both synthetic data and smaller-scale real data.
weaknesses: - The definitions of geometry in this work are based on expected second moments of the data. This captures the smallest supporting subspace of the data. But data can have very large linear dimension while having small intrinsic dimension, cf "manifold hypothesis". The theory here does not seem to capture this more involved geometric aspect, but this assumption has been hypothesized to be more realistic for real data. The lack of understanding here makes the experiments less likely to track with the theory and may preclude more prescriptive insights, beyond those generated by the preliminary experiments in Section 5.
- More minor: There is not much intuition provided about why the predictions made at initialization should transfer reasonably well to training, and thus (in my opinion) not too much motivation for the Conjectures (which are about training dynamics).

Meta-review: This paper offers an innovative approach to understanding generalization in machine learning through geometry. The introduction of the Geometric Invariance Hypothesis could inspire further research into input-space properties and their role in model performance. While some aspects require further elaboration and validation, the work provides a solid foundation for exploring new directions in theoretical and empirical studies of generalization.

Decision: Accept (Spotlight)