Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture

Authors: Sajad Movahedi, Antonio Orvieto, Seyed-Mohsen Moosavi-Dezfooli
Primary area: optimization
Keywords: optimization, deep neural networks, deep learning, deep learning theory, machine learning theory

Abstract:
In this paper, we propose the *geometric invariance hypothesis (GIH)*, which argues that the input space curvature of a neural network remains invariant under transformation in certain architecture-dependent directions during training. We investigate a simple, non-linear binary classification problem residing on a plane in a high dimensional space and observe that&#151;unlike MPLs&#151;ResNets fail to generalize depending on the orientation of the plane. Motivated by this example, we define a neural network's **average geometry** and **average geometry evolution** as compact *architecture-dependent* summaries of the model's input-output geometry and its evolution during training. By investigating the average geometry evolution at initialization, we discover that the geometry of a neural network evolves according to the data covariance projected onto its average geometry. This means that the geometry only changes in a subset of the input space when the average geometry is low-rank, such as in ResNets. This causes an architecture-dependent invariance property in the input space curvature, which we dub GIH. Finally, we present extensive experimental results to observe the consequences of GIH and how it relates to generalization in neural networks.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729400896494

thread_type:
official_review

<thread_object>
creator: reviewer_sttw

<content>
summary:
The authors study the input-space geometry of neural networks, i.e., the decision boundaries and loss surface at each point in the input space. They give mathematical characterizations of the expected geometry and dynamics of the geometry under gradient flow at initialization. They make some intuitive conjectures about how the geometry evolves under training, one of which is the Geometric Invariance Hypothesis, which says that the geometry of the model remains invariant in "un-important" directions of the input space while training. They then experimentally verify parts of their conjectures using some representative architectures on both synthetic data and smaller-scale real data.

strengths:
- The examination of input-space geometry of neural networks is a more novel approach: other more mainstream approaches such as NTK and AGOP consider the parameter-space geometry.
- The conclusions are interesting, and to some degree, sensible: if there are directions in the input space which do not support the training data distribution, then the training dynamics may not move the input-space geometry in these directions.
- The paper does a good job motivating a very theoretical research direction with plausible practical implications in Section 5.
- The paper is reasonably well-written; despite the material being technically dense, it mostly makes sense after a thorough read.

weaknesses:
- The definitions of geometry in this work are based on expected second moments of the data. This captures the smallest supporting subspace of the data. But data can have very large linear dimension while having small intrinsic dimension, cf "manifold hypothesis". The theory here does not seem to capture this more involved geometric aspect, but this assumption has been hypothesized to be more realistic for real data. The lack of understanding here makes the experiments less likely to track with the theory and may preclude more prescriptive insights, beyond those generated by the preliminary experiments in Section 5.
- More minor: There is not much intuition provided about why the predictions made at initialization should transfer reasonably well to training, and thus (in my opinion) not too much motivation for the Conjectures (which are about training dynamics).

questions:
- The experiments remove normalization from the neural networks. Is this crucial? Does the trend totally deteriorate with normalization? Is there any hypothesis made by the theory as to why?
- Regarding the Conjectures 1 and 2 about dynamics: Is there an infinite- or large-width assumption being implicitly made, for instance? Should readers interpret the Conjectures as holding in appropriate scaling limits or should they hold at reasonable scales?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645719866

thread_type:
official_review

<thread_object>
creator: reviewer_twdw

<content>
summary:
This paper introduces the geometric invariance hypothesis (GIH), which suggests that during the training of neural networks, the curvature of the input space remains invariant under transformations along specific directions determined by the network’s architecture. The authors begin by examining a non-linear binary classification problem situated on a hyperplane in a high-dimensional space. They observe that a multilayer perceptron (MLP) can solve this problem regardless of the hyperplane’s orientation, whereas a residual network (ResNet) cannot. Motivated by this finding, they define two architecture-dependent maps: the average geometry, which provides a compact summary of the network’s input-space geometry, and the average geometry evolution, which describes how this geometry changes during training. Through both theoretical analysis and empirical investigations, the paper demonstrates that GIH arises because the average geometry evolution closely approximates the projection of the data covariance onto the average geometry. This results in an invariance property, especially when the average geometry is low-rank. Finally, the authors present extensive experimental results to explore the consequences of GIH and its relationship to generalization in neural networks.

strengths:
The introduction of the geometric invariance hypothesis (GIH) offers a fresh and innovative perspective on the interplay between neural network architectures and the geometry of the input space during training. By proposing the concepts of average geometry and average geometry evolution, the authors provide novel tools for quantifying how different architectures influence learning dynamics. This approach moves beyond traditional analyses by directly linking architectural properties to geometric transformations in the input space, which is a significant conceptual advancement in understanding deep learning models.

weaknesses:
While the paper makes significant contributions, there are areas that could be improved:

- Lack of Intuitive Explanation: It is challenging to develop an intuition for why ResNets behave differently from MLPs. Providing more intuitive explanations or illustrative examples before introducing the mathematical formalism would help readers grasp the core concepts and follow the subsequent analysis more effectively.
- Limited Architectural Comparison: The focus on ResNets without discussing other architectures like AlexNet leaves some gaps. Clarifying whether the observed behaviors are due to specific features like skip connections or are common across different architectures would strengthen the generality of the findings.

questions:
I am interested in how your observations are affected by different training dynamics, specifically when using stochastic gradient descent (SGD) with large learning rates—a common practice for achieving high prediction accuracy. Have you explored how the geometric invariance hypothesis (GIH) and the behavior of average geometry and its evolution manifest under such training conditions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683920319

thread_type:
official_review

<thread_object>
creator: reviewer_xfyb

<content>
summary:
In their paper, the authors argue for the link between input-space geometry and the ability of models to generalize. They propose the geometric invariance hypothesis (GIH) which states that, depending on the architecture of the model, the curvature of the input-space remains invariant under transformations in certain directions. They then attempt to prove and study this phenomenon by separating the changes in the input-space geometry during training into architecture-dependent changes and data-dependent changes. This resulted in finding the special cases where the invariance occurs, and they link this to the generalization ability of the model and where it collapses to reliance on noise and memory.

strengths:
The paper is able to gradually build up to the main hypothesis being proposed while maintaining a clear chain of reasoning. The authors also provide extensive mathematical proofs for each step in the build-up and mention what assumptions are made and any limitations on what can be shown or derived. Finally, they are able to provide some insight into the effect of this hypothesis on an architecture's generalization ability while addressing any possible ideas with empirical results.

weaknesses:
While the "performance" gains of the paper do seem marginal, I see these experiments as more of a proof of concept of the ideas and the proposed hypothesis. However, it would have been nice to see these experiments on multiple datasets to verify if the claims still hold, especially given the simplicity of the current model choices as well.

questions:
Just as a small note, in line 211 you mention the interaction between two factors, specifically the data-dependent factor and another. I'm guessing this is the model's geometric inductive bias? If so, I think it could be written a bit clearer. 

Additionally, would you think this approach would be useful for architecture-based optimal dataset subsampling methodology?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710286553

thread_type:
official_review

<thread_object>
creator: reviewer_wqbe

<content>
summary:
The paper studies how the geometry of neural network predictors evolves in the input space during training. It proposes the Geometric Invariance Hypothesis (GIH), which posits that this evolution occurs within a constrained subspace of the input space determined by the network architecture. The GIH is supported through experiments with MLPs, CNNs, and ViTs on a subset of CIFAR with binarized labels. The paper also studies the link between the GIH and generalization.

strengths:
1. The introduction of the Geometric Invariance Hypothesis appears novel and extends findings of Neural Anisotropy Directions (Ortiz-Jimenez et al., 2021) to non-linear decision boundaries. This hypothesis has the potential to provide insights into the relationship between neural network architecture and the structure of data, contributing to our understanding of inductive biases in deep learning.

2. The experiments and the theoretical analysis are generally fair, although several imprecisions are present, and some clarifications are needed (see detailed points below).

3. The theoretical findings are tested using real networks (e.g., ResNets, ViTs) and datasets (CIFAR). These experiments appear to align with and support the GIH. Nevertheless, it is worth noting that the considered settings are highly overparameterized and relatively simple, which raises concerns regarding the broader validity and general significance of the results (see detailed points below).

weaknesses:
The paper is quite dense. There are multiple points of confusion and imprecisions affecting both clarity and soundness. Specifically:

4. The introduction and main text lack a comprehensive overview of the field and references to related work. Only a few broad papers are cited, despite the extended page limit of this year's edition. I strongly encourage the authors to move much of the discussion from Appendix A.1 into the main text to better place the work in context. In particular, NADs introduced in Ortiz-Jimenez et al. (2021) should be explicitly discussed in the main text, given how strongly related they are to this work.

5. Lines L058+ seem to confuse expressivity with generalization, i.e., approximation vs. statistical properties. The fact that an MLP is universal (as some deep and wide CNNs are) does not imply that it can generalize effectively. 

6. Assuming i.i.d. parameters is not enough to conclude that $G_F \propto I$. A Gaussian distribution with a non-zero mean is a counterexample.

7. The claim that MLPs have a lack of inductive bias (L210, L224, L249) is wrong. Even in the NTK regime, MLPs display a strong spectral bias for low modes. It may be more accurate to state that MLPs at initialization have no preferred directions in the input space, i.e., they have an isotropic prior in the input space.

8. Multiple terms are left undefined or used imprecisely: e.g., “SSE loss” and $y_{\mu}$. Some phrases are vague: What is the “input-space curvature”, and how can it depend on the training process (L011)? What is the “input geometry” of a neural network (L076, L521)? What does it mean for a dimension to be noise (L053)? What are “all the possible values of $\theta$” ( L145)? How are they distributed? Given that you consider gradient-flow in your analysis, how do you define the “first step” in L286?

questions:
9. From Figure 1, it seems that simple (and standard) early stopping would solve the generalization gap problem and actually result in a lower test loss for dataset $D_B$ compared to dataset $D_A$. Can you comment on this? How general is this observation? 

10. What’s the underlying distribution over which you are taking the expectation at L227? It should be made explicit.

11. When you write “momentums” (L244, L246) do you mean “moments” instead?

12. Is the fact that $\Delta_F$ becomes label-independent purely due to the fact that you assume $\mathbb{E}_\theta [\theta]=0$ at initialization?

13. Could the phenomena observed in the paper, particularly when testing conjecture 2 about the GIH, be due to the fact that networks in all experiments are strongly overparameterized (even with CIFAR, the paper considers only a small fraction of the full dataset) and training in the “lazy” or NTK regime where weights stay close to their initial values? Did the authors measure the weight evolution throughout training? Would you expect the same results when training with more data, e.g., the full CIFAR10 dataset, and/or with tasks requiring learning latent features of the data that might be less dependent on the exact geometry and statistics in input space, e.g., more complex image datasets such as ImageNet?

14. Could the authors comment on Conjecture 2 in the case of linear regression? Intuitively, it seems it should hold in that simpler setting.

15. What is the motivation behind adding label noise (L406)? Would the same results or trends be observed in its absence?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732551699405

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response by the Authors

comment:
We thank all the reviewers for their helpful suggestions and interesting and engaging comments. We have provided our responses to the reviewers, along with a revised version of the paper, in which we addressed their concerns with additional experiments and discussions. All of the edited or newly added sections have been distinguished from the rest of the paper with the color blue. Specifically, aside from fixing the typos and some of the text requiring clarification, we have also added a small related works section to the introduction following Reviewer #1’s demand. On the other hand, following Reviewer #1’s comment on the plots in Figure 1, we have replaced the test loss plot with the test accuracy plot, which would prevent any misunderstanding regarding the results of the experiment. 

Also, given the comments of Reviewer #1 and Reviewer #2 about the effect of datasize on the results of the paper, we addressed this concern in a newly added appendix section (App. A.8). Furthermore, following Reviewer #3’s question regarding the effect of learning rate on the results of the paper, we have added another appendix section (App. A.9) to the paper addressing this question fully. Also, we have added another section to address Reviewer #3’s concern regarding an intuitive example in the appendix (App. A.10), where we provide a simple example of a model with a low-rank average geometry and how GIH predicts its behavior during training.

On a final note, we would like to point out that we are currently waiting on experiment results on ResNet9 without skip-connection and Tiny ImageNet in order to further diversify our experiment results. We will add the experiment results here as soon as they are available.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper offers an innovative approach to understanding generalization in machine learning through geometry. The introduction of the Geometric Invariance Hypothesis could inspire further research into input-space properties and their role in model performance. While some aspects require further elaboration and validation, the work provides a solid foundation for exploring new directions in theoretical and empirical studies of generalization.