On the Anisotropy of Score-Based Generative Models

## Abstract

Abstract We investigate the role of network architecture in shaping the inductive biases of modern score-based generative models. To this end, we introduce the Score Anisotropy Directions (SADs), architecture-dependent directions that reveal how different networks preferentially capture data structure. Our analysis shows that SADs form adaptive bases aligned with the architecture’s output geometry, providing a principled way to predict generalization ability in score models prior to training. Through both synthetic data and standard image benchmarks, we demonstrate that SADs reliably capture fine-grained model behavior and correlate with downstream performance, as measured by Wasserstein metrics. Our work offers a new lens for explaining and predicting directional biases of generative models.

## 1 Introduction

Figure: Figure 1: Sphere modeling in subspaces of $\mathbb{R}^{D}$ ($D=256$) via DiT (Peebles & Xie, 2023). The only difference is the choice of subspace: the left “sphere” lies in a subspace aligned with the network’s geometry, $\mathbf{G}_{\mathcal{F}}$, while the right is in a non-aligned subspace. Despite identical setups, their quality differs consistently across repeated trials, suggesting that alignment with architectural geometry controls generalization. We formalize these ideas in Sections [3.2](https://arxiv.org/html/2510.22899v1#S3.SS2) and [3.3](https://arxiv.org/html/2510.22899v1#S3.SS3).
Refer to caption: x1.png

Neural networks generalize through inductive biases, i.e., biases that guide learning beyond training data (Goyal & Bengio, 2020; Wilson & Izmailov, 2020). For discriminative tasks, they are partially characterized through the Neural Anisotropy Directions (NADs), which reveal the architecture’s directional preferences in the input space (Ortiz-Jimenez et al., 2020). However, generative modeling lacks a cohesive theory that explains how architectural geometry interacts with data manifolds (Kadkhodaie et al., 2024; An et al., 2025). In this work, we present a unified approach to explaining and interpreting inductive biases of score-based generative models by examining anisotropy in the output space, where networks exhibit preferential learning along certain directions. As motivated in Figure [1](https://arxiv.org/html/2510.22899v1#S1.F1), we posit that generalization ability is largely characterized by the alignment of the data with the architecture’s “geometry” at initialization. Our contribution is making this notion of “geometry” precise and decomposing the output space in terms of the anisotropy directions induced by it, i.e., the Score Anisotropy Directions (SADs) that we introduce below:

###### Definition 1 ( Score Anisotropy Directions ) .

###### Definition 1 ( Score Anisotropy Directions ) .

## 2 Background

Here we give a brief overview of score-based generative models and provide some context on existing work that examines inductive biases of deep neural networks. We defer an extended discussion and comparison with related work to Section [4.1](https://arxiv.org/html/2510.22899v1#S4.SS1).

### 2.1 Score-Based Generative Models

At the heart of score-based modeling is the score function, i.e., the gradient of the log-density of the underlying data distribution $p$. Given an estimate of the score, one can, in theory, sample from $p$ via a gradient ascent procedure. Starting with an arbitrary prior, $\bm{x}_{0}\sim\pi$, we have the following Langevin dynamics iterations:

$$ $\bm{x}_{k+1}=\bm{x}_{k}+\frac{\eta}{2}\overbrace{\nabla_{\bm{x}_{k}}\log p(\bm{x}_{k})}^{\text{``score"}}+\sqrt{\eta}\bm{z}_{k},\quad k=0,1,\dots,K$ (1) $$

where $\bm{z}_{k}\sim\mathcal{N}(\bm{0},\bm{I})$ is standard Gaussian and $\eta>0$. If $\eta\to 0$ and $K\to\infty$, under certain technical conditions, the iterates in Equation [1](https://arxiv.org/html/2510.22899v1#S2.E1) converge to a sample from $p$ (Welling & Teh, 2011). However, a practical limitation of the above setup is that estimated scores are inaccurate in low-density regions (e.g., in early iterations where learning is intractable). Moreover, score functions may be undefined in the case of data residing in low-dimensional manifolds. That is, the overall approach breaks down under the commonly adopted manifold hypothesis (Song & Ermon, 2019).

The research community has therefore largely moved on to Denoising Score Matching (DSM) (Vincent, 2011) and annealed Langevin dynamics, i.e., diffusion models, which are also the main focus of this work.(^1^11Without loss of generality, we will standardize our notation to the variance exploding formulation of Song & Ermon (2019).) Specifically, consider noise scales, $\sigma\in[\sigma_{\text{min}},\sigma_{\text{max}}]$, and associated densities, $p_{\sigma}=p*\mathcal{N}(\bm{0},\sigma^{2}\bm{I})$, where $*$ represents convolution. Here, $\sigma_{\text{min}}$ is small enough such that $p_{\sigma_{\text{min}}}\approx p$ and $\sigma_{\text{max}}$ is large enough so we can write $p_{\sigma_{\text{max}}}\approx\mathcal{N}(\bm{0},\sigma^{2}_{\text{max}}\bm{I})$. With estimates of scores of the perturbed distributions, $\nabla_{\bm{x}_{\sigma}}\log p_{\sigma}(\bm{x}_{\sigma})$, one samples from $p$ by decaying $\sigma$ from $\sigma_{\text{max}}$ to $\sigma_{\text{min}}$ (Ho et al., 2020; Song et al., 2021). This way, accurate modeling along sampling trajectories is tractable and the noise ensures support over the entirety of the ambient space, overcoming the limitations of naive Langevin dynamics. In particular, the scores are equivalent to minimum mean squared error Gaussian denoisers (Efron, 2011). That is, for neural networks, $\mathcal{F}_{\bm{\theta}}:\mathbb{R}^{D}\times\mathbb{R}\to\mathbb{R}^{D}$, parameterized by $\bm{\theta}$, one can approximate the scores via the following DSM optimization:

$$ $\min_{\bm{\theta}}\mathbb{E}_{\bm{x}\sim p,\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I}),\sigma}\left[\left\|\mathcal{F}_{\bm{\theta}}(\bm{x}+\sigma\bm{\epsilon},\sigma)+\frac{\bm{\epsilon}}{\sigma}\right\|^{2}_{2}\right].$ (2) $$

### 2.2 Inductive Biases in Deep Learning

There is a vast literature on understanding the inductive biases of deep neural networks (Goyal & Bengio, 2020; Wilson & Izmailov, 2020). Of particular interest is the work of Ortiz-Jimenez et al. (2020), who identify directional biases in classifiers, i.e., the Neural Anisotropy Directions (NADs). More recently, Movahedi et al. (2025) have extended the NAD framework by formalizing the concept of input-space architectural geometry and exploring its evolution during training. Specifically, they propose the geometric invariance hypothesis, which posits that NADs persist through training.

To our knowledge, contrary to the discriminative case, there is no unified theory that explains the preferred modeling directions of diffusion. For example, a recent study by Kadkhodaie et al. (2024) argues that convolutional diffusion models are biased towards Geometry-Adaptive Harmonic Bases (GAHBs). However, they acknowledge that a mathematically precise definition of such bases remains an open question. More fundamentally, follow-up work by An et al. (2025) suggests that the theory of GAHBs does not extend to transformer-based diffusion models. Specifically, they instead resort to a transformer-specific analysis by inspecting and manipulating attention maps.

In contrast to the above-mentioned works on the inductive biases of diffusion models, we present a more unified treatment of directional biases that is agnostic to the denoising network’s architecture. In particular, our key insight is recognizing that the NAD framework can be adapted and extended to score-based modeling by assuming an underlying data log-density that assigns high probability to “on-manifold” data and low probability otherwise. With this approach, our analysis amounts to understanding the anisotropy directions of such implicitly induced discriminative models.

## 3 Directional Inductive Biases of Diffusion Models

Figure: Figure 2: $\text{MSW}_{2}$ distance (computed over 10k test samples and 16384 projections) of iDDPM U-Net (Nichol & Dhariwal, 2021) architecture. Each pixel corresponds to a rank-one dataset of 16$\times$16 images (with 10k training samples) that is aligned with a basis element of the canonical basis, DCT, DST, (ordered) Hadamard transform or Haar wavelet transform. That is, for a given location (canonical) or frequency / sequency (DCT, DST, Hadamard) or scale, channel and location (Haar), we visualize the performance on the corresponding dataset. For ease of visualization, in the case of DCT, DST and Hadamard, we center the zero frequency dataset and extend the images to the left and top regions while respecting the symmetries of the transforms. See Appendix [A](https://arxiv.org/html/2510.22899v1#A1) for details.
Refer to caption: x2.png

#### Notation

Matrix entries are indexed as $(\cdot)_{(i,j)}$ for row $i$ and column $j$. Unless otherwise specified, we write the $i^{\text{th}}$ row of a matrix as $(\cdot)^{(i)}$. $\otimes$ is the Kronecker product. $\delta_{(\cdot)}$ denotes the Kronecker delta or Dirac delta distribution, depending on the context. The term iid refers to independent and identically distributed random variables. Gaussian distributions with mean $\bm{\mu}$ and covariance $\bm{\Sigma}$ are represented as $\mathcal{N}(\bm{\mu},\bm{\Sigma})$. $\mathcal{U}(S)$ is the uniform distribution over the set $S$. We abbreviate (Stochastic) Gradient Descent as (S)GD. Eigendecompositions of matrices are assumed to have the eigenvalues in descending order, so that the first eigenvector corresponds to the largest eigenvalue. We assume that all expectations appearing in our arguments are finite and well-defined.

Central to our exploration of directional inductive biases in diffusion is the following question:

Among equidimensional manifolds, which are preferred by diffusion-based generative modeling and how can such preferences be quantified?

To answer the above, we consider data manifolds aligned with a particular direction, $\bm{v}\in\mathbb{S}^{D-1}$, and investigate generalization ability as a function of the direction. Concretely, we study distributions of the form $\mathcal{N}(\bm{0},D\bm{v}\bm{v}^{\top})$ and are tasked with finding a suitable basis for $\mathbb{R}^{D}$ from which we can draw $\bm{v}$ that reveal directional preferences. We include complete details of our experimental setup in Appendix [A](https://arxiv.org/html/2510.22899v1#A1). In summary, for a given basis, we independently train diffusion models on datasets formed by each normalized element under identical settings. Performance is quantified via the (Max-)Sliced Wasserstein $p$-distance, $\text{(M)SW}_{p}$, between the distribution obtained by sampling from the trained model, $\mu$, and the ground truth data, corresponding to $\nu=\mathcal{N}(\bm{0},D\bm{v}\bm{v}^{\top})$. Notably, these are valid statistical distances and the slice distance, $\text{W}_{p}$, is easily estimated via order statistics and Monte Carlo methods. Taking $p=2$ and letting $F^{-1}_{(\cdot)}$, $(\cdot)_{\#}$ represent quantile functions, the push-forward respectively, we express the Wasserstein metrics theoretically as:

$$ $\begin{split}\text{SW}^{2}_{2}(\mu,\nu)=\mathbb{E}&{}_{\bm{\theta}\sim\mathcal{U}(\mathbb{S}^{D-1})}\text{W}^{2}_{2}(\bm{\theta}^{\top}_{\#}\mu,\bm{\theta}^{\top}_{\#}\nu),\quad\text{MSW}^{2}_{2}(\mu,\nu)=\sup_{\bm{\theta}\in\mathbb{S}^{D-1}}\text{W}^{2}_{2}(\bm{\theta}^{\top}_{\#}\mu,\bm{\theta}^{\top}_{\#}\nu),\\ &\text{W}^{2}_{2}(\bm{\theta}^{\top}_{\#}\mu,\bm{\theta}^{\top}_{\#}\nu)=\int_{0}^{1}|F^{-1}_{\bm{\theta}^{\top}_{\#}\mu}(q)-F^{-1}_{\bm{\theta}^{\top}_{\#}\nu}(q)|^{2}\text{d}q.\end{split}$ (3) $$

Figure: Figure 3: Responses of the iDDPM architecture (Nichol & Dhariwal, 2021) with a symmetrical initialization scheme. We show the default implementation, which uses nearest resampling layers, and a modified architecture that uses area resampling. We probe the models with a centered impulse input, shown on the left. Observe that the default resampling introduces asymmetry.
Refer to caption: x3.png

Figure: Figure 4: Setting of Theorem [1](https://arxiv.org/html/2510.22899v1#Thmtheorem1) ($\sigma=1$) in $\mathbb{R}^{5}$ with SGD. We show error between $\bm{\Omega}_{t}=\bm{\Phi}\bm{\Theta}_{t}$ and the optimal operator, $\bm{\Omega}^{*}$, defined in Lemma [1](https://arxiv.org/html/2510.22899v1#Thmlemma1).
Refer to caption: x4.png

In Figure [2](https://arxiv.org/html/2510.22899v1#S3.F2) we report our findings (averages over five runs) with the above-described approach for common bases found in signal processing literature. Interestingly, contrary to the conventional wisdom that neural networks better adapt to lower frequencies (Rahaman et al., 2019), the standard U-Net used in DDPMs (Nichol & Dhariwal, 2021) struggles with low-frequency data. This is evidenced by the center points of the DCT, DST, Hadamard images and the top-left corner of the Haar image. Also, comparing results of the canonical and Haar bases with the frequency / sequency transforms, we see that vectors localized in space are better modeled, especially around the borders.

### 3.1 Anisotropic Conditioning of the Optimization Landscape

Why would certain directions be preferred? While the learning process has a number of hyperparameters that can potentially induce asymmetry, here we specifically focus on analyzing the role of the architecture. Prior work on the inductive biases of discriminative models argues that preferences may emerge due to anisotropic loss of information or, more generally, conditioning of the optimization landscape (Ortiz-Jimenez et al., 2020). Indeed, such conditioning also naturally manifests in multiscale U-Nets, for example, in the form of asymmetric resampling layers. We verify this in Figure [3](https://arxiv.org/html/2510.22899v1#S3.F3), where we observe that the default nearest interpolation leads to a clear directional bias. Similarly, we hypothesize that the border effects observed in the canonical and Haar basis experiments of Figure [2](https://arxiv.org/html/2510.22899v1#S3.F2) can be attributed to the padding employed in the convolutional layers. Intuitively, we expect data aligned with such padding artifacts to be poorly approximated by the network.

To more rigorously understand the effect of anisotropic conditioning, we now revisit the problem of learning rank-one distributions, introduced in the beginning of Section [3](https://arxiv.org/html/2510.22899v1#S3), from a theoretical perspective. Assuming a linear DSM architecture, where anisotropy is explicitly modeled via a fixed, linear transformation with decaying eigenvalues at the output, we make the following observation:

###### Theorem 1 (DSM under anisotropy, proof in Appendix C.2 ) .

Consider DSM with data drawn from $\mathcal{N}(\bm{0},\bm{v}\bm{v}^{\top})$ for a fixed $\sigma>0$ and $\|\bm{v}\|_{2}=1$. Let $\mathcal{F}:\mathbb{R}^{D}\to\mathbb{R}^{D}$ be linear networks expressed as $\bm{\Omega}(\cdot)$, where $\bm{\Omega}=\bm{\Phi}\bm{\Theta}$ with $\bm{\Phi}$ fixed. Denote the sorted eigenvalues of $\bm{\Phi}\bm{\Phi}^{\top}$ as $\{\lambda_{i}\}^{D}_{i=1}$ with $\lambda_{D-1}>\lambda_{D}>0$ and the corresponding normalized eigenvectors as $\{\bm{u}_{i}\}^{D}_{i=1}$. Assume a (S)GD procedure on initially zero-mean $\bm{\Theta}$, where the score is approximated by $\mathcal{F}$. After $t$ steps, and for small learning rate $\eta>0$, the mean error to the optimal solution for $\bm{v}=\bm{u}_{i}$ decays as $\mathcal{O}[(1-2\eta\rho_{i})^{t}]$ with $\rho_{1}=\rho_{i}<\rho_{D}$ $\forall i<D$. Moreover, near optimality, SGD steps for $\bm{v}=\bm{u}_{i}$ have covariance $\propto\lambda_{i}$.

Curiously, the effect of anisotropy is limited in GD, i.e., if $\lambda_{D}=\lambda_{D-1}$ then it completely disappears and convergence is the same for all eigenvectors of $\bm{\Phi}\bm{\Phi}^{\top}$. However, our derivations suggest that anisotropy is greatly amplified in SGD. To confirm, we design a experiment in $\mathbb{R}^{5}$, with the results in Figure [4](https://arxiv.org/html/2510.22899v1#S3.F4). Specifically, for small $t$, the optimization dynamics are well-predicted by the deterministic GD analysis of Theorem [1](https://arxiv.org/html/2510.22899v1#Thmtheorem1) since the iterates are far from the optimum and the deterministic drift, i.e., the gradient, dominates the stochastic fluctuation. Consequently, all vectors except $\bm{u}_{5}$ yield similar results. For large $t$, however, where the gradient norm is sufficiently small, noise dominates and it appears we have convergence to a stationary distribution. In such regimes, the error scales with the stochastic gradient covariance (Mandt et al., 2017), which we show to be $\propto\lambda_{i}$ near optimality.

The takeaway from this discussion is that the best performance is achieved when the data is not aligned with the “geometry” that is induced by the score network, but instead lives in the subspace defined by its smallest eigenvalues. Next, we will extend and formalize this intuition in a more general setting, for potentially non-linear architectures.

###### Theorem 1 (DSM under anisotropy, proof in Appendix C.2 ) .

### 3.2 Identifying Preferred Directions for Generative Modeling

Although the experiments of Figure [2](https://arxiv.org/html/2510.22899v1#S3.F2) are insightful, it is unclear whether standard bases can reliably describe the biases of diffusion models given that they are completely decoupled from the underlying architecture. As the choice of bases was arbitrary in these initial experiments, we argue that, in general, we cannot hope to uncover the intricacies of directional biases in this manner. Moreover, note that each considered basis amounts to training hundreds of diffusion models, which becomes impractical even for relatively small-dimensional data. We now present a more principled approach to these experiments, by establishing theoretically motivated and architecture-dependent bases that predict directional biases prior to any training.

Figure: Figure 5: Visualization of our argument for uncovering anisotropy directions in $\mathbb{R}^{2}$. We show contours of a hypothetical landscape, $\log p_{\bm{\theta},\sigma}(\bm{x}_{\sigma})$, where $\bm{v}_{\parallel}$ is parallel to the induced “manifold” and $\bm{v}_{\perp}$ is orthogonal.
Refer to caption: x5.png

Concretely, for each noise level, $\sigma$, and assuming a (conservative and normalizable) parameterization via a neural network family, $\mathcal{F}$, we intuitively treat a realization, $\mathcal{F}_{\bm{\theta}}:\mathbb{R}^{D}\times\mathbb{R}\to\mathbb{R}$, as implicitly defining a log-density function, $\bm{x}_{\sigma}\mapsto\log p_{\bm{\theta},\sigma}(\bm{x}_{\sigma})$, that assigns high probability to $\bm{x}_{\sigma}$ on the “data manifold” and low probability to “off-manifold” data, i.e., we can write $\mathcal{F}_{\bm{\theta}}(\bm{x}_{\sigma},\sigma)\approx\nabla_{\bm{x}_{\sigma}}\log p_{\bm{\theta},\sigma}(\bm{x}_{\sigma})$. Here, we use the term “manifold” loosely, meaning regions of $\mathbb{R}^{D}$ that are statistically likely to be sampled by score-based modeling via $\mathcal{F}_{\bm{\theta}}$, i.e., the spanning set of the top SADs that we aim to uncover. Now, intuitively, if one were to fix $\bm{x}_{\sigma}$ and consider a small perturbation along some direction, $\bm{v}$, an abrupt change in the log-density indicates falling off or entering the “manifold”, that is, $\bm{v}=\bm{v}_{\perp}$ is perpendicular to the “manifold”. Similarly, if $\bm{v}=\bm{v}_{\parallel}$ is parallel to the “manifold”, then we expect minimal changes in the log-density, i.e., we are traveling along a contour line. We illustrate our argument in Figure [5](https://arxiv.org/html/2510.22899v1#S3.F5). Now, taking the limit as the perturbation magnitude, $\tau$, tends to zero, we summarize these dynamics via directional derivatives, i.e., we have $|\log p_{\bm{\theta},\sigma}(\bm{x}_{\sigma}+\tau\bm{v})-\log p_{\bm{\theta},\sigma}(\bm{x}_{\sigma})|\propto|\bm{v}^{\top}\nabla_{\bm{x}_{\sigma}}\log p_{\bm{\theta},\sigma}(\bm{x}_{\sigma})|$. With this simplification, a straightforward application of Markov’s inequality bounds the a priori probability of crossing the “manifold” by moving along $\bm{v}$, suggesting that directions attaining the minimum upper bound are inherently easier to model. That is, such directions are aligned with the prior density induced by $\mathcal{F}_{\bm{\theta}}$. Specifically, for the family of networks, $\mathcal{F}$, parameterized by $\bm{\theta}\sim\Theta$ over noise levels, $\sigma$, and probing with $(\bm{x}_{\sigma},\sigma)\sim\mathcal{P}$, we write the bound:

$$ $\mathbb{P}\bigl(\bigl|\bm{v}^{\top}\nabla_{\bm{x}_{\sigma}}\log p_{\bm{\theta},\sigma}(\bm{x}_{\sigma})\bigr|\geq\eta\bigr)\;\leq\;\frac{\bm{v}^{\top}\overbrace{\Bigl[\mathbb{E}_{(\bm{x}_{\sigma},\sigma)\sim\mathcal{P},\,\bm{\theta}\sim\Theta}\mathcal{F}_{\bm{\theta}}(\bm{x}_{\sigma},\sigma)\,\mathcal{F}_{\bm{\theta}}(\bm{x}_{\sigma},\sigma)^{\top}\Bigr]}^{\text{``geometry"}}\,\bm{v}}{\eta^{2}}.$ (4) $$

We note that by applying this bound on the setting of Theorem [1](https://arxiv.org/html/2510.22899v1#Thmtheorem1) with $\bm{\theta}$ iid and for any $\mathcal{P}$ the quantity under the overbrace, namely the average geometry, recovers the conditioning matrix, $\bm{\Phi}\bm{\Phi}^{\top}$, whose eigendecomposition defined the SADs in the case of linear networks. Moreover, the Markov bound correctly predicts that the small-eigenvalue vectors are easier to model compared to larger-eigenvalue eigenvectors. We therefore expect that this quantity also captures directional preferences of more general architectures. As the average geometry is central to our study of directional biases, let us formally introduce it below.

###### Definition 2 ( Average Geometry ) .

Figure: (a)
Refer to caption: x6.png

At this point it is important to note that, unlike in the linear networks investigated in Theorem [1](https://arxiv.org/html/2510.22899v1#Thmtheorem1), the notion of average geometry is, in general, local and adapts to the probing data distribution $\mathcal{P}$. To decouple the geometry from the data, one may consider an isotropic probe, e.g., $\mathcal{N}(\bm{0},\sigma^{2}_{\mathcal{P}}\bm{I})$. However, in practice, we find that the probe is a non-critical hyperparameter and not central to our analysis. For simplicity, we default to $\mathcal{P}=\delta_{\bm{0}}\times\mathcal{U}(\{\sigma_{\text{min}},\dots,\sigma_{\text{max}}\})$, i.e., we fix $\bm{x}_{\sigma}=\bm{0}$ and draw $\sigma$ uniformly from the noise levels of interest. We shall write the geometry under this default probe as $\mathbf{G}_{\mathcal{F}}$, where the parameters, $\bm{\theta}$, of $\mathcal{F}$ are assumed to be drawn from the default initialization scheme for the architecture. Having formalized the concept of output-space geometry, we are now ready to state our main conjecture regarding the SADs, linking them to the average geometry at initialization:

###### Conjecture 1 .

In the remainder of the paper, we provide empirical evidence to justify Conjecture [1](https://arxiv.org/html/2510.22899v1#Thmconjecture1). First, we verify our claims on the rank-one datasets introduced in the beginning of Section [3](https://arxiv.org/html/2510.22899v1#S3) in Figure [6](https://arxiv.org/html/2510.22899v1#S3.F6), where we draw the directions, $\bm{v}\in\mathbb{R}^{D}$, from the eigenvectors of $\mathbf{G}_{\mathcal{F}}$ and benchmark via Wasserstein metrics.

On the left in Figure [6](https://arxiv.org/html/2510.22899v1#S3.F6), we focus on the iDDPM U-Net (Nichol & Dhariwal, 2021), which is representative of convolutional diffusion models. The experiments show a clear trend in support of the conjecture, where eigenvectors corresponding to small eigenvalues achieve the best performance and large-eigenvalue vectors have the worst performance. Interestingly, with reference to the visualizations below the plots, we also observe harmonic patterns in the eigenvectors, where large eigenvalues correspond to low-frequencies and small eigenvalues to high-frequencies. In this sense, our findings provide further evidence in support of the theory of GAHBs (Kadkhodaie et al., 2024), which posits that harmonic representations are fundamental in convolutional models.(^2^22See Figure [7](https://arxiv.org/html/2510.22899v1#S3.F7) for more visualizations that connect our work with Kadkhodaie et al. (2024).) Crucially, beyond the harmonic structure that only loosely characterizes the inductive biases, our notion of geometry is flexible as it captures and adapts to architectural details. This is evident in the first eigenvectors, $\bm{u}_{1}$, when we ablate the resampling method, also investigated previously in Figure [3](https://arxiv.org/html/2510.22899v1#S3.F3).

As further proof of the flexibility of our proposed average geometry, we now focus our analysis on the DiT architecture (Peebles & Xie, 2023), which is representative of transformer-based diffusion models. The results, shown on the right in Figure [6](https://arxiv.org/html/2510.22899v1#S3.F6), are also in support of Conjecture [1](https://arxiv.org/html/2510.22899v1#Thmconjecture1). However, as also noted in (An et al., 2025), it is clear in this case that the transformer architecture does not induce harmonic representations and the theory of GAHBs does not apply. It is also interesting to observe that the DiT geometry exhibits considerable eigen-multiplicity, with larger patch sizes amplifying this, reflecting a looser structure and weaker inductive biases compared to convolutional networks. In fact, in Proposition [3](https://arxiv.org/html/2510.22899v1#Thmproposition3) we show that the number of transformer tokens, $T$, is an upper bound on the number of distinct eigenvalues of $\mathbf{G}_{\mathcal{F}}$. More generally, we refer the reader to Appendix [B](https://arxiv.org/html/2510.22899v1#A2) for an analytical treatment of geometries of common architectures, noting that a surprising amount of structure can be inferred simply by inspecting the output layers.

Figure: Figure 7: Eigendecomposition of iDDPM average geometry, $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)$, at initialization with probing distribution $\mathcal{P}=\mathcal{N}(\bm{x},\sigma^{2}\bm{I})\times\mathcal{U}(\{\sigma_{\text{min}},\dots,\sigma_{\text{max}}\})$, i.e., we probe along the standard forward diffusion process with the data sample $\bm{x}$. The input image, $\bm{x}$, is shown on the left and the first few eigenvectors, together with the corresponding unnormalized eigenvalues, are shown on the right. Despite our approach being fundamentally different from existing works, the results resemble the GAHBs of Kadkhodaie et al. (2024). Specifically, we see that our eigenvectors are also adaptive to the geometry of the input image. Moreover, they have harmonic structure as we also observe oscillating patterns whose frequency increases as the eigenvalue decreases.
Refer to caption: x14.png

###### Definition 2 ( Average Geometry ) .

###### Conjecture 1 .

### 3.3 Score Anisotropy Directions in the Wild

Having identified the preferred modeling directions in our experiments on rank-one datasets, we now turn to more realistic data distributions that are encountered in practice. Based on our analysis in Section [3.2](https://arxiv.org/html/2510.22899v1#S3.SS2), we hypothesize that generalization is largely determined by the (mis)alignment of the data with the average geometry. For an arbitrary data distribution, $p$, we can extend our setup by defining the alignment with the network, $\alpha$, as follows:

$$ $\alpha:=\mathbb{E}_{\bm{x}\sim p}[\bm{z}^{\top}\mathbf{G}_{\mathcal{F}}\bm{z}],\quad\bm{z}=\bm{W}\bm{x},\quad\bm{W}^{\top}\bm{W}=\bm{I}.$ (6) $$

Note, in order to vary $\alpha$ in a way that preserves underlying structure, we have introduced the orthogonal matrix $\bm{W}\in\mathbb{R}^{D\times D}$, which models simple and lossless data transformations, effectively defining a linear autoencoder and a kind of “latent” diffusion model that operates on the transformed data. With this setup, Conjecture [1](https://arxiv.org/html/2510.22899v1#Thmconjecture1) amounts to the claim that the best performance is observed when $\alpha$ is minimized and the worst when $\alpha$ is maximized. The corresponding orthogonal matrices, $\bm{W}_{\min}$ and $\bm{W}_{\max}$, that achieve such extreme (mis)alignment are given by Theorem [2](https://arxiv.org/html/2510.22899v1#Thmtheorem2):

###### Theorem 2 (Extreme alignment, proof in Appendix C.3 ) .

Stacking the (descending by eigenvalue magnitude) eigenvectors of $\mathbf{G}_{\mathcal{F}}$ and $\mathbb{E}_{\bm{x}\sim p}[\bm{x}\bm{x}^{\top}]$ as columns in matrices $\bm{U},\bm{V}\in\mathbb{R}^{D\times D}$ respectively, $\alpha$ is minimized for $\bm{W}_{\min}=\bm{U}\bm{J}\bm{V}^{\top}$ and maximized for $\bm{W}_{\max}=\bm{U}\bm{V}^{\top}$. Here, $\bm{J}$ is the row-reversed identity matrix, i.e., it reverses the order of the columns in $\bm{V}$.

Figure: (a)
Refer to caption: figures/mnist-ascending.png

Figure: Figure 9: iDDPM average performance ($\text{SW}_{2}$) over five independent runs on standard image datasets as the alignment with the geometry, $\alpha$, varies. We show the effect of minimizing, maximizing $\alpha$ as well as the default setting. These experiments correspond to matrices $\bm{W}_{\min}$, $\bm{W}_{\max}$ and $\bm{I}$ respectively.
Refer to caption: x15.png

Therefore, to test our hypothesis, for each $\bm{W}$, we will train such “latent” diffusion models under identical settings. To compare with the standard diffusion training, we also consider a baseline corresponding to $\bm{W}=\bm{I}$, i.e., the natural alignment of the data with the average geometry. In particular, we conduct experiments on the MNIST (LeCun et al., 2010) ($28\times 28$), CelebA-HQ (Karras et al., 2018) ($56\times 56$) and CIFAR-10 (Krizhevsky, 2009) ($32\times 32$) datasets, with implementation details included in Appendix [A](https://arxiv.org/html/2510.22899v1#A1). Our findings, shown in Figures [8](https://arxiv.org/html/2510.22899v1#S3.F8) and [9](https://arxiv.org/html/2510.22899v1#S3.F9), agree with our central conjecture. For example, focusing on the MNIST samples obtained by our models, which are shown in Figure [8](https://arxiv.org/html/2510.22899v1#S3.F8), we observe that large $\alpha$ results in artifacts on the distribution-level.(^3^33Note that, in general, we do not expect perceptual quality to correlate with the Wasserstein distances.) Specifically for the default alignment, which is already significant, a considerable fraction of the samples do not contain any digit. Moreover, maximizing $\alpha$ leads to mode collapse, with the digit “1” being overrepresented. When we instead minimize $\alpha$, substantial and consistent improvements are observed across all datasets, as quantified by the Wasserstein metrics in Figure [9](https://arxiv.org/html/2510.22899v1#S3.F9). Interestingly, we find that, by default, the considered datasets are already well-aligned with $\mathbf{G}_{\mathcal{F}}$ and our analysis shows that maximizing $\alpha$ yields similar scores to the baseline $\bm{W}=\bm{I}$. In particular, since both $\bm{W}=\bm{W}_{\max}$ and $\bm{W}=\bm{I}$ exhibit high alignment, their performance differences may not be statistically significant. The more meaningful comparison is with $\bm{W}=\bm{W}_{\min}$, where alignment is minimal and the Wasserstein metrics are substantially improved.

###### Theorem 2 (Extreme alignment, proof in Appendix C.3 ) .

## 4 Discussion

We presented and validated a theory for determining the directional biases of diffusion models. We find that, despite the highly non-linear nature of modern networks, these biases are well-described via architecture-specific, fixed bases that are decoupled from the data. Specifically, we showed that the directions defined by these bases impose strong priors during the training process and define the order by which features of the data are learned, i.e., we identify them as the SADs.

### 4.1 Related Work

Kadkhodaie et al. (2024) observe that convolutional diffusion models are inductively biased toward GAHBs. Specifically, by assuming a homogeneous model (Mohan et al., 2020), they give a shrinkage-based interpretation of denoising with adaptive eigenbases given by the local Jacobian of the trained networks. A similar Jacobian-based analysis has also been explored in transformer-based diffusion (An et al., 2025). However, the observations of Kadkhodaie et al. (2024) are architecture-specific and largely empirical, therefore they cannot be used to predict preferred modeling directions in general. For example, transformer-based diffusion exhibits no obvious regularities in the eigenbases, such as harmonic structure, that could be systematically exploited in An et al. (2025). Despite using a fundamentally different approach, our theory of SADs appears to extend and is compatible with the findings of these works. In particular, we provide a more rigorous treatment of architectural geometry, decoupling it from the data and we demonstrate a method that can predict generalization ability prior to any training. We achieve this by decomposing and, crucially, ordering the output space via the SADs, which are defined explicitly. This ordering provides fine-grained information about inductive biases, rather than a loose characterization. Moreover, unlike Jacobian-based analyses, we make no assumptions regarding homogeneity, which may be suboptimal / not hold across domains or architectures, and our geometry only requires forward passes through the networks.

We note that, with some variations, a notion similar to our average geometry also appeared previously in Ortiz-Jimenez et al. (2020); Movahedi et al. (2025), who study the directional inductive biases of discriminative networks. Interestingly, their analysis predicts that classifiers actually perform better for data aligned with the largest eigenvalues of the geometry. Though their setup is not identical to ours, and therefore not directly comparable, we believe that these seemingly contrasting conclusions may hint at a fundamental trade-off between discriminative and generative modeling. However, such an investigation is out of the scope of this paper and we leave it for future research.

### 4.2 Limitations

Similarly to Kadkhodaie et al. (2024), our experiments, while thorough, focus on relatively small scale settings, which may not reflect the current state-of-the-art. This choice was deliberate: a rigorous validation of our claims at large scales is computationally prohibitive and would require significantly more resources and time. Ultimately, we have prioritized insight as opposed to completeness. Moreover, it is important to stress that we have specifically studied directional inductive biases imposed by the architecture. In principle, the overall dynamics of diffusion models may also be influenced or dominated by different factors such as explicit regularization or other implicit priors, e.g., the double descent phenomenon (Nakkiran et al., 2020). In general, we do not make any claims regarding foundation models nor do we claim to have a complete theory of inductive biases.

### 4.3 Broader Impact

We investigate the directional inductive biases inherent in score-based generative models and examine their influence on the approximation capabilities of neural networks. We propose a method to systematically characterize these biases, offering insights into both the successes and potential limitations of modern diffusion models. Although our focus is primarily theoretical, understanding these mechanisms may guide the design of more effective and efficient generative modeling algorithms.

Specifically, deep learning is an empirical science enabled by scale and heavy reliance on heuristics. We believe our insights and further developments along this line of research could allow for more cost-effective and principled development of generative technologies. For example, we see potential applications in AutoML (Bergstra et al., 2011) and neural architecture search (Zoph & Le, 2017).

Importantly, characterizing the inductive biases of generative models has implications for understanding and mitigating undesirable behaviors such as memorization of data (Carlini et al., 2023; Somepalli et al., 2023; Wen et al., 2024; Gu et al., 2025) and hallucinations (Aithal et al., 2024; Lu et al., 2025; Floros et al., 2025). By making explicit which patterns the model is predisposed to reproduce, our approach could help identify circumstances where models are likely to overfit or generate spurious content. This may inform the development of techniques aimed at reducing privacy risks and improving reliability. In this sense, we contribute not only to the theoretical understanding of generative modeling but also to the broader goal of creating safer and more trustworthy systems.

## References

- Aithal et al. (2024)
Sumukh K Aithal, Pratyush Maini, Zachary Chase Lipton, and J Zico Kolter.
Understanding hallucinations in diffusion models through mode interpolation.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=aNTnHBkw4T](https://openreview.net/forum?id=aNTnHBkw4T).
- An et al. (2025)
Jie An, De Wang, Pengsheng Guo, Jiebo Luo, and Alex Schwing.
On inductive biases that enable generalization in diffusion transformers, 2025.
- Bergstra et al. (2011)
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.
Algorithms for hyper-parameter optimization.
In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), *Advances in Neural Information Processing Systems*, volume 24. Curran Associates, Inc., 2011.
URL [https://proceedings.neurips.cc/paper_files/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf).
- Carlini et al. (2023)
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace.
Extracting training data from diffusion models.
In *Proceedings of the 32nd USENIX Conference on Security Symposium*, SEC ’23, USA, 2023. USENIX Association.
ISBN 978-1-939133-37-3.
- Efron (2011)
Bradley Efron.
Tweedie’s formula and selection bias.
*Journal of the American Statistical Association*, 106(496):1602–1614, 2011.
ISSN 01621459.
- Floros et al. (2025)
Andreas Floros, Seyed-Mohsen Moosavi-Dezfooli, and Pier Luigi Dragotti.
Trustworthy image super-resolution via generative pseudoinverse.
In *ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy*, 2025.
URL [https://openreview.net/forum?id=UsYosqkokz](https://openreview.net/forum?id=UsYosqkokz).
- Goyal & Bengio (2020)
Anirudh Goyal and Yoshua Bengio.
Inductive biases for deep learning of higher-level cognition.
*Proceedings of the Royal Society A*, 478, 2020.
- Gu et al. (2025)
Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang.
On memorization in diffusion models.
*Transactions on Machine Learning Research*, 2025.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=D3DBqvSDbj](https://openreview.net/forum?id=D3DBqvSDbj).
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 6840–6851. Curran Associates, Inc., 2020.
- Kadkhodaie et al. (2024)
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat.
Generalization in diffusion models arises from geometry-adaptive harmonic representations.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=ANvmVS2Yr0](https://openreview.net/forum?id=ANvmVS2Yr0).
- Karras et al. (2018)
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of GANs for improved quality, stability, and variation.
In *International Conference on Learning Representations*, 2018.
- Krizhevsky (2009)
Alex Krizhevsky.
Learning multiple layers of features from tiny images, 2009.
- LeCun et al. (2010)
Yann LeCun, Corinna Cortes, and CJ Burges.
Mnist handwritten digit database.
*ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist*, 2, 2010.
- Lu et al. (2025)
Rui Lu, Runzhe Wang, Kaifeng Lyu, Xitai Jiang, Gao Huang, and Mengdi Wang.
Towards understanding text hallucination of diffusion models via local generation bias.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=SKW10XJlAI](https://openreview.net/forum?id=SKW10XJlAI).
- Mandt et al. (2017)
Stephan Mandt, Matthew D. Hoffman, and David M. Blei.
Stochastic gradient descent as approximate bayesian inference.
*Journal of Machine Learning Research*, 18(134):1–35, 2017.
URL [http://jmlr.org/papers/v18/17-214.html](http://jmlr.org/papers/v18/17-214.html).
- Mohan et al. (2020)
Sreyas Mohan, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos Fernandez-Granda.
Robust and interpretable blind image denoising via bias-free convolutional neural networks.
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=HJlSmC4FPS](https://openreview.net/forum?id=HJlSmC4FPS).
- Movahedi et al. (2025)
Sajad Movahedi, Antonio Orvieto, and Seyed-Mohsen Moosavi-Dezfooli.
Geometric inductive biases of deep networks: The role of data and architecture.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Nakkiran et al. (2020)
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever.
Deep double descent: Where bigger models and more data hurt.
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=B1g5sA4twr](https://openreview.net/forum?id=B1g5sA4twr).
- Nichol & Dhariwal (2021)
Alexander Quinn Nichol and Prafulla Dhariwal.
Improved denoising diffusion probabilistic models.
In Marina Meila and Tong Zhang (eds.), *Proceedings of the 38th International Conference on Machine Learning*, volume 139 of *Proceedings of Machine Learning Research*, pp. 8162–8171. PMLR, 18–24 Jul 2021.
- Ortiz-Jimenez et al. (2020)
Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi, and Pascal Frossard.
Neural anisotropy directions.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 17896–17906. Curran Associates, Inc., 2020.
- Peebles & Xie (2023)
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 4195–4205, October 2023.
- Rahaman et al. (2019)
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville.
On the spectral bias of neural networks.
In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), *Proceedings of the 36th International Conference on Machine Learning*, volume 97 of *Proceedings of Machine Learning Research*, pp. 5301–5310. PMLR, 09–15 Jun 2019.
URL [https://proceedings.mlr.press/v97/rahaman19a.html](https://proceedings.mlr.press/v97/rahaman19a.html).
- Somepalli et al. (2023)
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.
Diffusion art or digital forgery? investigating data replication in diffusion models.
In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 6048–6058, 2023.
doi: 10.1109/CVPR52729.2023.00586.
- Song & Ermon (2019)
Yang Song and Stefano Ermon.
Generative modeling by estimating gradients of the data distribution.
In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 32. Curran Associates, Inc., 2019.
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *International Conference on Learning Representations*, 2021.
- Vincent (2011)
Pascal Vincent.
A connection between score matching and denoising autoencoders.
*Neural Computation*, 23(7):1661–1674, 2011.
doi: 10.1162/NECO˙a˙00142.
- Welling & Teh (2011)
Max Welling and Yee Whye Teh.
Bayesian learning via stochastic gradient langevin dynamics.
In Lise Getoor and Tobias Scheffer (eds.), *Proceedings of the 28th International Conference on Machine Learning (ICML-11)*, ICML ’11, pp. 681–688, New York, NY, USA, June 2011. ACM.
ISBN 978-1-4503-0619-5.
- Wen et al. (2024)
Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu.
Detecting, explaining, and mitigating memorization in diffusion models.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Wilson & Izmailov (2020)
Andrew G Wilson and Pavel Izmailov.
Bayesian deep learning and a probabilistic perspective of generalization.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 4697–4708. Curran Associates, Inc., 2020.
- Zoph & Le (2017)
Barret Zoph and Quoc Le.
Neural architecture search with reinforcement learning.
In *International Conference on Learning Representations*, 2017.

## Appendix A Experimental Setup

All of our experiments were conducted on a Linux machine with 128GB of RAM and a NVIDIA RTX 4090 GPU. We train and evaluate diffusion models according to the DDPM framework (Ho et al., 2020), with our hyperparameters included in Table [1](https://arxiv.org/html/2510.22899v1#A1.T1). To compute the sliced Wasserstein metrics, we use an overcomplete set of $L=64D$ random directions. All experiments are repeated five times with different random seeds. Each dataset we consider consists of 10k samples. For sphere modeling, shown in Figure [1](https://arxiv.org/html/2510.22899v1#S1.F1), the data is aligned with the first three and the last three SADs. When estimating the average geometry, $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)$, we use 1M randomly initialized networks.

**Table 1: Hyperparameters of iDDPM and DiT architectures used in the paper. *We train DiT/4 for double the iterations such that the running time is roughly the same as DiT/2 and iDDPM networks.**
| iDDPM | $\mathcal{N}(\bm{0},D\bm{v}\bm{v}^{\top})$ | MNIST | CelebA-HQ | CIFAR-10 |
| --- | --- | --- | --- | --- |
| shape | $1\times 16\times 16$ | $1\times 28\times 28$ | $1\times 56\times 56$ | $3\times 32\times 32$ |
| diffusion_steps | 1000 | 1000 | 1000 | 1000 |
| noise_schedule | linear | linear | linear | linear |
| channels | 32 | 32 | 32 | 32 |
| channel_mults | 1, 1 | 1, 1 | 1, 1, 1 | 1, 1, 1 |
| depth | 1 | 2 | 2 | 2 |
| attn_resolutions | - | - | - | - |
| num_heads | 4 | 4 | 4 | 4 |
| batch_size | 1000 | 500 | 500 | 500 |
| iterations | 2k | 100k | 100k | 200k |
| learning_rate | 1e-4 | 1e-3 | 1e-3 | 1e-3 |

## Appendix B Geometries of Common Neural Network Architectures

###### Proposition 1 (MLP geometry, proof in Appendix C.4 ) .

Let $\mathcal{F}$ be the family of networks of the form $\bm{z}=\phi(\bm{W}\bm{h}+\bm{b})$, where $\phi:\mathbb{R}\to\mathbb{R}$ is applied element-wise and $\bm{h}\in\mathbb{R}^{L}$ is an arbitrary function of the input. Assume that $\bm{W},\bm{b}$ are completely independent and parameters within each group are identically distributed. Letting $\bm{1}$ represent a vector of ones, the geometry takes the form:

$$ $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)=\alpha_{\mathcal{F}}(\mathcal{P},\Theta)\bm{I}+\beta_{\mathcal{F}}(\mathcal{P},\Theta)\bm{1}\bm{1}^{\top}.$ (7) $$

###### Proposition 2 (CNN geometry, proof in Appendix C.5 ) .

Let $\mathcal{F}$ be the family of networks of the form $\bm{z}=\phi(\bm{W}\bm{h}+\bm{b})$, where $\phi:\mathbb{R}\to\mathbb{R}$ is applied element-wise and $\bm{h}\in\mathbb{R}^{C_{\text{in}}\times L}$ denotes an arbitrary function of the input. $\bm{W}$ represents convolution with kernel size $k$, $C_{\text{in}}$ input channels and $C_{\text{out}}$ output channels. Assume $\bm{W},\bm{b}$ are completely independent and parameters within each group are identically distributed. Then, letting, $\otimes$ denote the Kronecker product, the geometry is a block-diagonal matrix with a block-constant perturbation:

$$ $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)=\bm{I}_{\text{C}_{out}}\otimes\bm{A}_{\mathcal{F}}(\mathcal{P},\Theta)+(\bm{1}\bm{1}^{\top})_{C_{\text{out}}}\otimes\bm{B}_{\mathcal{F}}(\mathcal{P},\Theta).$ (8) $$

###### Proposition 3 (Transformer geometry, proof in Appendix C.6 ) .

Let $\mathcal{F}$ be the family of networks of the form $\bm{z}=\bm{Q}(\bm{W}\bm{h}+\bm{b})$, where $\bm{Q}$ is fixed, orthonormal (e.g., unpatchify) and $\bm{h}\in\mathbb{R}^{T\times L_{\text{in}}}$ denotes an arbitrary function of the input. $\bm{W}:\mathbb{R}^{T\times L_{\text{in}}}\to\mathbb{R}^{T\times{L}_{\text{out}}}$ and $\bm{b}$ represent a transformer-style layer operating on $T$ tokens separately. Assume $\bm{W},\bm{b}$ are completely independent and parameters within each group are zero-mean, identically distributed. Then, the geometry has at most $T$ distinct eigenvalues and takes the form:

$$ $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)=\bm{Q}[\bm{A}_{\mathcal{F}}(\mathcal{P},\Theta)\otimes\bm{I}_{L_{\text{out}}}]\bm{Q}^{\top}.$ (9) $$

###### Proposition 1 (MLP geometry, proof in Appendix C.4 ) .

###### Proposition 2 (CNN geometry, proof in Appendix C.5 ) .

###### Proposition 3 (Transformer geometry, proof in Appendix C.6 ) .

## Appendix C Deferred Proofs

### C.1 Lemmas

###### Lemma 1 .

Consider DSM with data drawn from $\mathcal{N}(\bm{0},\bm{v}\bm{v}^{\top})$ for a fixed noise level $\sigma>0$ and $\|\bm{v}\|_{2}=1$. The associated score function is linear and of the form $\bm{\Omega}(\cdot)$, with $\bm{\Omega}=\frac{1}{\sigma^{2}}\left(\frac{\bm{v}\bm{v}^{\top}}{\sigma^{2}+1}-\bm{I}\right)$.

###### Proof.

We have the density $p_{\sigma}=\mathcal{N}(\bm{0},\bm{v}\bm{v}^{\top}+\sigma^{2}\bm{I})$. The log-density is therefore quadratic, resulting in a linear score function as follows:

$$ $\nabla_{\bm{x}_{\sigma}}\log p_{\sigma}(\bm{x}_{\sigma})=-\frac{1}{2}\nabla_{\bm{x}_{\sigma}}\left[\bm{x}_{\sigma}^{\top}(\bm{v}\bm{v}^{\top}+\sigma^{2}\bm{I})^{-1}\bm{x}_{\sigma})\right]=-(\bm{v}\bm{v}^{\top}+\sigma^{2}\bm{I})^{-1}\bm{x}_{\sigma}.$ (10) $$

We can further simplify this via the Sherman-Morrison formula:

$$ $\bm{\Omega}=-(\bm{v}\bm{v}^{\top}+\sigma^{2}\bm{I})^{-1}=\frac{1}{\sigma^{2}}\left(\frac{\bm{v}\bm{v}^{\top}}{\sigma^{2}+1}-\bm{I}\right).$ (11) $$

∎

###### Lemma 2 .

Let $\bm{W}$ be a random matrix with entries that are iid, with mean zero and variance $\sigma^{2}$. For any compatible matrix $\bm{X}$, $\mathbb{E}_{\bm{W}}[\bm{W}^{\top}\bm{X}\bm{W}]=\sigma^{2}\operatorname{tr}(\bm{X})\bm{I}$, where $\operatorname{tr}(\cdot)$ is the trace.

###### Proof.

Let $(\cdot)^{(i)}$ be column $i$. For entry $(i,j)$ we can write the expectation as follows:

$$ $\sum_{k}\mathbb{E}\left[{\bm{W}^{(i)}}^{\top}\bm{X}^{(k)}\bm{W}_{(k,j)}\right]=\sum_{k}\sigma^{2}\delta_{i-j}\bm{X}_{(k,k)}=\sigma^{2}\operatorname{tr}(\bm{X})\delta_{i-j}.$ (12) $$

∎

###### Lemma 1 .

###### Proof.

###### Lemma 2 .

###### Proof.

### C.2 Theorem 1

###### Proof.

Let $\bm{\Omega}=\bm{\Phi}\bm{\Theta}$ be a linear model. The optimization objective is:

$$ $\begin{split}\mathcal{J}_{\text{DSM}}(\bm{\Theta})&=\mathbb{E}_{\bm{x}\sim\mathcal{N}(\bm{0},\bm{v}\bm{v}^{\top}),\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I})}\left[\left\|\bm{\Phi}\bm{\Theta}(\bm{x}+\sigma\bm{\epsilon})+\frac{\bm{\epsilon}}{\sigma}\right\|^{2}_{2}\right]\\ &=\mathbb{E}\left[\left\|\bm{\Phi}\bm{\Theta}\bm{x}+\left(\sigma\bm{\Phi}\bm{\Theta}+\frac{\bm{I}}{\sigma}\right)\bm{\epsilon}\right\|^{2}_{2}\right]\\ &\overset{(*)}{=}\mathbb{E}\left[\left\|\bm{\Phi}\bm{\Theta}\bm{x}\right\|^{2}_{2}\right]+\mathbb{E}\left[\left\|\left(\sigma\bm{\Phi}\bm{\Theta}+\frac{\bm{I}}{\sigma}\right)\bm{\epsilon}\right\|^{2}_{2}\right]=\|\bm{\Phi}\bm{\Theta}\bm{v}\|^{2}_{2}+\left\|\sigma\bm{\Phi}\bm{\Theta}+\frac{\bm{I}}{\sigma}\right\|^{2}_{F},\end{split}$ (13) $$

where $(*)$ follows by independence of $\bm{x}$ and $\bm{\epsilon}$. The gradient with respect to $\bm{\Theta}$ is then given by:

$$ $\nabla_{\bm{\Theta}}\mathcal{J}_{\text{DSM}}(\bm{\Theta})=2\bm{\Phi}^{\top}[\bm{\Phi}\bm{\Theta}(\bm{v}\bm{v}^{\top}+\sigma^{2}\bm{I})+\bm{I}].$ (14) $$

With this, and for a suitable $\eta>0$, we express the GD learning dynamics with respect to $\bm{\Omega}$ as:

$$ $\bm{\Omega}_{t}=\bm{\Omega}_{t-1}-\eta\bm{\Phi}\nabla_{\bm{\Theta}}\mathcal{J}_{\text{DSM}}(\bm{\Theta})=\bm{\Omega}_{t-1}-2\eta\bm{\Phi}\bm{\Phi}^{\top}[\bm{\Omega}_{t-1}(\bm{v}\bm{v}^{\top}+\sigma^{2}\bm{I})+\bm{I}].$ (15) $$

Moreover, we study the error dynamics defined by $\bm{E}_{t}=\bm{\Omega}_{t}-\bm{\Omega}^{*}$. Here, $\bm{\Omega}^{*}=\frac{1}{\sigma^{2}}\left(\frac{\bm{v}\bm{v}^{\top}}{\sigma^{2}+1}-\bm{I}\right)$ is given by Lemma [1](https://arxiv.org/html/2510.22899v1#Thmlemma1) and we can verify that $\bm{\Omega}^{*}(\bm{v}\bm{v}^{\top}+\sigma^{2}\bm{I})+\bm{I}=\bm{0}$, so it is a stationary point. Therefore, we write:

$$ $\bm{E}_{t}=\bm{E}_{t-1}-2\eta\bm{\Phi}\bm{\Phi}^{\top}\bm{E}_{t-1}(\bm{v}\bm{v}^{\top}+\sigma^{2}\bm{I}).$ (16) $$

Let us focus on the sequence of expected errors, $\mathbb{E}[\bm{E}_{t}]$, where the randomness is over the initialization.(^4^44More generally, our analysis here is also applicable to fluctuations due to SGD.) Assuming $\mathbb{E}[\bm{\Theta}_{0}]=\bm{0}$, $\mathbb{E}[\bm{E}_{0}]=-\bm{\Omega}^{*}$. Additionally, if $\bm{v}\in\{\bm{u}_{i}\}_{i=1}^{D}$ is an eigenvector of $\bm{\Phi}\bm{\Phi}^{\top}$ with corresponding eigenvalues $\{\lambda_{i}\}_{i=1}^{D}$, matrices in Equation [16](https://arxiv.org/html/2510.22899v1#A3.E16) commute since they share eigenvectors. This also forces the eigenspaces of all subsequent error terms. In particular, we have:

$$ $\mathbb{E}[\bm{E}_{t}]=\mathbb{E}[\bm{E}_{t-1}]-2\eta\bm{\Phi}\bm{\Phi}^{\top}(\bm{u}_{i}{\bm{u}_{i}}^{\top}+\sigma^{2}\bm{I})\mathbb{E}[\bm{E}_{t-1}]=-[\bm{I}-2\eta(\lambda_{i}\bm{u}_{i}{\bm{u}_{i}}^{\top}+\sigma^{2}\bm{\Phi}\bm{\Phi}^{\top})]^{t}\bm{\Omega}^{*},$ (17) $$

where it is clear that the error decays exponentially. For sufficiently small $\eta$, the iterated matrix is positive semidefinite and therefore convergence depends on the minimum eigenvalue of $\lambda_{i}\bm{u}_{i}{\bm{u}_{i}}^{\top}+\sigma^{2}\bm{\Phi}\bm{\Phi}^{\top}$, i.e., it is $\mathcal{O}[(1-2\eta\rho_{i})^{t}]$ with $\rho_{i}=\min[(\sigma^{2}+1)\lambda_{i},\sigma^{2}\min_{j\neq i}\lambda_{j}]$. Suppose $\exists\lambda_{j}<\lambda_{i}$, then $\rho_{i}=\sigma^{2}\lambda_{D}$, i.e., the convergence rate is fixed for $i<D$. However, for $i=D$ we have $\rho_{D}=\min[(\sigma^{2}+1)\lambda_{D},\sigma^{2}\lambda_{D-1}]>\sigma^{2}\lambda_{D}$. That is, we converge faster for $i=D$.

Now, to complete the proof, we focus on the stochastic gradient at optimality:

$$ $\nabla_{\bm{\Theta}}\widehat{\mathcal{J}}_{\text{DSM}}(\bm{x},\bm{\epsilon};\bm{\Theta}^{*})=2\bm{p}\bm{q}^{\top},\quad\bm{p}=\bm{\Phi}^{\top}\left(\bm{\Omega}^{*}\bm{q}+\frac{\bm{\epsilon}}{\sigma}\right),\quad\bm{q}=\bm{x}+\sigma\bm{\epsilon}.$ (18) $$

Note, by construction, all of the above quantities are zero-mean. In particular, this implies that $\bm{p}$, $\bm{q}$ are uncorrelated and, since they are jointly Gaussian, we claim that they are independent. Therefore, by setting $\bm{v}=\bm{u}_{i}$ and vectorizing, we can write the stochastic gradient covariance as:

$$ $\begin{split}4\mathbb{E}\left[\text{vec}(\bm{p}\bm{q}^{\top})\text{vec}(\bm{p}\bm{q}^{\top})^{\top}\right]&=4\mathbb{E}\left[(\bm{q}\bm{q}^{\top})\otimes(\bm{p}\bm{p}^{\top})\right]\\ &=4\mathbb{E}\left[\bm{q}\bm{q}^{\top}\right]\otimes\mathbb{E}\left[\bm{p}\bm{p}^{\top}\right]\\ &=\frac{4}{\sigma^{2}(\sigma^{2}+1)}(\bm{u}_{i}\bm{u}_{i}^{\top}+\sigma^{2}\bm{I})\otimes(\bm{\Phi}^{\top}\bm{u}_{i}\bm{u}_{i}^{\top}\bm{\Phi})\propto\lambda_{i}.\end{split}$ (19) $$

∎

###### Proof.

### C.3 Theorem 2

###### Proof.

We first simplify Equation [6](https://arxiv.org/html/2510.22899v1#S3.E6):

$$ $\alpha=\mathbb{E}_{\bm{x}\sim p}[\bm{x}^{\top}(\bm{W}^{\top}\mathbf{G}_{\mathcal{F}}\bm{W}\bm{x})]=\operatorname{tr}(\bm{W}^{\top}\mathbf{G}_{\mathcal{F}}\bm{W}\bm{C}),$ (20) $$

where $\bm{C}:=\mathbb{E}_{\bm{x}\sim p}[\bm{x}\bm{x}^{\top}]$ is the second moment of the data. Let $\mathbf{G}_{\mathcal{F}}=\bm{U}\bm{\Lambda}\bm{U}^{\top}$ and $\bm{C}=\bm{V}\bm{\Sigma}\bm{V}^{\top}$ be eigendecompositions with eigenvalues $\{\lambda_{i}\}^{D}_{i=1}$ and $\{\sigma_{i}\}_{i=1}^{D}$ respectively. By defining the orthogonal matrix $\bm{Q}=\bm{U}^{\top}\bm{W}\bm{V}$, Equation [20](https://arxiv.org/html/2510.22899v1#A3.E20) can be equivalently stated as follows:

$$ $\begin{split}\alpha&=\operatorname{tr}(\bm{W}^{\top}\bm{U}\bm{\Lambda}\bm{U}^{\top}\bm{W}\bm{V}\bm{\Sigma}\bm{V}^{\top})\\ &=\operatorname{tr}(\bm{V}^{\top}\bm{W}^{\top}\bm{U}\bm{\Lambda}\bm{U}^{\top}\bm{W}\bm{V}\bm{\Sigma})\\ &=\operatorname{tr}(\bm{Q}^{\top}\bm{\Lambda}\bm{Q}\bm{\Sigma})=\sum_{i=1}^{D}\sum_{j=1}^{D}\lambda_{i}\sigma_{j}[\bm{Q}_{(i,j)}]^{2}.\end{split}$ (21) $$

Observe that, since $\bm{Q}$ is orthogonal, the matrix defined by $\bm{P}_{(i,j)}=[\bm{Q}_{(i,j)}]^{2}$ is doubly stochastic. Moreover, Equation [21](https://arxiv.org/html/2510.22899v1#A3.E21) shows that optimizing $\alpha$ is a linear problem in $\bm{P}$ over the the convex set of doubly stochastic matrices. In particular, by Birkhoff’s theorem, $\alpha$ is extremized when $\bm{P}$ is a permutation matrix. The minimizing permutation is the one that misaligns the eigenvalues, i.e., $\bm{P}=\bm{J}$ is achieved if $\bm{Q}=\bm{J}\iff\bm{W}=\bm{U}\bm{J}\bm{V}^{\top}$. Similarly, $\alpha$ is maximized when the eigenvalues are aligned, i.e., $\bm{P}=\bm{I}$ if $\bm{Q}=\bm{I}\iff\bm{W}=\bm{U}\bm{V}^{\top}$. Indeed, one verifies that these permutations are optimal as they satisfy von Neumann’s trace inequalities with equality (cf. Ruhe).
∎

###### Proof.

### C.4 Proposition 1

###### Proof.

Let $\Theta$ denote the specified parameter distribution and consider a probe $\mathcal{P}$. The geometry is:

$$ $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)=\mathbb{E}_{\bm{h}}\mathbb{E}_{\bm{W},\bm{b}}\left[\phi(\bm{W}\bm{h}+\bm{b})\phi(\bm{W}\bm{h}+\bm{b})^{\top}\right].$ (22) $$

First, compute the inner expectation, i.e., $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)|_{\bm{h}}$. For indices $i,j$, since the parameters are iid:

$$ $[\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)|_{\bm{h}}]_{(i,j)}=\left\{\begin{array}[]{ll}\mathbb{E}_{\bm{W},\bm{b}}[\phi(\bm{W}^{(1)}\bm{h}+\bm{b}^{(1)})^{2}]&\text{if }i=j\\ \mathbb{E}_{\bm{W},\bm{b}}[\phi(\bm{W}^{(1)}\bm{h}+\bm{b}^{(1)})]^{2}&\text{otherwise}\\ \end{array}\right..$ (23) $$

Writing $\bm{z}^{(1)}=\phi(\bm{W}^{(1)}\bm{h}+\bm{b}^{(1)})$, we can express the above via the conditional mean, $\mu_{\bm{z}^{(1)}|\bm{h}}$, and conditional variance $\sigma^{2}_{\bm{z}^{(1)}|\bm{h}}$:

$$ $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)|_{\bm{h}}=\sigma^{2}_{\bm{z}^{(1)}|\bm{h}}\bm{I}+\mu^{2}_{\bm{z}^{(1)}|\bm{h}}\bm{1}\bm{1}^{\top},$ (24) $$

where $\bm{1}$ is a vector of ones. Now, taking the outer expectation yields the desired result:

$$ $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)=\mathbb{E}_{\bm{h}}[\sigma^{2}_{\bm{z}^{(1)}|\bm{h}}]\bm{I}+\mathbb{E}_{\bm{h}}[\mu^{2}_{\bm{z}^{(1)}|\bm{h}}]\bm{1}\bm{1}^{\top}.$ (25) $$

∎

###### Proof.

### C.5 Proposition 2

###### Proof.

Similarly to our treatment of MLPs in Appendix [C.4](https://arxiv.org/html/2510.22899v1#A3.SS4), we first compute the geometry conditioned on $\bm{h}$, i.e., $\mathbf{G}(\mathcal{P},\Theta)|_{\bm{h}}$. Vectorizing, write the $i^{\text{th}}$ input channel as $\bm{h}^{(i)}\in\mathbb{R}^{L}$. Then, $\bm{W}$ is a $C_{\text{out}}\times C_{\text{in}}$ block matrix with each $\bm{W}_{(i,j)}$ representing convolution with a filter of size $k$. Therefore, $\bm{z}^{(i)}=\phi(\sum_{j}\bm{W}_{(i,j)}\bm{h}^{(j)}+\bm{b}^{(i)})$. Since the parameters are iid, the conditional geometry is expressed as the following $C_{\text{out}}\times C_{\text{out}}$ block matrix:

$$ $[\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)|_{\bm{h}}]_{(m,n)}=\left\{\begin{array}[]{ll}\mathbb{E}_{\bm{W},\bm{b}}[\bm{z}^{(1)}{\bm{z}^{(1)}}^{\top}]&\text{if }m=n\\ \mathbb{E}_{\bm{W},\bm{b}}[\bm{z}^{(1)}]\mathbb{E}_{\bm{W},\bm{b}}[\bm{z}^{(1)}]^{\top}&\text{otherwise}\\ \end{array}\right..$ (26) $$

We can rephrase this result in terms of the conditional mean, $\bm{\mu}_{\bm{z}^{(1)}|\bm{h}}$, and the conditional covariance $\bm{\Sigma}_{{\bm{z}^{(1)}}|\bm{h}}$. The manipulation is identical to the one used to derive the MLP geometry. Averaging $\bm{h}$:

$$ $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)=\bm{I}_{C_{\text{out}}}\otimes\mathbb{E}_{\bm{h}}[\bm{\Sigma}_{\bm{z}^{(1)}|\bm{h}}]+(\bm{1}\bm{1}^{\top})_{C_{\text{out}}}\otimes\mathbb{E}_{\bm{h}}[\bm{\mu}_{\bm{z}^{(1)}|\bm{h}}\bm{\mu}^{\top}_{\bm{z}^{(1)}|\bm{h}}].$ (27) $$

∎

###### Proof.

### C.6 Proposition 3

###### Proof.

We first focus on $\bm{Q}=\bm{I}$. After vectorizing $\bm{h}$, $\bm{W}$ is block-diagonal with $\bm{W}_{(i,i)}=\bm{W}_{(1,1)}$ operating on each token, which we write as $\bm{h}^{(i)}\in\mathbb{R}^{L_{\text{in}}}$. Conditioned on $\bm{h}$, the geometry is a block matrix with block $(i,j)$:

$$ $\begin{split}[\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)|_{\bm{h}}]_{(i,j)}&=\mathbb{E}_{\bm{W},\bm{b}}[(\bm{W}_{(1,1)}\bm{h}^{(i)}+\bm{b}^{(i)})(\bm{W}_{(1,1)}\bm{h}^{(j)}+\bm{b}^{(j)})^{\top}]\\ &=(\sigma^{2}_{\bm{W}}{\bm{h}^{(i)}}^{\top}\bm{h}^{(j)}+\sigma^{2}_{\bm{b}})\bm{I},\end{split}$ (28) $$

where the last equality is by Lemma [2](https://arxiv.org/html/2510.22899v1#Thmlemma2), assuming parameters in $\bm{W}$, $\bm{b}$ have variances $\sigma^{2}_{\bm{W}}$, $\sigma^{2}_{\bm{b}}$ respectively. Notice that every block is $\propto\bm{I}$ and depends on entries of $\bm{h}\bm{h}^{\top}\in\mathbb{R}^{T\times T}$, where, with a slight abuse of notation, we have reverted to the matrix representation $\bm{h}\in\mathbb{R}^{T\times L_{in}}$. Writing this compactly as $(\sigma^{2}_{\bm{W}}\bm{h}\bm{h}^{\top}+\sigma^{2}_{\bm{b}}\bm{I})\otimes\bm{I}_{L_{\text{out}}}$, applying $\bm{Q}$ and averaging over $\bm{h}$ yields:

$$ $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)=\bm{Q}\mathbb{E}_{\bm{h}}[(\sigma^{2}_{\bm{W}}\bm{h}\bm{h}^{\top}+\sigma^{2}_{\bm{b}}\bm{I})\otimes\bm{I}_{L_{\text{out}}}]\bm{Q}^{\top}.$ (29) $$

The eigenvalues of the above are products of the eigenvalues of the factors in the Kronecker product, i.e., they are the $T$ eigenvalues of $\sigma^{2}_{\bm{W}}\mathbb{E}_{\bm{h}}[\bm{h}\bm{h}^{\top}]+\sigma^{2}_{\bm{b}}\bm{I}$.

∎

###### Proof.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Could the authors elaborate on "directional preferences" stated? Are directional preferences equivalent to importance of features in training and inference processes? I am wondering how the directional preferences in network architectures will enhance or reduce the features in data such as images.

2. In terms of the motivation, why is it necessary to study the directional preferences in network architectures for generative models, except its absence but prominence in studies of discriminative models? How does it influence the generation quality of generative models?

3. Could the authors clarify the meaning of $\boldsymbol{v}$? What is the relation between $\boldsymbol{v}$ and $\sigma$?

4. Why is  the best performance achieved when the data is not aligned with the “geometry” that is induced by the score network, but instead lives in the subspace defined by its smallest eigenvalues? Could the authors provide an intuitive explanation?
soundness: 3
strengths: 1. **Clarity**: This work is clearly presented with informative visualization of preliminary results on iDDPM which enhances the motivations. The logic is generally linear and the whole paper is easy to follow. Theories are well articulated and key takeaways are summarized with clarity at the end of each block of analysis and discoveries.

2. **Quality**: The proposed theories are proved comprehensively and coherently on the importance of directional preference in network architectures and the impact of it. The claimed importance of anisotropic conditions is also in general supported by the phenomena in numerical experiments.
summary: In this work, the directional preferences of network architecture is theoretically identified and analyzed from the harmonic analysis point of view. The geometry in generative model architecture is clarified and the output space induced by the directional preference of network architecture is decomposed.
weaknesses: 1. Although the claimed discoveries are supported by experiments, datasets used in experiments are in low resolutions where the highest is only $56\times 56$. Experiments on higher dimensionality may indicate results on the contrary to the theoretical discoveries. It would be more comprehensive to conduct experiments on datasets such as ImageNet with the resolution of $256\times 256$.

2. Before this work, there is already existing research discussing and modifying generative models on directional sensitivity, such as Kuramoto Orientation Diffusion Models [1] and Artificial Kuramoto Oscillatory Neurons [2] where [1] proposes a new diffusion model. It would improve the novelty and significance to discuss [1] and [2] and highlight the novelty accordingly.
 
[1] Yue Song, T. Anderson Keller, Sevan Brodjian, Takeru Miyato, Yisong Yue, Pietro Perona, and Max Welling, Kuramoto Orientation Diffusion Models, 39th Conference on Neural Information Processing Systems, 2025.

[2] Takeru Miyato, Sindy Löwe, Andreas Geiger, Max Welling, Artificial Kuramoto Oscillatory Neurons, ICLR 2025.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. In Theorem 1, is having $\lambda_{D-1} \neq \lambda_D$ an (implicit) assumption? Also, it would be helpful if the statements are written more precisely and rigorously, such as the "*mean error* to the optimal solution" and "SGD steps with respect to $\Theta$ *for $\boldsymbol{v} = \boldsymbol{u}_i$* have *covariance* $\propto \lambda_i$; what is the *mean error*, and how does a SGD step be "with respect to $\Theta$" and "for $\boldsymbol{v} = \boldsymbol{u}_i$" simultaneously, and what could it having a *covariance* mean? I personally think a theorem statement should be clearly understandable without referring to its proof. 

2. I am not sure if Markov's inequality (4) is the right way to back up the claim of "small eigenvalue directions are easier to learn". After all, Markov's inequality is just an upper bound for the tail probability; it can say something about small eigenvalue directions as the right hand side will be small, but nothing can be inferred from it about the behavior along large eigenvalue directions. 

3. The intuitive explanations in Section 3.2 seem reasonable, but only when considered locally around a point *$\boldsymbol{x}_\sigma$* that happens to have a large log-density. In defining the *average geometry*, we average out the "geometry" with respect to the distribution of *$\boldsymbol{x}_\sigma$,* which will smooth (or even worse, possibly cancel) out directions that are only meaningful in a local sense. Thus, although it seems like there is an explanation about this right after Definition 2, it is still unclear to me how the local intuition applies to the notion of average geometry. For example, in an extreme toy case where the data distribution is bimodal in $\mathbb{R}^2$, with each mode having its own "meaningful" directions that are orthogonal to those of the other mode, the resulting average geometry would be meaningless. Is there any intuition as to why we should expect the *average geometry* to possess a global semantic meaning?

4. It seems like Figure 7 is arguing that the traditional training recipe (which amounts to setting $\boldsymbol{W} = \boldsymbol{I}$) should not perform well, which is contrary to what we are observing in reality. What is the punchline of this figure, and what am I missing?

5. Why is $\mathrm{SW}_2$ chosen as the performance measure? Is there a particular reason why more “standard” metrics such as FID or the Inception Score were not used?

6. As a follow-up question of the previous one, how exactly was $\mathrm{SW}_2$ calculated?
It is well known that reliably computing $\mathrm{SW}_2$ in high-dimensional spaces is challenging, since two random vectors are almost always nearly orthogonal; as a result, a randomly chosen direction is unlikely to serve as a meaningful proxy for $\boldsymbol{\theta}$ in equation (3), with very high probability. How is this issue addressed?

7. I hope the authors will consider adding visual results for the experiments on CelebA and CIFAR-10 in the paper. It would also be helpful if they could include a brief verbal description of these results in the rebuttal, since presenting figures may be difficult at that stage.

8. For the notion of average geometry to provide meaningful insights into training diffusion models in practice, it seems like it should ideally capture information that evolves as the model parameters (or their distribution) change during training. However, the current definition appears to depend only on the distribution used at initialization, offering no indication of how the geometry of the outputs evolves as the parameters are updated, nor how it would interact with the data distribution during the training. How does the concept of average geometry account for this discrepancy?
soundness: 3
strengths: This work presents a new lens for studying the performance of score-based generative models. It even offers some novel insights, such as a training recipe that applies appropriate rigid motions to align the data with the SADs, and the bold claim that subspaces with small eigenvalues are easier for the models to learn. 
The central topic itself is a timely one, as the impact of generative models is rapidly growing, while we still have little understanding on how they work. 
I can see this work serving as a great starting point for the geometric/inductive bias approach to the theoretical understanding of score-based generative models.
summary: This paper proposes the framework of *Score Anisotropy Directions* (SADs) as a tool for a better understanding of how score-based generative models perform, by providing a rough prediction on how well the model will perform, according to the geometry of the model output and that of the given dataset. This framework identifies the intrinsic geometric bias which the model in use has, and tells us how well this aligns with the dataset geometry is an indicator of the performance.
weaknesses: As much as I like the idea and the approach, I also have some questions and concerns about the current state of the paper. 

The writing, especially in the introduction, has room for improvement. For example, while the introduction claims to provide a precise notion of "geometry" (around line numbers 46--48), the very first definition of SADs (Definition 1) is somewhat vague. How is a "preference" measured, and what does it mean to "generate data *along* a direction"?

I would also suggest moving the related works section to the front instead of keeping it at the end with the conclusions, as this would give readers stronger motivation to delve further into the paper.

Meanwhile, I find it quite unclear what the figures are meant to convey, even after reading the entire paper. I don't see what the 3D projection of the $\mathbb{R}^{256}$ space in Figure 1 is intended to depict, especially with the point clouds obscuring the geometry. It is also not clear how I should interpret the 2D plots in Figure 2 and 3. It would be helpful if the authors could further elaborate on these visualizations.

For further questions, please see below.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 4
questions: Could the authors provide intuition for why Conjecture 1 should hold for non‑linear networks, and clarify how your average geometry matrix relates to NTK in the wide‑network/linearization limit?
soundness: 3
strengths: Overall, I think this is a good empirical paper that proposes an interesting phenomenon for score-based generative models. I especially appreciate the clarity around SADs, how to quantify the "preferred directions" in generative models, and the demonstration that lower data–geometry alignment leads to better generation quality. The result is simple to grasp through the manifold hypothesis of data distribution, yet seems to broadly applicable across architectures and optimization algorithms.
summary: The paper studies directional inductive biases of score-based generative models. The authors introduce Score Anisotropy Directions (SADs), which is an set of orthonormal directions in the output space, obtained as the eigenvectors (smallest to largest) of an average geometry matrix. In a linear DSM setting, the authors show that SGD training leads to anisotropy and converging fastest along small eigenvalue directions. For the general nonlinear models, they conjecture the same ordering and support it with: (i) synthetic rank‑one datasets aligned to different bases; (ii) experiments where performance follows the eigenvector order for U‑Net/iDDPM and DiT. On real datasets, they vary an alignment measure between the data and the model's geometry via orthogonal transforms and find that lower alignment improves the performance, while high alignment underperforms or offers no gain over the baseline. The paper also analyzes average geometry matrix for common architectures and connects SADs to Geometry‑Adaptive Harmonic Bases.
weaknesses: The only weakness I can think of is that, while extensive experiments are conducted to verify the main conjecture, it is rigorously proved only for a linear DSM toy model. The paper would be stronger with an intuitive argument in a wide‑network/NTK or mean‑field limit showing why the SAD eigen‑ordering should persist and clarifying conditions under which their conjecture holds true.

---

confidence: 2
contribution: 3
initial_rating: 4
presentation: 3
questions: - The matrix $G_\mathcal{F} = \mathbb{E}[ F_\theta (x_\sigma, \sigma)F_\theta(x_\sigma, \sigma)^\top] $ depends on both $P$ and the initialization scheme. How sensitive are your results to these choices?
- In Theorem 1, the model is assumed linear. How do you justify extrapolating these linear results to complex, nonlinear diffusion architectures?
- The conjecture linking SADs to eigenvectors of  $G_\mathcal{F}$ is intuitive, but is there any formal result supporting it for nonlinear or trained networks?
soundness: 3
strengths: 1. **Novel conceptual framing:**
The paper offers an original perspective by proposing SADs as a way to quantify architectural inductive biases in score-based generative models.
2. **Synthesis of architecture and anisotropy analysis:**
It connects the anisotropic conditioning of the optimization landscape to generative modeling performance, extending concepts like Neural Anisotropy Directions (NADs) from discriminative to generative contexts.
3. **Theoretical–empirical bridge:**
The authors combine analytical derivations for simplified cases with experimental validation on U-Net and DiT architectures, showing distinct anisotropy profiles across architectures.
4. **Insightful empirical angle:**
The work provides technical examples illustrating how geometric alignment affects generalization and how this connects to data augmentation effects in diffusion training.
5. **Potential interpretability contribution:**
The proposed framework could, in principle, help explain why certain architectures generalize better or exhibit specific generative artifacts.
summary: The paper introduces a new theoretical framework to analyze the directional inductive biases of score-based generative models (diffusion models). It defines Score Anisotropy Directions (SADs) — architecture-dependent directions in the model output space — which are said to describe how different neural network architectures (e.g., U-Nets, DiTs) preferentially model certain data structures. The authors propose the concept of an average geometry matrix whose eigendecomposition predicts which data directions are more easily learned. They provide both analytical derivations (for simplified settings) and empirical evidence on synthetic and image datasets to support the claim that alignment between data and architectural geometry determines generalization quality. The experiments mainly focus on Wasserstein distances as a measure of generative performance.
weaknesses: 1. **Conceptual overstatement and lack of rigor:**
Most results are heuristic and not rigorously proven. The central claims (Conjecture 1 and its supporting theorems) are largely empirical conjectures, not formal results. The link between anisotropy and generalization remains speculative.
2. **Theoretical fragility:**
The formal parts (e.g., Theorem 1) rely on linearized and oversimplified assumptions that are far removed from the nonlinear reality of diffusion networks. The claimed generalization to complex architectures is not mathematically justified.
3. **Limited experimental depth:**
The experiments use small-scale datasets (MNIST, CIFAR-10, CelebA-HQ) and low-dimensional toy problems. The evidence is insufficient to validate claims about “predicting generalization before training.”
4. **Ambiguous interpretation of metrics:**
The use of Wasserstein distances as the sole quantitative indicator of generative quality is weak — no perceptual or likelihood-based metrics are provided. The connection between these distances and “generalization” is asserted but not justified.
5. **Unclear practical relevance:**
While the average geometry can be computed at initialization, it is not demonstrated that this analysis provides actionable insights for model design or training.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: I am confused by the statement of Theorem 1, specifically the line ‘the mean error to the optimal solution $v = u_i$ decays as…”. In what sense is $v = u_i$ an ‘optimal solution’, if the optimization is being done over $\Theta$? It seems like $v = u_i$ is an assumption of the theorem, and indeed the proof uses this fact in Equation (17). Also, I believe Equation (19) has an error, since the vectors $p$ and $q$ are not independent (although the proof claims that they are). In reality, I think you should have something like $$\mathbb{E}[\langle p, q \rangle] =  \mathbb{E}_{N(0,vv^T)}[x^T (\Omega^{\ast})^T \Phi x] + \sigma^2 \mathbb{E}[\epsilon^T (\Omega^{\ast})^T \Phi \epsilon] + \mathbb{E}[\epsilon^T \Phi \epsilon] > 0.$$

Thus, it is not clear to me whether the final sentence of Theorem 1 is true. Please let me know if I have misunderstood something here.
soundness: 2
strengths: - Understanding the inductive biases of diffusion models is an integral problem in deep learning, and the authors provide some progress towards this endeavor.
- The ideas of the paper are generally architecture-independent.
- The paper is generally well-written and it is easy to understand their central claims.
summary: The authors study the inductive biases of neural networks trained on denoising score matching and hypothesize that these inductive biases can be captured by fixed linear bases. In an idealized setting, where the inner weights of a linear network $\Omega = \Phi \Theta$ are trained on denoising score matching for a rank one Gaussian $N(0,vv^T)$, the authors demonstrate that the inductive bias of the family of linear networks is described by the eigenvectors of $\Phi$. More generally, the authors define the average geometry of a family of score networks in terms of the averaged outer product, where the averaging is done over the parameter distribution and a ‘probing distribution’ on the inputs of the score network. The authors claim that the eigenvectors of this matrix capture the inductive biases of the family of score networks and provide experiments which evidence for this claim.
weaknesses: - The novelty of the paper is limited, as the paper borrows significantly from previous attempts to formalize inductive biases in the discriminative setting [1], [2]. While the paper makes some improvements over previous work [3] in understanding inductive biases for diffusion models (e.g., moving beyond CNNs), some similar ideas are borrowed.
- I don’t find the central claim of the paper (i.e., that the average geometry describes the inductive bias of score networks) to be very well supported. Theoretically, it is essentially unprincipled except for some results in idealized settings, and the scope of the experiments is pretty narrow. 

References:
[1] Ortiz-Jiménez, Guillermo, Apostolos Modas, Seyed-Mohsen Moosavi, and Pascal Frossard. "Neural anisotropy directions." Advances in Neural Information Processing Systems 33 (2020): 17896-17906.
[2] Radhakrishnan, Adityanarayanan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. "Mechanism for feature learning in neural networks and backpropagation-free machine learning models." Science 383, no. 6690 (2024): 1461-1467.
[3] Kadkhodaie, Zahra, Florentin Guth, Eero P. Simoncelli, and Stéphane Mallat. "Generalization in diffusion models arises from geometry-adaptive harmonic representations." arXiv preprint arXiv:2310.02557 (2023).


Decision: Reject