Generalization in diffusion models arises from geometry-adaptive harmonic representations
Authors: Zahra Kadkhodaie
Florentin Guth
Eero P Simoncelli
Stéphane Mallat
Venue: ICLR 2024 oral (2024)
Keywords: diffusion models
memorization
generalization
inductive bias
curse of dimensionality
denoising
geometry-adaptive harmonic basis
Primary area: generative models
TL;DR: Diffusion models transition from memorization to generalization by being inductively biased towards geometry-adaptive harmonic representations.

Abstract: Deep neural networks (DNNs) trained for image denoising are able to generate high-quality samples with score-based reverse diffusion algorithms. These impressive capabilities seem to imply an escape from the curse of dimensionality, but recent reports of memorization of the training set raise the question of whether these networks are learning the "true" continuous density of the data. Here, we show that two DNNs trained on non-overlapping subsets of a dataset learn nearly the same score function, and thus the same density, when the number of training images is large enough.  In this regime of strong generalization, diffusion-generated images are distinct from the training set, and are of high visual quality, suggesting that the inductive biases of the DNNs are well-aligned with the data density. We analyze the learned denoising functions and show that the inductive biases give rise to a shrinkage operation in a basis adapted to the underlying image. Examination of these bases reveals oscillating harmonic structures along contours and in homogeneous regions. We demonstrate that trained denoisers are inductively biased towards these geometry-adaptive harmonic bases since they arise not only when the network is trained on photographic images, but also when it is trained on image classes supported on low-dimensional manifolds for which the harmonic basis is suboptimal. Finally, we show that when trained on regular image classes for which the optimal basis is known to be geometry-adaptive and harmonic, the denoising performance of the networks is near-optimal.

Published as a conference paper at ICLR 2024
GENERALIZATION IN DIFFUSION MODELS ARISES FROM GEOMETRY-ADAPHE ADAPTIVE HARMONIC REPRESENTATIONS
Zahra Kadkhodaie
Florentin Guth
Ctr. for Data Science, New York University zk388@nyu edd
Ctr. for Data Science, New York University Flatiron Institute, Simons Foundation florenting guthhnyu.ucuccc... Moreentin.uuchenu.ccm
edu
Eero P. Simoncelli
Stéphane Mallat
New York University Flatiron Institute, Simons Foundation eero. s simonce lireeell@@neee 1iiueuu.... edu
Collège de France
Flatiron Institute. Simons Foundation stephane.. ma1 lateens.. fr
ABSTRACT
Deep neural networks (DNNs) trained for image denoising are able to generate high- quality samples with score-based reverse diffusion algorithms. These impressive capabilities seem to imply an escape from the curse of dimensionality, but recent reports of memorization of the training set raise the question of whether these networks are learning the "true" continuous density the data. Here, we show that two DNNs trained non-overlapping subsets of dataset learn nearly the same score function, and thus the same density, when the number of training images is large enough. In this regime of strong generalization, diffusiion-geeerated images are distinct from the training set, and are of high visual quality, suggesting that the inductive biases of the DNNs are well-aligned with the data density. We analyze the learned denoising functions and show that the inductive biases give rise to a shrinkage operation in a basis adapted the underlying image. Examination of these bases reveals oscillating harmonic structures along contours and in homogeneous regions. We demonstrate that trained denoisers are inductively biased towards these geometry-adaptiv harmonic bases since they arise not only when the network trained on photographic images, but also when it is trained image classes supported on low- -iwdiimnsioonl manifolds for which the harmonic basis is suboptimal. Finally, we show that when trained on regular image classes for which the optimal basis is known to be ceometry-adapheee and harmonic, the denoising performance of the networks is near-optimal.
INTRODUCTION
Deep neural networks (DNNs) have demonstrated ever-more impressive capabilities for sampling from high-dimensional image densities, most recently through the development of diffusion methods. These methods operate by training denoiser, which provides estimate of the score (the gradient of the log of the noisy image distribution). The score is then used to sample from the corresponding estimated density, using an iterative reverse diffusion procedure (Sohl-Dickstein al., 2015: Song & Ermon, 2019; Ho et al., 2020; Kadkhodaie & Simoncelli, 2020). However, approximating continuous density in a high-ding high-dimensional space is notoriously difficult: do these networks actually achieve this feat, learning from a relatively small training set to generate high-quality samples, in apparent defiance of the curse of dimensionality? so, this must due to their inductive biases, that is, the restrictions that the architecture and optimization place on the learned denoising function. But the approximation class associated with these models is not well understood. Here, we take several steps toward elucidating this mystery.
Several recently reported results show that, when the training set small relative to the network capacity, diffusion generative models do not approximate continuous density, but rather memorize samples of the training set, which are then reproduced (or recombined) when generating new samples
Source code: hetpa:// //git won/ wttp://alloorioo CorPorrrggtt at fonal Vision tesiion//eee /wwe/eeerriin ion general fon in Fuxiion modells

---

Published as a conference paper at ICLR 2024
(Somepalli et al., 2023; Carlini et al., 2023). This is form of overfitting (high model variance). Here, we confirm this behavior for DNNs trained on small data sets, but demonstrate that these same models do not memorize when trained sufficiently large sets. Specifically, we show that two denoisers trained on sufficiently large non-overlapping sets converge to essentially the same denoising function. That is, the learned model becomes independent of the training set (i.e., model variance falls to zero), As a result, when used for image generation, these networks produce nearly identical samples. These results provide stronger and more direct evidence of generalization than standard comparisons of average performance on train and test sets. This generalization can be achieved with large but realizable training sets (for our examples, roughly 10 images suffices), reflecting powerful inductive biases of these networks. Moreover, sampling from these models produces images of high visual quality, implying that these inductive biases are well-matched to the underlying distribution of photographic images (Wilson & Izmailov, 2020; Goyal & Bengio, 2022; Griffiths et al., 2023).
To study these inductive biases, we develop and exploit the relationship between denoising and density estimation. We find that DNN denoisers trained on photographic images perform a shrinkage operation in an orthonormal basis consisting of harmonic functions that are adapted to the geometry of features in the underlying image. We refer to these as adopeiry-ddaptive harmonic bases (GAHBs). This observation, taken together with the generalization performance of DNN denoisers, suggests that optimal bases for denoising photographic images are GAHBS and, moreover, that inductive biases of DNN denoisers encourage such bases. To test this more directly, we examine particular class of images whose intensity variations are regular over regions separated by regular contours. A particular type of GAHB, known as "bandlets" (Peyré & Mallat, 2008), have been shown to be near-optimal for denoising these images (Dossal et al., 2011). 1). We observe that the DNN denoiser operates within a GAHB similar a bandlet basis, also achieving near-optimal performance. Thus the inductive bias enables the network to appropriately estimate the score in these cases.
If DNN denoisers induce biases towards the GAHB approximation class, then they should perform sub-optimally for distributions whose optimal bases are not GAHBs. To investigate this, we train DNN denoisers on image classes supported on low-dimensional manifolds, for which the optimal denoising basis is only partially constrained. Specifically, an optimal denoiser (for small noise) should project a noisy image on the tangent space the manifold. We observe that the DNN denoiser closely approximates this projection, but also partially retains content lying within subspace spanned by set of additional GAHB vectors. These suboptimal components reflect the GAHB inductive bias.
2 DIFFUSION MODEL VARIANCE AND DENOISING GENERALIZATION
Consider an unknown image probability density, p(r). Rather than approximating this density directly, diffusion models learn the scores of the distributions of noise-corupted images. Here, we show that the denoising error provides a bound on the density modeling error, and use this to analyze the convergence of the density model.
2.1 DIFFUSION MODELS AND DENOISING
Let = x + 2 where: ~ N(, (1d.) The density Po(y) of noisy images is then related to p(x) through marginalization over a::
Po(y) = p(y)))p(z)dd>
go(y x)) p(x) dr,
(1)
where (2) is the density of z. Hence, (1) is obtained by convolving p(z) with a Gaussian with standard deviation a. The family of densities {po (y): a 2 0) forms scale-space representation of p(x), analogous to the temporal evolution of a diffusion process.
Diffusion models learn an approximation so()) (dropping the a dependence for simplicity) of the scores V log Po (v) of the blurred densities (3) at all noise levels a. The collection of these score models implicitly defines a model (2()) of the density of clean images p(x) through a reverse diffusion process. The error of the generative model, as measured by the KL divergence between p(z) and (I)), is then controlled by the integrated score error across all noise levels (Song al., 2021):
(pl() po(()) <
E V log (v) - so(w)]?] do.
(2)
2

---

Published as a conference paper at ICLR 2024
The key to learning the scores is equation due to Robbins (1956) and Miyasawa (1961) (proved in Appendix D.1 for completeness) that relates them to the mean of the corresponding posteriors:
V lo (v) = [E]] æ y] y)/o².
(3)
The score is learned by training a denoiser fo( to minimize the mean squared error (MSE) (Raphan & Simoncelli, 2011; Vincent, 2011):
MSF(Jo, --)
= E || fo()))
z.y
so that fo(v) 2 [x v]. This estimated conditional mean is used to recover the estimated score using eq. (3); sp(v) (v) y)/o². As we show in Appendix D.2, the error in estimating the density p(x) is bounded by the integrated optimality gap of the denoiser across noise levels:
DKL. (p(x) Po -
MSE(f", ,o") do,
(5)
where (n()) = Egg y] is the optimal denoiser. Thus, learning the true density model is equivalent to performing optimal denoising at all noise levels. Conversely, a suboptimal denoiser introduces a score approximation error, which in turn result in error in the modeled density.
Generally, the optimal denoising function f" (as well as the "true" distribution, p(x)) is unknown for photographic images, which makes numerical evaluation of sub-optimality challenging. We can however separate deviations from optimality arising from model bias and model variance. Model variance measures the size of the approximation class, and hence the strength (or restrictiveness) of the inductive biases. It can be evaluated without knowledge of f*. Here, we define generalization as near-zero model variance (i.e., an absence of overfitting), which is agnostic model bias. This is the subject of Section 2.2. Model bias measures the distance of the true score to the approximation class, and thus the alignment between the inductive biases and the data distribution. In the context of photographic images, visual quality of generated samples can be a qualitative indicator of the model bias, although high visual quality does not necessarily guarantee low model bias. We evaluate model bias in Section 3.2 by considering synthetic image classes for which f is approximately known.
2.2 TRANSITION FROM MEMORIZATION TO GENERALIZATION
DNNs are susceptible to overfitting, because the number of training examples is typically small relative to the model capacity. Since density estimation, in particular, suffers from the curse of dimensionality, overfitting is of more concern in the context of generative models. An overfitted denoiser performs well on training images but fails to generalize to test images, resulting in low- diversity generated images. Consistent with this, several papers have reported that diffusion models can memorize their training data (Somepalli al., 2023; Carlini et al., 2023; Dar al., 2023; Zhang et al., 2023). To directly assess this, we compared denoising performance on training and test data for different training set sizes N. We trained denoisers on subsets of the (downsampled) CelebA dataset (Liu al., 2015) of size N 10" 10a 10 10 10 10 We used a UNet architecture (Ronneberger et al., 2015), which is composed of 3 convolutional encoder and decoder blocks with rectifying non-linearities. These denoisers are universal and blind: they operate on all noise levels without having noise level as an input Mohan et al. (2020). Networks are trained to minimize mean squared error (4). See Appendix A for architecture and training details.
Results are shown in Figure 1. When N = 1, the denoiser essentially memorizes the single training image, leading to high test error. Increasing N substantially increases the performance on the test set while worsening performance on the training set, as the network transitions from memorization to generalization. At N = 10a, empirical test and train error are matched for all noise levels.
To investigate this generalization further, we train denoisers on non-overlgpiing subsets of CelebA of various size N. We then generate samples using the scores learned by each denoiser, through the reverse diffusion algorithm of Kadkhodaie & Simoncelli (2020)-sec see Appendix A for details. Figure 2 shows samples generated by these denoisers, initialized from the same noise sample. For small N, the networks memorize their respective training images. However, for large N, the networks converge to the same score function (and thus sample from the same model density), generating nearly identical samples. This surprising behavior provides a much stronger demonstration of convergence than comparison of average train and test performance.
3

---

Published as a conference paper at ICLR 2024
Train
Test
57-
54-
Ou
24
Impat PANR
Imput PEWR
Figure I: Transition from memorization to generalization, for a UNet denoiser trained on face images. Each curve shows the denoising error (output PSNR, ten times log10 ratio of squared dynamic range to MSE) as a function of noise level (input PSNR), for a training set of size N. As increases, performance on the training set generally worsens (left), while performance (right). For N independent of noise level, sign of memorization. The increase in test performance small noise levels at N = 1000 is indicative of the transition phase from memorization to generalization. At N = 10" test and train PSNR are essentially identical, and the model is no longer overfitting the training data.
the test set improves and N = 10, the train PSNR improves with unit slope, while test PSNR is poor,
3 INDUCTIVE BIASES
The number of samples needed for estimation of an arbitrary probability density grows exponentially with dimensionality (the "curse of dimensionality"). As a result, estimating high- dibbeiimnnsinall distributions is only feasible if one imposes strong constraints or priors over the hypothesis space. In diffusion model, these arise from the network architecture and the optimization algorithm, and are referred to as the inductive biases of the network (Wilson & Izmailov, 2020; Goyal & Bengio, 2022; Griffiths et al., 2023). In Section 2.2, we demonstrated that DNN denoisers can learn scores (and thus density) from relatively small training sets. This generalization result, combined with the high quality of sampled images, is evidence that the inductive biases are well-matched to the "true" distribution of images, allowing the model to rapidly converge good solution through learning. On the contrary, when inductive biases are not aligned with the true distribution, the model will arrive poor solution with high model bias.
For diffusion methods, learning the right density model is equivalent to performing optimal denoising at all noise levels (see Section 2.1). The inductive biases the density model thus arise directly from inductive biases in the denoiser. This connection offers a means of evaluating the accuracy of the learned probability models, which is generally difficult in high-dimensions.
3.1 DENOISING AS SHRINKAGE IN AN ADAPTIVE BASIS
The inductive biases of the DNN denoiser can be studied through an eigendecomposition of its Jacobian. We describe the general properties that are expected for an optimal denoiser, and examine several specific cases for which the optimal solution is partially known.
adaptive basis. To analyze inductive biases, we perform local analysis of denoising estimator é(y) = f(y) by looking at its Jacobian (v)) For simplicity, assume that the Jacobian is symmetric and non-negative (we show below that this holds for the optimal denoiser, and is approximately true of the network Jacobian (Mohan* * et al., 2020)). We can then diagonalize it obtain eigenvalues (v))sssa and eigenvectors (c& (v))
Jacobian cigenvectors
fyy is computed with a DNN denoiser with no additive "bias" parameters, its input-output mapping is piecewise linear, opposed to piecewise affine (Mohan* al., 2020; Romano et al., 2017).
4

---

Published as a conference paper at ICLR 2024
N=1
N=10
N=100
N-1000
N- 10600
N=10000D
Closest image from S₁;
Generated by models trained on S1:
Generated by models trained on Sy:
Closest image from Sy
100000 Sa Sample dcsest
Samples
0.
0.23
3.0 0.3 0.2
0.0
0.25
2.73
Figure 2: Convergence of model variance. Diffusion models are trained non-overlapping subsets S and S2 of face dataset (filtered for duplicates). The subset size N varies from to 10². We then generate a sample from each model with a reverse diffusion algorithm, initialized from the same noise image. Top. For training sets of size N 1 to N 100. the networks memorize, producing samples nearly identical to examples from the training set. For N 1000, generated samples are similar to training example, but show distortions in some regions. This transitional regime corresponds to qualitative change in the shape of the PSNR curve (Figure 1). For N 10". the two networks generate nearly identical samples, which no longer resemble images in their corresponding training sets. Bottom. The distribution of cosine similarity (normalized inner product) between pairs of images generated by the two networks (blue) shifts from left to right with increasing N, showing vanishing model variance. Conversely, the distribution of cosine similarity between generated samples and the most similar image in their corresponding training set (orange) shifts from right to left. For comparison, Appendix shows the distribution of cosine similarities closest pairs between the two training subsets, and additional results the LSUN bedroom dataset (Yu et al., 2015) and for the BF-CNN architecture (Mohan* et al., 2020).
follows that the denoiser mapping can be rewritten in terms of the Jacobian eigendecompositiin as
f(v) V Σ (v(v) (u. (v)) (v).
The denoiser thus be interpreted as performing shrinkage with factors (v) along axes of basis specified by e (y). Note that both the eigenvalues and eigenvectors depend on the noisy image (i.e., both the basis and shrinkage factors adaptive (Milanfar, 2012)).
Even the denoiser is not bias-free, small eigenvalues (y) reveal local invariances the denoising function: small perturbations in the noisy input along the corresponding eigenvectors e, (v) do not affect the denoised output. Intuitively, such invariances are desirable property for denoiser, and they are naturally enforced by minimizing mean squared error (MSE) as expressed with Stein's unbiased risk estimate (SURE, proved in Appendix D.3 for completeness):
MSE(f, o²) = E 2202 tr vf( + - f())11 -oad]
o'd]
To minimize MSE, the denoiser must trade off the approximate "rank" of the Jacobian (the trace is the sum of the eigenvalues) against an estimate of the denoising error: lyy - f(v)l odd. The denoiser thus locally behaves (soft) projection subspace whose dimensionality corresponds to the
5

---

Published as a conference paper at ICLR 2024
rank of the Jacobian. As now explain, this subspace approximates the support of the posterior distribution p(aly), and thus gives a local approximation the support of p(z).
It is shown in Appendix D.1 that the optimal minimum MSE denoiser and its Jacobian given by
(*) = + Vog (v) = B B/alyl,
(8)
()"(v) = ld
log (v) - 2 Cov [ar [v]-
That is, the Jacobian the optimal denoiser is proportional to the posterior covariance matrix, which symmetric and non-negative. This gives another interpretation of the adaptive eigenvector basis as providing an optimal approximation of the unknown clean image I given the noisy observation Further, the optimal denoising error is then given by (see Appendix D.1 for the first equality)
MSE(S*, .o²) = E((r Cov(a v]] a² E L1 E(LrV//())
= o² E Σ XL(W)
(10)
A small denoising error thus implies an approximately low-rank Jacobian (with many small eigenval- ues) and thus an efficient upproximation of given y-
In most cases, the optimal adaptive basis (e (v)) is not known. Rather than aiming for exact optimality, classical analyses (Donoho, 1995) thus focus on the asymptotic decay of the denoising error as the noise level o² falls, up multiplicative constants. This corresponds finding basis (ex (3))isaed which captures the asymptotic slope of the PSNR plots in Figure 1 but not necessarily the intereept. This weaker notion of optimality obtained by showing matching upper and lower- bounds on the asymptotic behavior of the denoising error. To provide intuition, we first consider fixed orthonormal basis eg (1) = Egp and then consider the more general case of best bases selected from fixed dictionary.
Denoising in a fixed basis. Consider a denoising algorithm that is restricted to operate in a fixed basis eg but can adapt its shrinkage factors ' (z). An unreachable lower-bound on the denoising error-and thus an upper-hound on the PSNR slope-is obtained by evaluating the performance of an "oracle" denoiser where the shrinkage factors A depend the unknown clean image rather than the noisy observation y (Mallat, 2008). Appendix D.4 shows that the denoising error of this oracle is
E ( - (x(x))) )c(c)Po7)
(11)
k
which is minimized when {x(x)) = (x,ee)
(a.ex))+o" The coefficient (x) thus acts soft threshold:
(x) s 1 when the signal dominates the noise and (x( 2 0 when the signal is weaker than the noise. Appendix D.4 then shows that the oracle denoising error the expected value of
o² (x,ex)
>x((t)=
(x, ( +00 Σ minn((x, ex) (. 2 Mo² + ||a xf x
E
(12)
where TM - "(xee))>>'"" (2, Ce) eg is the M-term approximation of x with the M basis coefficients (s, e&) above the noise level, and means that the two terms of the same order up to multiplicative constants (here smaller than 2). The denoising error is small if æ has a sparse representation in the basis, so that both A and the approximation error ||x - =I are small. For example, if the coefficients decay (x, ex)² k: (a+1) (up reordering). Appendix D.4 shows that
Mo²
+ || TM 112 ~ 2a/(a+11)
(13)
which is lower bound on the MSE of any denoising algorithm in the basis eg. Reciprocally, this oracle denoising error is nearly reached with a soft-thresholding estimator that computes the shrinkage factors (v) by comparing (y, ex,² (rather than (ar, ex) with threshold proportional to ? (Donoho & Johnstone, 1994), and achieves the decay (13) up to logarithmic factor. The decay (13) of the MSE with decreasing a corresponds to an asymptotic slope of a/(a + 1) in the PSNR curve when the input PSNR increases. Thus, larger sparsity/reullariy exponent a, which corresponds to a faster decay of the small coefficients x in the basis (ex) (a)ssksdd leads to improved denoising performance.

---

Published as a conference paper at ICLR 2024
Clean
Nolsy
Denoised
1244
1.04
mom.1.co 1000
B,93
0.93
App 3.000
Anp
Aue
0.637
Aus Ass=0612
0.569
0.553
App 0.53
Auu 0051
Figure 3: Analysis of a denoiser trained on 10 face images, evaluated on a noisy test image. Top left. Clean, noisy (o 0.15) and denoised images. Bottom left. Decay of shrinkage values (x (red), and corresponding coefficients (zr, ex()) (blue), evaluated for the noisy image The rapid decay of the coefficients indicates that the image content highly concentrated within the preserved subspace. Right. The adaptive basis vectors (B) contain oscillating patterns, adapted to lie along the contours and within smooth regions of the image, whose frequency increases as (y) decreases.
Best adaptive bases. Adapting the basis (S the noisy image y allows obtaining sparser representations of the unknown clean image x with a faster decay, and thus a larger PSNR slope. To calculate the optimal adaptive basis, we need to find an oracle denoiser that has the same asymptotic MSE as a non-oracle denoiser, yielding matching lower and upper bounds on the asymptotic MSE.
Consider an oracle denoiser which performs thresholding in an oracle basis (x)) that depends on the unknown clean image x. The above analysis then still applies, and if the coefficients (z, ex (z))
decay as A (a++1) then the asymptotic PSNR slope is again a/(a+ + 1). The best oracle basis satisfies e, (z) = z/ ||= but yields a loose lower bound as it cannot be estimated from the noisy image y alone. We thus restrict the oracle denoiser to choose the basis (ex (z)) within a fixed dictionary. larger dictionary increases adaptivity, but it then becomes harder to estimate the basis that best represents from y alone. If the dictionary of bases is constructed from number of vectors eg which is polynomial in the dimension (the number of bases can however be exponential in d) then a thresholding in the basis (eg (v)) that best approximates the noisy image y achieves the same slope as the oracle denoiser (Barron et al., 1999; Dossal et al., 2011). This near optimmpttmallty despite the presence of noise comes from the limited choice of possible basis vectors ex. in the dictionary, which limits the variance of the best-basis estimation, e.g. by preventing e, (v) = y/lly/l [lvil- The main difficulty is then to design a small-enough dictionary that gives optimal representations of images from the data distribution in order to achieve the optimal PSNR slope.
We now evaluate the inductive biases of DNN denoisers through this lens. In Section 2, we showed that the DNN denoisers overcome the curse of dimensionality: their variance decays to zero in the generalization regime. In the next section, we explain this observation by demonstrating that they are inductively biased towards adaptive bases (g()) from a particular class.
3.2 GEOMETRY-ADAPTIVE HARMONIC BASES IN DNNS
Figure shows the shrinkage factors (A& (z)), adaptive basis vectors (Ex (y)), and signal coefficients ((x, (2)))) of a DNN denoiser trained on 10S face images. The eigenvectors have oscillating patterns both along the contours and in uniformly regular regions and thus adapt the geometry of the input image. We call this a geometry-adapiie harmonic basis (GAHB). The coefficients are sparse in this basis, and the fast rate of decay of eigenvalues exploits this sparsity. The high quality of generated images and the strong generalization results of Section show that DNN denoisers rely on inductive biases that are well-aligned to photographic image distributions. All of this suggests that DNN denoisers might be inductively biased towards GAHBs. In the following, we provide evidence supporting this conjecture by analyzing networks trained on synthetic datasets where the optimal solution is (approximately) known.
C images and bandlet bases. If DNNs are inductively biased towards GAHBs, we expect that they generalize and converge to the optimal denoising performance when such bases are optimal. We
7

---

Published as a conference paper at ICLR 2024
1L-1
Ap A--2
Maaee 24610.5)
sasee 061 (0.67)
2. Slossm 2.75 10.751 gw3 Slogem 2.55 16-531 Icertity
4=0
Au-6144
A1y-8881
Au
Au -m
Au 6.50
Jo
Au-67H
Insu P5N1
Figure 4: UNet denoisers trained on 10 C images achieve near-optimal performance. Left. PSNR curves for various regularity levels The empirical slopes closely match the theoretical optimal slopes (parenthesized values, dashed lines). Right. A C" image (a = 4) of size 80 x 80 and its top eigenvectors, which consist of harmonics on the regions and harmonics along the boundary. The frequency of the harmonics increases with k. More examples are given in Appendix C.I.
consider the so-called geometric C class of images (Korostelev & Tsybakov, 1993; Donoho, 1999; Peyré & Mallat, 2008) which consist of regular contours regular backgrounds, where the degree of regularity is controlled by a. Examples of these images are shown Figure and Appendix C.I. A mathematical definition and algorithm for their synthesis are presented in Appendix E.
Optimal sparse representations C" images are obtained with "bandlet" bases (Peyré & Mallat, 2008). Bandlets are harmonic functions oscillating different frequencies, whose geometry is adapted to the directional regularity of images along contours. Geometric C images can be represented with few bandlets having (ow-frequency oscillations in regular regions and along contours but sharp
variations across contours. The k-th coefficient the best bandlet basis then decays as -a+++1) follows that the optimal denoiser has PSNR which asymptotically increases with a slope ar(ca= + 1) as a function of input PSNR (Korostelev Tsybakov. 1993; Dossal et al., 2011).
Figure 4 shows that DNN denoisers trained on C images also achieve this optimal rate and learns GAHBs, similarly to bandlets but with a more flexible geometry. This generalization performance confirms that inductive biases of DNNs favor GAHBs.
Low-dimensional manifolds. DNNs are inductively biased towards GAHBs, then we expect these bases emerge even in cases where they are suboptimal. To test this prediction, we consider dataset of disk images with varying positions, sizes, and foreground/backgnng defines a five-dimensional curved manifold, with a tangent space evaluated at disk image x that spanned by deformations of x along these five dimensions. When the noise level a is much smaller than the radius of curvature the manifold, the posterior distribution p(2 yy) is supported on an approximately flat region of the manifold, and the optimal denoiser is approximately projection onto the tangent space. Thus, the optimal Jacobian should have only five non-negligible eigenvalues, whose corresponding cigenvectors span the tangent space. The remaining eigenvectors should have shrinkage factors = 0, but are otherwise unconstrained. The optimal MSE is asymptotically equal to 5 corresponding to a PSNR slope of one.
intensities. This
Figure shows an analysis of a denoiser trained on 10t disk images, of size 80 x 80. We observe additional basis vectors with non-negligible eigenvalues that have GAHB structure, with oscillations on the background region and along the contour of the disk. We also find that the number of non-zero eigenvalues increases as the noise level decreases, leading than 1.0. These results reveal that the inductive biases of the DNN are not perfectly aligned with dindeimesional manifolds, and that the presence of the curvature, this suboptimality increases as the noise level decreases. We obtain similar results on two additional examples of a distribution supported on dim-dimnsii manifold, given in Appendix C.2.
suboptimal PSNR slope that less
Shuffied faces. We also consider in Appendix C.3 a dataset of shuffled faces, obtained by applying
a common permutation to the pixels of each face image. This permutation does not preserve locality between neighboring pixels, and thus the optimal basis does not have harmonic structure. The resulting mismatch between the DNN inductive biases and the data distribution result substantially worse performance than for the original (unscrambled) faces.
8

---

Published as a conference paper at ICLR 2024
Clean
Noisy
Deneksed
A
Ay 1007
6:96
Aum AmAA
ousput PSMR
22
Figure 5: UNet denoiser trained dataset of translating and dilating disks, with variable fore- ground/background intensity. Top center. Clean, noisy (a 0.04), and denoised images. Bottom center. The decay of shrinkage factors A (y) and coefficients (xx ex(y)) indicates that the network achieves and preserves a sparse representation of the true image. Top right. denoising performance sub-optimal, with PSNR slope below the optimal value of 1.0 for small noise. Top left. An optimal basis (in the small-noise limit) spanning the 5-dimensional tangent space of the image manifold. Bottom left. Top eigenvectors of the adaptive basis. The first five basis vectors closely match the basis of the tangent space of the manifold evaluated at the clean image. In contrast, the next five are GAHBs that lie along contours and within background regions of the clean image.
DISCUSSION
Diffusion generative models, which operate through iterative application a trained DNN denoiser, have recently surpassed all previous methods of learning probability models of images. Their training objective (minimization of squared denoising error) is simple and robust, and they generate samples of impressive quality. In this paper, we elucidate the approximation properties that underlic this success, by analyzing the trained denoiser, which is directly related to the score function, and to the density from which the samples are drawn.
We show empirically that diffusion models memorize samples when trained on small sets, but transition strong form of generalization as the training set size increases, converging unique density model that is independent of the specific training samples. The amount of data needed to reach this phase transition very small relative the size of dataset needed for convergence without any inductive biases, and depends on the image size and complexity relative to the neural network capacity (Yoon et al., 2023). of interest to extend both the theory and the empirical studies to account for the interplay of these factors. Appendix B.4 shows preliminary results in this direction.
We also examined the inductive biases that enable this strong generalization. Using well-established mathematical framework, showed that DNN denoisers perform shrinkage of noisy coefficients in cooometry-adattiin harmonic basis (GAHB) which shaped by geometrie features of the image. For the C class of images, such geometric bases known be optimal, and DNN denoisers achieve near-optimal performance on this class. Previous mathematical literature has shown that bandlet bases, which are specific type of GAHB, are near-optimal for this class, but the GAHBs learned by the DNN denoiser are more general and more flexible. For images drawn from low-dimensional manifolds, for which the optimal basis spans the tangent subspace of the manifold, we find that DNN denoisers achieve good denoising within a basis aligned with this subspace, but also incorporate GAHB vectors in the remaining unconstrained dimensions. The suppeeseee noise along these additional GAHB components leads to suboptimal denoising performance. This observation, along with similar ones shown Appendix C.2, provide more supporting evidence for the hypothesis that inductive biases of DNN denoisers promote GAHBs.
We do not provide a formal mathematical definition of the class of GAHBs arising from the inductive biases of DNNs. Convolutions in DNN architectures, whose eigenvectors are sinusoids, presumably engender GAHB harmonic structure, but the geometric adaptivity must arise from interactions with rectification nonlinearities (ReLUs), A more precise elucidation of this GAHB function class, and its role shaping inductive biases of the DNNs used in wide variety of other tasks and modalities, is of fundamental interest.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We gratefully acknowledge the support and computing resources of the Flatiron Institute (a research division of the Simons Foundation), and NSF Award 1922658 to the Center for Data Science at NYU.
REFERENCES
Andrew Barron, Lucien Birgé, and Pascal Massart. Risk bounds for model selection via penalization. Probability theory and related fields, 13:301-413, 1999.
Giulio Biroli, Tony Bonnaire, Valentin de Bortoli, and Mare Mézard. Dynamical regimes of diffusion models. arXiv preprint arXiv:2402.18499, 2024.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023.
Salman UI Hassan Dar, Arman Ghanaat, Jannik Kahmann, Isabelle Ayx, Theano Papavassiliou, Stefan Schoenberg. and Sandy Engelhardd. Investigating data memorization in 3d latent diffusion models for medical image synthesis. arXiv preprint ar arXiv:2307.01 148, 2023.
Donoho. Denoising by soft-thresholding. IEEE Trans Information Theory, 43:613-627, 1995.
David I. Donoho. Wedgelets: Nearly minimax estimation of edges. the Annals of Statistics, 27(3):
859-897, 1999.
David I. Donoho and lain M Johnstone. Ideal spatial adaptation by wavelet shrinkage. biometrika, 81 (3):425-455, 1994.
Ch. Dossal, E. Le Pennec, and S. Mallat. bandlet image estimation with model selection. Signal Processing, 91:2743-2755 2011.
Bradley Efron. Tweedie's formula and selection bias. J American Statistical Association, 106(496): 1602-1614, Dec 2011.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 4778(2266):202100668 2022.
Thomas L Griffiths, Jian-Qiao Zhu, Erin Grant, and R Thomas McCoy. Bayes in the age of intelligent machines. arXiv preprint arXiv:2311.10200 2023.
J Ho, A Jain, and P Abbeel. Denoising diffusion probabilistic models. Adv Neural Information Processing Systems (NeurIPS), 33, 2020.
Kadkhodaie and E P Simoncelli. Solving linear inverse problems using the prior implicit in
denoiser. arXiv preprint arXiv:2007.13640, Jul 2020.
Kadkhodaic and P Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. In Adv Neural Information Processing Systems (NeurIPS*21), volume 34, Dec 2021.
Kadkhodaie, F Guth, S Mallat, and E Simoncelli. Learning multi-scale local conditional probability models of images. In Int'l Conf on Learning Representations (ICLR), Kigali, Rwanda, May 2023.
T Karras, T Aila, S Laine, and Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10190 2018.
A. P. Korostelev and A. B. Tsybakov. Minimax theory of image reconstruction. Springer New York, NY, 1993.
Liu, Luo. X Wang. and X Tang. Deep learning face attributes in the wild. In Proc Int'l Conference Computer Vision (ICCV), Dec 2015.
10

---

Published as a conference paper at ICLR 2024
S Mallat. A wavelet tour of signal processing: The sparse way. Academic Press, 2008.
Peyman Milanfar. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE signal processing magazine, 30(1):106-128, 2012.
K Miyasawa. An empirical Bayes estimator of the mean of normal population. Bull, Inst. Internat. Statist., 38:181-188 1961.
Mohan Z. Kadkhodaie*,
Simoncelli, and C Fernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In Int'l Conf on Learning Representations (ICLR), Addis Ababa, Ethiopia, Apr 2020.
G. Peyré and S. Mallat. Orthogonal bandlet bases for geometric images approximation. Comm. on Pure and Applied Math.,
173-1212, 2008.
M Raphan and E P Simoncelli. Least squares estimation without priors or supervision. Neural Computation, 23(2):374 420, Feb 2011. doi: 10.1162/NECO 000006
H Robbins. An empirical bayes approach to statistics. In Proc Third Berkeley Symposium Mathematical Statistics and Probability, volume I, pp. 157-163. University of CA Press, 1956.
Yaniv Romano, Michael Flad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal Imaging Sciences, 10(4): 1844):8004-844, 2017.
Ronneberger, P Fischer, and Brox. U-net: Convolutional networks for biomedical image segmentation. In Int' I Conf Medical Image Computing and Computer-assisted Intervention, pp. 234-241. Springer, 2015.
Sohl-Dickstein, Weiss, N Maheswaranathan, and Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In Francis Bach and David Blei (eds.), Proc 32nd Int'l Conf on Machine Learning (ICML), volume 37 of Proceedings of Machine Learning Research, pp- Lille, France, 07-09 Jul 2015. PMLR.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048-6058, 2023.
Song and S Ermon. Generative modeling by estimating gradients the data distribution. Adv Neural Information Processing Systems (NeurIPS), 32, 2019.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1422 2021.
Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):16611174, 2011.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708, 2020.
TaeHo Yoon, Joo Young Choi, Schyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICMI. 2023 Workshop on Structured Probabilistic Inference & Generative Modeling, 2023.
Fisher Yu, Ari Seff. Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv 2015.
preprint arXiv:1506.033:
Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency in diffusion models. arXiv preprint arXiv:2310005264, 2023.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing. 26 (7):3142-3155, 2017, doi: 10. 10.1 10.1109/T 2.262200.6...66....

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL DETAILS
A.I TRAINING AND ARCHITECTURE DETAILS
Architectures. We performed empirical experiments using two different architectures: UNet, and BF-CNN. All the denoisers are "bias-free": we remove all additive constants from convolution and natch-normalization operations (i.e., the batch normalization does not subtract the mean). This facilitates unversality (denoisers can operate at all noise levels), and interpretability (network trans- formations homogeneous of order and the Jacobian provides local characterization) Mohan et al. (2020).
UNet networks contain 3 decoder blocks, one mid-level block, and decoder blocks (Ronneberger et al., 2015). Each block consists of 2 convolutional layers followed by ReLU non-linearity and bias-free batch- nornalizaaiiiin.. Each encoder block is followed by 2 spacial down-sampling and 2 fold increase in the number of channels. Each decoder block is followed by a 2 x 2 spacial upsampling and fold reduction of channels. The total number of parameters 7.6m.
BF-CNN networks Mohan et al. (2020) are bias-free versions of DNCNN networks (Zhang et al., 2017), contain convolutional layers with no subsampling, each consisting 64 channels. Fach layer, except for the first and the last, is followed by ReL.U non-linearity and bias-free batch- normalization. All convolutional kernels are size x 3, resulting 700k parameters in total.
Training. We follow the training procedure described in Mohan et al. (2020), minimizing the mean squared error denoising images corrupted by i.i.d. Gaussian noise with standard deviations drawn from the range [0, 1] (relative to image intensity range 00, 1).. Training is carried out batches of size 512, for 1000 epochs. Note that denoisers are universal and blind: they are trained to handle range of noise, and the noise level is not provided input to the denoiser. These properties are exploited by the sampling algorithm, which can operate without manual specification of the step size schedule (Kadkhodaie & Simoncelli, 2020). This method produces high-quality results in generative sampling, as well as sampling conditioned on linear measurements (Kadkhodaie & Simoncelli, 2021).
Datasets. For experiments shown in Figures to 3 and 7, we use the CelebA dataset (Liu et al., 2015) downsampled to 80 80 resolution. For experiments shown in Figures and 10, we use images drawn from the LSUN bedroom dataset (Yu et al., 2015) downsampled to 80 x 80 resolution. This dataset is downsampled to 32 x 32 resolution for experiments shown in Figure 13. For experiments shown in Figure 11 we use CelebA HQ dataset (Karras et al., 2018) downsampled to 40 x 40 resolution.
A.2 SAMPLING ALGORITHM
Sampling from both the DNN denoisers is achieved using the algorithm presented in Kadkhodaie
Simoncelli (2020), which is specified below in Algorithm 1. Aside from initial and final noise levels (ag, oo0), this method uses two hyperparameters h e [0. 1] and 3 e (0, 1), which control the step size and injected noise respectively. We chose 0.01, 8 = 0.1, do 1, and x 0.05.
12

---

Published as a conference paper at ICLR 2024
Algorithm 1 Sampling via ascent of the log-likelihood gradient from a denoiser residual
Require: denoiser J. step size h, stochasticity from injected noise B, initial noise level du. final noise level zo. distribution mean m
1: t
2: Draw 20 ~ ~N,m, Ifdd)
3: while o 2 0c do
4:
t ++1
5:
St f(x;-1) 3tt-1
Compute the from the denoiser residual Compute the current noise level for stopping criterion
6:
dt + ||0
2
2
(1 -3h² - -(1 -11-))²)
7:
8:
Draw 2, I)
9:
33 - x- + hd, + 76
D Perform partial denoiser step and add noise
end while
11:
return 22
B ADDITIONAL NUMERICAL RESULTS ON GENERALIZATION
B. SIMILARITY BETWEEN DATA SUBSETS
Co0 siietarity
coslin
Figure 6: Histogram of cosine similarity between pairs of closest images in the non-overlapping subsets S, and S2 of CelebA (left) and LSUN bedroom (right). Images with similarity score higher than 0.95 removed from the datasets before training to eliminate replicated images. This should be compared with the histograms in Figures and 10.
B.2 GENERALIZATION OF UNET MODEL
In this section, we show that convergence of model variance is robust to the change of data distribution and architecture. The minimum size of the training set, N, for which the model transitions from memorization to generalization indeed depends on the architecture, image size and data distribution. Nevertheless, with enough data, models trained non-overlapping subsets of data converge to virtually the same function.
B.2.1 TRAINED ON CELEBA DATASET
Figure 7: More examples to illustrate convergence of model variance for models shown in Figure 2, at N 10. Samples generated by each denoiser are shown in separate rows, where each column shows same initialization across the networks. The networks generate nearly identical samples, showing convergence the same function.
13

---

Published as a conference paper at ICLR 2024
Figure 8: Bifurcation of trajectories. Sampling trajectories for the two samples shown in the last column of Figure 7. The two diffusson models arrive at different samples starting from the same initial point. The bifurcation of gradients appears emerge somewhere around the middle of the trajectories, which illustrates instabilities predicted by recent dynamical models (Biroli et al., 2024). All the intermediate samples in the trajectories have been denoised in a on-shot denoising manner using the corresponding denoisers. This example shows that the convergence is not perfect, hence the distribution of cosine similarities at N = 10 is not perfectly delta function at 1.
B.2.2 TRAINED ON LSUN BEDROOM DATASET
Train
Test
62
N:
Idenniity
12
nout
Figure 9: Transition from memorization to generalization, for a UNet denoiser trained on bedroom LSUN images (Yu et al., 2015) downsampled to 80 x 80. Similarly to denoisers trained on face images shown in Figure 1, the model transitions from memorizing the training set to generalizing outside of the training set. At N = 10 the performance almost identical on training and test sets, and the model is no longer overfitting the training data.
14

---

Published as a conference paper at ICLR 2024
N=1
N=
N 100
N-10B0
N-10000
N 1 100000
Closest image from Sy:
Generated by models trained on Sy
Generated by models trained on Sz:
Closest image from S2:
N 10
N 1000
N 10000
N 100000
Samslactro tat
and Image
0.25 3.0
5.75
0.75
Figure 10: Convergence of model variance. Diffusion models are trained on non-overlapping subsets S and Sy of bedroom LSUN dataset. The subset size N varies from to 10 Notice the samples generated by network trained on N 100 images: they combinations of patches of training images. This type of memorization has been previously reported in (Somepalli et al., 2023). See caption of Figure 2 for complete description of the figure.
B.3 GENERALIZATION OF BF-CNN MODEL
B.3.1 TRAINED ON CELEBA DATASET
Train
Test
63
54
Identity
21
30
18
27
10
Figure 11: Transition from memorization to generalization, for a BF-CNN denoiser trained on CelebA
HQ dataset (Karras al., 2018) downsampled to 40 40 resolution. See caption of Figure 1.
15

---

Published as a conference paper at ICLR 2024
N-1
N=10
N-100
N=1000
10000
Closest image from $1z
Generated by models trained on S₁
Generated by models trained on Sz:
Closest image from S
N 10
N 100
1000
10000
Samples TAE deno cre ciocss bult image
Figure 12: Convergence of model variance. BF-CNN denoisers are trained on non-overlapping subsets S, and S₂ CelebA HQ dataset. The subset size N varies from to 10*. See caption of Figure 2.
B.3.2 TRAINED ON LSUN BEDROOM DATASET
Figure 13: Convergence of model variance on LSUN bedroom dataset (Yu et al., 2015). dataset of bedroom images is partitioned into two non non-overlapping datasets, S₁ and S,, each containing N 20, 000 images down-sampled to size x 32. We train two networks (BF-CNN architecture described in Appendix A) S2 and Sz. Each network is then used in an iterative deterministic reverse diffusion algorithm to generate sample, with both networks initialized with the same noise image. Samples generated by each denoiser are shown in separate rows, where each column shows same initialization across the networks. The networks generate nearly identical samples, showing convergence the same function.
16

---

Published as a conference paper at ICLR 2024
N 20000
denetsers
Figure 14: Blue histogram: cosine similarity between samples generated by two denoisers trained on non-overlapping training sets of size N 20, 000 from LSUN bedroom dataset downsampled to 32 x 32 resolution. Orange histograms: cosine similarity between generated samples and the closest image from the corresponding training set. Images drawn from the two denoisers are very similar to each other, compared to the closest image in their respective training sets.
B.4 CONVERGENCE AS FUNCTION OF TRAINING SET SIZE N AND IMAGE RESOLUTION
100.0
PSNR gap (Train Test)
1.0
0.1
20x20
0.01
40x40
80x80
160x160
1.0
10.0
200.0
1000,0
10000.0
100000.0
Figure 15: Generalization as function training set size, N. Generalization measured as the difference between train and test PSNRs averaged across noise levels E [0, 1). Each curve shows average PSNR gap for specific image resolution d x d. The capacity of the UNet is adjusted to the image resolution. See Table for specific UNet architecture used for each image resolution. As expected, to reach the threshold PSNR gap, denoisers trained on larger images require more training data. However, the number of training images does not increase proportionally with the image size: the increase in needed to hit the threshold AN from 80 x 80 to 160 x 160 images is much smaller than the AN from 40 40 80 80. This observation is consistent with previous reports indicating that conditioned on coarser content of the image, learning the finer details requires less data due to the conditional Markov property images (Kadkhodaic et al., 2023).
Image resolution
number of encoder
Receptive field size
number of parameters
decoder blocks
20 20
18 18
360k
40 x 40
44 x 44
1.8m
80 80
92 92
7.6m
160 x 160
188 x 188
31m
Table UNet architectures used in experiments shown in Figure 15. With the four-fold increase of the image size, the number of parameters increases approximately four times.
17

---

Published as a conference paper at ICLR 2024
C
ADDITIONAL NUMERICAL RESULTS ON INDUCTIVE BIASES
C.I MORE C" EXAMPLES
slapem 0.55 10.51
d slspe 0.74 (0.75) slope
output PSMP
Icentity
75
Input PSNR
1.02
0.68
1p51775 1232
Ap-10024
1024
0.99.99
0g.00.922 0.982
A.550.95
0:96
Aut 0.700
Au- 0.824
Figure 16: BF-CNN denoisers trained on C images of size 40 x 40 achieve near-optimal performance. Top. PSNR curves of trained networks for various regularity levels The empirical slopes achieved for different values of a closely match the optimal slopes (dashed lines). Bottom. Eigenvectors for two C images (top row: and harmonics along the boundary. The frequency of the harmonics increases with k. For less regular images, the harmonics are more localized along the contours.
4. bottom row: 2), which consist of harmonies the two regions
Clean
Notsy
Denoised
Clean
Noisy
Denoised
Clean
Noiay
Denoised
2=1.007.0
A
0.9..18.22
0.9.22..14
0=000010
A=00701.28
0
A=0.50043
A=
A=0.62164
A=0.01167
1.0
0
A 900
0:957
0 0.903
0
0.699
A
0.5555
A=4.977
2=101 2m0977
0.29
3.766
m
Am0.63
Am0573
Aw0.488
Am0.499
Figure 17: Ceometric-adaptii harmonic basis shown for three test images from C class. Here the
regularity of the one-dimensional contours Or different from the regularity of the two-dimensional background 02g. Top. Three example images. The regularity of the contour increases from left to right: 0 = 1.5, 2, 4. Background regularity is the same in all three examples, 0r = 8, and = 0.2. Bottom. Top 10 basis vectors for each image are shown. With increasing a, the contours become more regular, and the harmonics along the boundaries become less localized. This allows for a faster decay of coefficients and a lower denoising error.
18

---

Published as a conference paper at ICLR 2024
A- 123
A-0997
Figure 18: Top. additional example a C test image with a Bottom, Top eigenvectors of the geometric harmonic adaptive basis.
C.2 ADDITIONAL LOW-DIMENSIONAL MANIFOLD EXAMPLES
Cean
Notty
Denatord
12.
400
08
3000
1200
1400
150
Figure 19: BF-CNN denoiser trained on single face image, with intensity rescaling. We consider an image class consisting of a single image I IRd and its positive rescalings 8 z for s > 0. The resulting images lie ray emanating from the origin, and optimal denoising corresponds to projecting the noisy image onto this ray. The optimal denoising basis should therefore include the normalized vector 2 ||=||| with associated shrinkage factor À = 1, whereas the remaining basis vectors should have shrinkage factors of À 0 but are otherwise unconstrained. This optimal denoiser achieves an MSE of o² and thus linear PSNR curve with unit slope and intercept 10 log (d)). Top left. Denoising of the training image with 0.04. Right. Decay of the coefficients (I, eq) and the shrinkage factors Ay. The DNN denoiser exhibits slower decay of shrinkage factors than the optimal solution, which results suboptimal performance. Bottom left. Top basis vectors (y). The first basis vector is nearly identical to the (normalized) train image. The next vectors, which have non-zero shrinkage factors, exhibit 2D harmonics. These GAHBB components underlie the non-optimal behavior of the denoiser. Specifically, the N = 1 curve in the left panel of Figure 11 function noise level falls below the optimal solution (dotted line). The intercept
shows that performance DNN performance has unit slope over most of the noise range but has a less-than-opting (the flattening of the curve at small noise levels is result of de-emphasis of small noise levels during training).
19

---

Published as a conference paper at ICLR 2024
Clean
Noisy
Denoised
20
15
06
00
-100
B00
1000
1200
3400
An-
2 Ar
Au A- A
Figure 20: A BF-CNN denoiser is trained set of 2D sine wave images with unit frequency and varying phases and intensities. The train images thus lie on a 2D cone manifold with low curvature. For small the manifold can be assumed to be locally flat, that the optimal denoising achieved by projecting the noisy image on the two-dimensional subspace tangent to the manifold. This subspace is spanned by sine waves with unit frequency and a / phase shift. Top left. Clean, noisy (o 0.08), and denoised test image. Middle left. The unit vectors spanning the tangent subspace. The optimal denoising results from projection Empirical basis obtained from the network Jacobian. The empirical solution has a slower decay than optimal (i.e.. (a, ex 0 for k 2, seen in the right panel), with harmonic patterns. This sub-optimality reveals the nature of the inductive bias.
this subspace. Bottom row.
C.3
SHUFFLED FACES
Clean
Destied
25
PONK
Inpu: PSNR
Figure 21: DNN denoiser trained on dataset of shuffled faces obtained by permuting the pixels of 100 face images in the CelebA dataset. The permutation was chosen randomly, and does not preserve locality, neighboring pixels mapped independent positions. By construction, the optimal denoiser on shuffled faces has the same performance as the optimal denoiser on ordinary faces (unshuffling the image pixels, optimally denoising the face image, and then shuffling the pixels back). For visualization purposes, we "unshuffle" the pixels by applying the inverse of the permutation to the images before display. Top left. Clean (shuffled then unshuffled), noisy (unshuffled, and denoised (unshuffled) images. Middle. The shrinkage factors X (v) decay more slowly than when the denoiser is trained on non-shuffled faces (Figure 3), which is indicative suboptimality. Right. The denoiser performs significantly worse than the denoiser trained on unshuffled faces: the MSE is much higher with a much lower PSNR slope. Bottom left. Basis vectors (top row: shuffled, bottom row: unshuffled). After unshuffling, observe GAHBs adapted to the geometry of the face, although these are noisier and less precisely aligned with the image features than the non-shuffled examples in Figure 3.
0.3),
20

---

Published as a conference paper at ICLR 2024
D MATHEMATICAL DERIVATIONS
D.1 MIYASAWA RELATIONSHIPS
The relationship of MMSE estimation of signal corrupted by additive Gaussian noise the score was published in Miyasawa (1961), and generalized in Raphan & Simoncelli (201 1); Efron (201 1). For completeness, and notational consistency, we provide derivation here. We begin by expressing the score V log p(y) (dropping the dependence to simplify notation) and its Jacobian V log p(y) in terms of the measurement density p(y (which is Gaussian) and the posterior density p(a y). Using Bayes' rule and marginalization, the probability density of the noisy images is expressed as
ply) - p(x)) plylx) dr.
Taking the logarithm and differentiating with respect to and using the fact that for any function h, Vh(y) h(y) V log hly), find
Vlogp(y) / p(x) p())))(()) V, logr(ylz) dr / p(y)
p v, log
E [V, ((yl()) y).
(14)
which can be thought of as an equivalent of the chain rule on the scores as opposed to the densities. Differentiating again with respect to y, we have
22 log = p(zly) v, log logg(<))) logglull logp(ulz)) + v2 logp(ulz)) dr.
(15)
The term V, logp(xly) can be calculated by differentiating the logarithm of Bayes rule:
logp(zle) =
- log p(y) + logp(z),
Vy logl()))) V, log p(y/z) logp(y),
(16)
so that when injected into eq. (15) we obtain
22 loggl(y = / p(z|y) (V, logp(ul>) V Vlog((y) V, logl(
V log p(u) dx
= E (V, logp(y)x) llgp(v)) v, logp(ylæ)) y +
E
logp(y/>) y
= Cov V log p(yl=)))))))))))))(())))) y]]
+ E 2 logp(ulæ) y
(17)
where the last line used v log pvv) E V log p(ylæ) w].
We then use the fact that y obtained from æ by adding Gaussian white noise of variance IId:
1
logp(vlæ)
|I
+ est,
(18)
20
log p(ylx)
σ² (y x),
(19)
logp(u/>>
o² Id,
(20)
so that eqs. (14) and (17) become
V logg(y) (E[x y] y),
log
Cov(z lv] - ld.
21

---

Published as a conference paper at ICLR 2024
Finally, the above identities can be rearranged to yield the first- and second-order Miyasawa relation- ships:
B| M] = y + o V logp(u),
(21)
Cov [z y = o² (id 0² logply))
(22)
Note that the optimal denoising error satisfies
- Elr MIl]]]
L tr
B|z ul) - B| b))"
y
Eitr Cov(r [y]].
D.2 CONTROL ON KULLBACK LEIBLER DIVERGENCE
Equation (2) results from Theorem 1 of Song et al. (2021), considering the so-called "variance-
exploding" SDE dz, dw, where (wx) is Brownian motion (t O then corresponds to the noise variance), and letting the stopping time T' go to infinity.
To reformulate the score-matching error denoising objective, we insert the Miyasawa equation (3) as well as the expression of the score model so (v) (f)()) y)/a² into the score-matching enror:
log Po (v) - s0(2)] = - ||ا vl - fa(w))
(23)
We recall the decomposition of the denoising error when conditioning on
E || - fo())) = E ||= - Eix =1ll
+
EE
[E x vl {=
(24)
so that inserting eq. (24) into eq. (23) yields
V (w) - se(n))1 =
E || - fo(w))1 - E | - Ex wlll()
1
-
-MSE(F",')) o²))
Combined with eq. (2), this proves eq. (5).
D.3 SURE OBJECTIVE
We decompose the MSE as follows:
1 || [())]] = [[y f(y)) - (y - =]]] 21)
= lly - f(u))1] - -2E x. - f(y))} + E [ly x|
(25)
The last term is the total variance of the noise and is thus equal to a²d. dd The middle term can be rewritten with an integration by parts, using the fact that y x -o² - V, log p(ylæ))
B - x,y f(y)i = -o² // V (ogg(()))) y f(v)) p(x) p(y|z)) dr dy,
002
(V,p(y/z),)) f(v)) p(=) dx dy.
2 ||
(())) dx dy,
o B(d - tr [w)]
(26)
Inserting eq. (26) into eq. (25), we then obtain
| s())1
T.. |ly
f())) + 20 "E(r vS(y) o²d,
(27)
proving the Stein's Unbiased Risk Estimator of the MSE.
22

---

Published as a conference paper at ICLR 2024
D.4 OPTIMAL THRESHOLDING IN A BASIS
For completeness, we derive here the error of the fixed-basis oracle denoiser (Donoho & Johnstone, 1994; Donoho, 1995; Mallat, 2008).
We consider an oracle denoiser which computes
(x(( (y, ex) ex-
In practice, the denoiser does not have access to the clean image and the shrinkage factors Xg. thus have to be estimated from the noisy image y alone. Note however that optimizing this oracle estimator is non-trivial as the shrinkage factors have be independent from the noise.
We can then compute its denoising error on a clean image x by averaging over the noise
E
(x) (w) Ck
x = E Σ ((x,ek) - (x(x) (v, x
E
((( - (x(x)) - (2, x
(1 (I=)A {x(())
(28)
where the last step used the fact that (z) o( independently from For each and k the optimal oracle shrinkage factor (x( thus minimizes the quadratic function
(1 - (x)) (x, (x) + 0z
which is achieved when
>x(x) (z, (x)
(29)
(I) +0"
Injecting eq. (29) into eq. (28) gives the denoising error
lz ex)
Σ Ac(x)) (v, ex) ek
Σ
(30)
(1, +
Incidentally, this error is also equal o² Ex (x))), similarly the optimal denoiser as shown in eq. (10).
(mree)" is of the same order as min((x, ex)² ) up to a factor of 2, as we have the The fraction (Ir.e)) to" inequalities for any a, 6
min(a, b) S ab a min(a, b),
2
which follow from ab min b) max(a, b) and max(a, b) S a aabbs (max(a,b)) b). We thus have
E
(x( (y, (x) ex
a ~
min (z, (x) 2,00
0²
Σ
(x,
(31)
(x.0g) 2o0
Let M be the number of terms in the left sum (that is, the number of ranks i such that (x, ex)² > o²),
and x = (( >00 (x, Ex) e be the M-term approximation æ. then have
|| - TM 2=
Σ
(cr ex
Σ (I,
(32)
=o0
23

---

Published as a conference paper at ICLR 2024
so that plugging eq. (32) into eq. (31) gives
E
(z) ex)
Mo² +1- - zM
(33)
This realizes a decomposition of the oracle denoising error into a denoising bias || - F II which corresponds to the signal variance that has been lost, and denoising variance Mo which cor- responds to the noise variance that has been preserved (note that denoising bias and variance different than the model variance and model bias studied in the paper). The sum of the two terms captures the efficiency of the approximation of in the basis (ex)-
Let us reorder the coefficients so that (ar, exr, 1), and assume that (x, ex) ~ k -(a+)) for some a > O. More precisely, we assume that there exists two constants e' independent of and & such that e -(++1) I (x, S dd k (a+1) By definition of M,
(xr, 2a) (note that the ordering depends
(x, eM) 20² (20, eM+1
so that
d' 1M-(a+)) > 0² 2 c(M + 1) ((+1)
We then have M- (a+1) 2 i.e. M ~ 2/(a++1) and thus Mo² 20/la+1), We also have
f too ((a+1)
d' M
Σ ex) <d' Σ k: (a+1)
<d'
KSM
M
a
Σ (z, (x)
k (a+1) 2 c -lat1)
- (M +1)
K
M+1
so that || 12 M σ 2x/(a+1) Finally, we have shown that the two terms in eq. (33) of the same order, and it follows that
Σ (e(x) (U,e
Mo² + || xMIl? 2a/(a+1)
Because the constants have been assumed to be independent of x, one can average over x to obtain that the oracle MSE is 2a/(a+))
E GEOMETRIC Co IMAGES
continuous image x: 1] + is part the geometric C class (Korostelev Tsybakov, 1993; Donoho, 1999; Peyré & Mallat, 2008) it is uniformly a-Lipschitz over [0, 1|2 \ {m),, where the are uniformly a-Lipschitz curves in j0, 1]² which do not intersect tangentially. A function f is uniformly a-Liischitz over a domain Q if there exists a constant C such that for all Ω there exists polynomial 4u of degree |a] such that for all E Ω
If()) 9u (v)) < C | yl
(34)
We explain how to generate numerically such images in Algorithm 2.
24

---

Published as a conference paper at ICLR 2024
Algorithm 2 Synthesis of C" image via integration
Require: regularity a, Fast Fourier Transform FFT
1: Make contour
2: Define a ID filter fi(w) = |-|-a
3: Draw random ID C function with i.idd. uniform entries c(t) ~ (((-0.5,055
4: Integrate in the Fourier domain to define C FFT x FFF(())
5: Make the background
6: Define a 2D filter f2(w) (wi + w
7: Draw two random 2D C functions with i..d. uniform entries 61 (I, y), -.([0,,,.0
8: Integrate in the Fourier domain to define B; = FFT FFT "(2 x (FT( (i=1,2)
9: Make mask and combine
10: Define a binary mask M = 1
Let r M B1 (1 MM) B2
12: return x
25


Reviews:
[{"review_id": "oXoNoMQswW", "reviewer": "Reviewer_rH4m", "summary": "The authors study the issue of generalization and memorization in diffusion\nmodels. Diffusion models have as a backbone trained denoisers, and it has been\nobserved that the use of powerful deep networks as denoisers can lead to\nsituations where diffusion models memorize their training data (rather than\nbeing able to generate novel images from the high-dimensional data\ndistribution), whereas in other cases the models seem to generalize. The authors\nstudy this for a specific class of deep network denoisers (bias-free CNNs) both\nempirically and theoretically. Empirically, they show that training these\ndenoisers on CelebA and LSUN bedrooms for varying training set sizes witnesses a\nclear transition (qualitative and quantitative) between \"memorization\"\nperformance of the trained diffusion model when the training set size is small,\nand \"generalization\" performance when it is sufficiently large. The authors\nhypothesize that the ability to generalize with relatively few samples from the\nhigh-dimensional distribution is due to inductive biases in the deep network\ndenoiser, and in particular they make a connection with classical ideas on\ndenoising from harmonic analysis to posit that DNN denoisers are biased towards\n\"geometry-adaptive harmonic bases\" (eg bandlets/wedgelets).  They test this\nhypothesis empirically, demonstrating that these trained denoisers learn\nbandlet-like bases in which to perform shrinkage on toy image classes where such\nbases are optimal (horizon classes), and moreover that they persist in learning\nthese types of bases even for classes of signals where they are suboptimal\n(image articulation manifolds).", "questions": "- How robust the main empirical insights are to the\n  specific training setup and model architecture being studied in the\n  experiments? \n  For example, what if one used a modern noise-conditional diffusion model\n  instead of training a single unconditioned model on multiple noise scales;\n  what if one considered larger-scale training (eg on datasets of\n  higher-resolution, photorealistic images, as one considers for modern\n  diffusion models); what if one performed generation with an ODE/SDE-type\n  sampling procedure, rather than Algorithm 1 used in the paper? It would be\n  helpful to know how the authors see Figures 1 and 2 translating into these\n  settings, whether they would expect changes, etc.\n\n- Could you clarify behind the thinking behind the claim in the discussion that\n  \"deep networks are more adapted towards high-dimensional structures [than\n  low-dimensional structures]\"? Although I think I follow at a high-level --- eg\n  in Figure 5, the adaptive basis at the noisy image is not exactly equal to the\n  optimal 5-dimensional basis for the low-dimensional tangent space --- it still\n  seems to me that there is a bias in what has been learned towards\n  low-dimensional structure, because the correct basis for the true tangent\n  space seems to be contained in the actual adaptive basis, and the eigenvalues\n  of the Jacobian decay at a reasonable rate (making the spectrum overall\n  \"compressed\", if not sparse/low-dimensional). Hence when I try to interrogate\n  the claim at a lower level of detail, I cannot exactly 'compile' it.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper presents detailed (but not overly technical/complex) background on\n  diffusion models, to allow a broad audience to appreciate the experimental\n  results. (I do wonder here if it might be helpful to present somewhere the\n  functional form of the 'optimal' denoiser for the empirical risk, eg as it's\n  done in Karras et al 2021, to ground the memorization issue a bit more -- I\n  find this concept helpful to have in mind as I'm reading the background of the\n  paper.)\n\n- The core question the paper considers -- issues of generalization in diffusion\n  models trained to generate samples from high-dimensional data distributions --\n  is both important and currently without a completely satisfying\n  conceptual/mathematical explanation, despite much concurrent work. The authors\n  present a compelling empirical study of this issue under controlled\n  conditions, showing (among other things) that it is indeed real, and give a\n  plausible theoretical explanation for why it occurs.\n\n- The theory of inductive bias that the authors put forth is nontrivial,\n  involving, through equation (6), a specific representational formula for\n  piecewise-linear denoisers, and thereby a connection to classical shrinkage\n  estimators.", "weaknesses": "- It seems that the theoretical framework\n  posited in section 3 is specific to the particular denoiser architectures\n  being studied in the paper (BF-CNN). For example, PixelNet++-type denoisers\n  used in modern diffusion models do not seem to be amenable to a decomposition\n  like equation (6), because they involve attention layers and positional\n  embeddings (presumably with affine components) to implement different\n  conditioning operations. It would be good if the authors could comment on this\n  issue, and how they see the theory extending to this modern setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55DcG6nWm7", "reviewer": "Reviewer_4Rws", "summary": "The authors address recent concerns regarding the memorization of denoising DNNs for generative image modeling by showing that two networks trained on disjoint datasets converge to the same score/density and hence generate similar images. Those experiments are supported by theoretical derivations clearly relating the inductive bias of the denoiser to that of the density. Leveraging recent work, e.g., (Monoho et al.,2020), the authors elucidate the sparse optimal basis, adapted to the geometry of the input image, along which denoising can be understood as a shrinkage operation. Furthermore, the authors validate and connect the results by evaluating the inductive bias on image classes with known optimal bases.", "questions": "**Presentation:**\n- Abstract:\n    - Please mention the empirical nature of those findings, while also highlighting the supporting theoretical derivations.\n    - It would help to point to the newly highlighted GAHB as a particularly interesting topic for future work, as done on the very last paragraph.\n- Section 2:\n    - It would help to point to the figures within the main text.\n    - Related to this: in many places the authors use figure captions the same way as main text, more so in the appendices. (understandable if that was wrapped up close to the deadline) I strongly recommend to add more supporting text, with pointers to parts in the main text to which each figure is most relevant. That is, to collect those pieces into a coherent narrative that's easy to follow.\n- Section 3:\n    - S3.1 seems to be largely based on (Mohan et al., 2020). If so, please state this clearly on the onset, or make it clear where the new contributions begin. One point where a citation seemed needed is the notion of projection below Eq.7.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Presents compelling empirical evidence, supported by plausible theoretical derivations, of the inductive bias of denoising DNNs.\n- Explains with clear examples how they transition from memorization to generalization (even when trained on disjoint partitions of the datasets).\n- Repeats the analysis on 2 datasets.\n- Validates on image classes with known optimal bases.", "weaknesses": "Nothing stands out, just needs to finalize the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RYRddAksK6", "reviewer": "Reviewer_SQqg", "summary": "Diffusion generative models that use denoising DNNs have surpassed previous methods of learning probability models from images. The authors introduce a methodology to evaluate the properties of the trained denoiser and the density from which the data are drawn. They show empirically that diffusion models can converge to a unique continuous density model that is independent of the specific training samples. The convergence exhibits a phase transition between memorization and generalization as training data grows. They demonstrate that two denoising DNNs trained on non-overlapping subsets of a dataset learn nearly the same score function and, thus, the same density, indicating the existence of powerful inductive biases in the DNN architecture and/or training algorithm. The inductive bias of the network appears through the best basis, which is a geometry-adaptive harmonic basis (GAHB) when trained on photographic images. The DNN denoisers achieve near-optimal performance for the Cα class of images. They also investigate the inductive biases that enable this rapid convergence and show that DNN denoisers perform poorly for distributions whose optimal bases are not GAHB. For images drawn from low-dimensional manifolds, DNN denoisers achieve an accurate basis for the subspace and incorporate GAHB vectors in the remaining unconstrained dimensions. The denoiser performs a shrinkage operation on a basis adapted to the underlying image, which reveals oscillating harmonic structures along contours and inhomogeneous image regions. The paper leaves important open questions regarding the formal mathematical definition of this larger class of GAHB bases and how they result from the DNN computational architecture.", "questions": "Would you be able to add some of this line of literature to your discussion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a well-written paper that puts forth a compelling hypothesis.  The hypothesis is presented in a clear and concise manner, and the paper's overall readability is very good. The authors suggest that, instead of attempting to learn a low dimensional structure, DNN denoisers learn a well-regularized geometry-adaptive harmonic basis (GAHB) with small coefficients. Hence, the denoising algorithm performs a shrinkage operation on an image-adapted basis, explaining some of the impressive results that have been observed recently in the literature. \n\nThe authors provide a counter-example to argue that if the hypothesis were not true, the DNN denoisers should perform poorly for those images for which GAHB is not the optimal basis. They construct such an example dataset using images drawn from low-dimensional structures and show that the trained DNN denoiser is not perfectly aligned with the manifold, and the bias increases with increasing noise.  Similar results were reported by creating another dataset using shuffled versions of the CelebA dataset, which would also not have GAHB as the optimal basis.", "weaknesses": "The paper's results were obtained using a straightforward CNN architecture. However, prior studies have indicated that CNN architectures can effectively learn and utilize harmonic bases (https://arxiv.org/abs/1810.12136). In my opinion, the paper's only weakness lies in not acknowledging this perspective that has been previously established in the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9tzjYhFzaa", "reviewer": "Reviewer_89Fo", "summary": "The paper studies the inductive bias of the models which involve a denoising step, i.e., score-based diffusion algorithms. To do so, the authors take a spectral approach and analyze the eigenspace and eigenvalues of the corresponding DNN denoiser Jacobian. The authors conjecture that the DNN denoiser are implicitly biased toward so called geometry-adaptive harmonic bases (GAHBs). The authors first show that such biases emerge on a set of synthetic $C^{\\alpha}$ images. Further strengthening a point,  the authors empirically show that the bias still emergences similarly even in suboptimal scenario (disk images and shuffled CelebA).", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the paper is easy to follow and elaborate\n- a good set experiments which validate the main points of paper fairly, e.g., optimal PSNR for $C^\\alpha$ images and slow eigenvalues decay on CelebA\n- clean mathematical framework that is used as a foundation for the empirical part \n- indication of the fact that DNNs are adapted to the high-dim structures (with certain regularity) that allows to infer them from small portions of data", "weaknesses": "- further validation on a more realistic data might be beneficial to strengthen the main points of the paper\n- lack of any sort of description of more \"general DNN GAHBs\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)