Continuous-Time Functional Diffusion Processes
Authors: Giulio Franzese
Giulio Corallo
Simone Rossi
Markus Heinonen
Maurizio Filippone
Pietro Michiardi
Venue: NeurIPS 2023 poster (2023)
Keywords: Hilbert spaces
Diffusion models
Stochastic Partial Differential Equations

Abstract: We introduce Functional Diffusion Processes (FDPs), which generalize score-based diffusion models to infinite-dimensional function spaces. FDPs require a new mathematical framework to describe the forward and backward dynamics, and several extensions to derive practical training objectives. These include infinite-dimensional versions of Girsanov theorem, in order to be able to compute an ELBO, and of the sampling theorem, in order to guarantee that functional evaluations in a countable set of points are equivalent to infinite-dimensional functions. We use FDPs to build a new breed of generative models in function spaces, which do not require specialized network architectures, and that can work with any kind of continuous data.
Our results on real data show that FDPs achieve high-quality image generation, using a simple MLP architecture with orders of magnitude fewer parameters than existing diffusion models.

Continuous-Time Functional Diffusion Processes
Giulio Franzese EURECOM, France
Giulio Corallo FURECOM, France
Simone Rossi" Stellantis, France
Markus Heinonen Aalto University, Finland
Maurizio Filippone EURECOM, France
Pietro Michiardi EURECOM, France
Abstract
We introduce Functional Diffusion Processes (FDPS), which generalize score- based diffusion models to infiniteddimensional function spaces. FDPs require a new mathematical framework to describe the forward and backward dynamics, and several extensions to derive practical training objectives. These include infinite- dimensional versions of Girsanov theorem, in order to be able to compute an ELBO, and of the sampling theorem, in order to guarantee that functional evaluations in a countable set of points are equivalent to Slinite-dimensions functions. We use FDPs build new breed of generative models function spaces, which do not require specialized network architectures, and that can work with any kind continuous data. Our results on real data show that FDPs achieve high-quality image generation, using simple MI.P architecture with orders of magnitude fewer parameters than existing diffusion models. Code available here.
1 Introduction
Diffusion models have recently gained a lot of attention both from academia and industry. The seminal work on denoising diffusion DSohl-Dickstein et al., 2015)) has spurred interest in the understanding of such models from several perspectives, ranging from denoising autoencoders (Vincent) 201 with
multiple noise levels (Ho et al. 2020). variational interpretations (Kingma et al. 2021 annealed (Song & Ermon 2019) and continuous-time score matching (Song & Ermon] 2020; Song al.) 2021).
Several recent extensions of the theory underpinning diffusion models tackle alternatives to Gaussian noise (Bansal et al.) 2022; Rissanen et al., 2022), second order dynamics (Dockhorn el al. 2022). and
improved training and sampling (Xiao et al. 2022 Kim et al. 2022b Franzese et al. 2022).
Diffusion models have rapidly become the go-to approach for generative modeling, surpassing GANs DDhariwal & Nichol, (021) for image generation, and have recently been applied to various modalities
such as audio (Kong al. 2021 Liu al.] [2022), video (Ho et al. [2022] He et al., (2022), molecular structures and general 3D shapes (Trippe et al. 2022 Hoogeboom et al. [022]] Luo & Hu 2021
Zeng et al. 2022). Recently, the generation of diverse and realistic data modalities (images, videos. sound) from open-ended text prompts (Ramesh et al. 2022]] (ahaaiia et al., [202 Rombach et al. 2022] has projected practitioners into a whole new paradigm for content creation.
A common trait of diffusion models is the need to understand their design space (Karras et al. 2022 and tailor the inner working parts the chosen application and data domain. Diffusion models require specialization, ranging from architectural choices of neural networks used to approximate the score (Dhariwal& & Nichol 2021 Karras al. (022), fine- ane-ggrinedd details such an appropriate definition of noise schedule (Dhariwal & Nichol 2021; (Salimans & Hoj 2022). and mechanisms to deal with resolution and scale (Ho et al.) 2021). Clearly, the data domain impacts profoundly such design choices. As a consequence, a growing body of work has focused on the projection of data
"This work was done while working at EURECOM
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

modalities into a latent space (Rombach et al. 2022), either by using auxiliary models such as VAEs (Vahdat et al. 202I). or by using functional data representation (Dupont al. 2022a). These approaches lead to increased efficiency, because they operate on smaller dimensional spaces, and constitute step toward broadening the applicability of diffusion models to general data.
The idea of modelling data with continuous functions has several advantages (Dupont et al. 2022a): allows working with data at arbitrary resolutions, it enjoys improved memory-efficiencyy and it allows simple architectures to represent a variety of data modalities. However, a theoretically grounded understanding of how diffusion models can operate directly on continuous functions has been elusive so far. Preliminary studies apply established diffusion algorithms on a discretization of functional data by conditioning on point-wise values (Dutordoir et al.] [022] Zhuangg al. 2023). A line of work that is closely related to ours include approaches such as Kerrigan et al. (2022), who consider a Gaussian noise corruption process Hilbert space and derive loss function formulated on
measures to approximate the conditional mean of the reverse process. Within this line of works, Mittal et al. (022) consider diffusion of Gaussian processes. We are aware of other concurrent works that study diffusion process in Hilbert spaces (Lim et al. 2023 Pidstrigach ict al. 2023; Hagemann et al. 2023). However, differently from us, these works do not formally prove that the score matching optimization is proper evidence lower bound (ELBO), but simply propose it as an heuristic. None of these prior works discuss the limits of discretization, resulting in the failure of identifying which subset of functions can be reconstructed through sampling, Finally, the parametrization we present in our work merges how functions and score are approximated using a single, simple model.
The main goal of our work is to deepen our understanding of diffusion models in function space. We present new mathematical framework to lift diffusion models from minee-dimensionll inputs to function spaces, contributing to general method for data represented by continuous functions.
In S 2] we present Functional Diffusion Processs (FDPs), which generalize diffusion processes intinitediimensional functional spaces. We define forward I 2.T) and backward ( 2.2) FDPs, and consider generic functional perturbations, including noising and Laplacian blurring. Using an extension of Girsanov theorem, we derive in $ 2.3 an ELBO, which allows defining a parametric model to approximate the score of the functional density of FDPs. Given FDP and the associated ELBO, we are one-step closer to the definition of loss function to learn the parametric score. However, our formulation still resides in an abstract, infiiie-dimeenionall Hilbert space.
Then, for practical reasons, in we specify for which subclass of functions can perfectly reconstruct the original function given only its evaluation in countable set of points. This is an extension of the sampling theorem, which use move from the infiniteddimensionall domain of functions to a finite-dimensional domain of discrete mappings.
In S 4. discuss various options to implement such discrete mappings. In this work, we explore in particular the usage of implicit neural representations (INRs) (Sitzmann et al. (2020) and Transformers functions, and the score network, which is central to the training of FDPs, and is required to simulate the backward process. Our training procedure, discussed in 5 involves approximate, kmite-dimensione Stochastic Differential Equations (SDEs) for the forward and backward processes, as well as for the ELBO.
Vaswani et al. (2017) to jointly model both the sampled version of
We complement our theory with a series of experiments to illustrate the viability of FDPs, in 6 In our experiments, the score network is a simple multilayer perceptron (MLP), with several orders of magnitude fewer parameters than any existing score-based diffusion model. To the best of our knowledge, we are the first to show that a functional-spun diffusion model can generate realistic image data, beyond simple data-sets and toy models.
2 Functional Diffusion Processes (FDPs)
We begin by defining diffusion processes in Hilbert Spaces, which we call functional diffusion processes (FDPs). While the study of diffusion processes in Hilbert spaces is not new (Föllmer & Wakolbinger 1986 Millet et al. 1989 Da Prato & Zabezyk 2014), our objectives depart from prior work, and call for an appropriate treatment of the intricacies of FDPs, when used in the context of generative modeling. In 2. we introduce a generic class of diffusion processes Hilbert spaces. The key object is Equation together with its associated path measure Q and the time varying

---

measure Pt, where Po represents the starting (data) measure. In S 2.2 we derive the reverse FDP with the associated path-reversed measure Q and in $ 2.3 we use an extension of Girsanov theorem for infinitediimnssional SDEs to compute the ELBO, The ELBO is a training objective involving a generalization of the score function (Song al. 202
2.1 The forward diffusion process
We consider H to be real, separable Hilbert space with inner product (t,-), norm |I-Il and countable orthonormal basis 1=-1 Let L(H) be the set of bounded linear operators on H. B(H) be its Borel a allgebra, B&(FF) be the set of bounded meeasualll functions H - R. and P(H) be the set of probability measures on (H, B()). Consider the following H-valued SDE:
[xx, = (AX, + f(X,, t)) dt + dW, [x Po E P(H),
(1)
where t E [0, We is a R- Wiener process on H defined on the quadruplet (S2 F, (Fi) Q)
and J are the sample space and canonical filtration, respectively. The domain of f is D(f) B[ x H). where f D(f) C [0. x H + H is a measurable map. The operator A D(A) H - H is the infinitesimal generator of Co- semigroup exp(t.A) in H (t 2 0), and Po is a probability measure in H. We consider Q to be C' [[0,T]), that is the space of all continuous mappings [0, T| + H, and (w) w(t), w Z to be the canonical process. The requirements on the terms A. f that ensure existence of solutions to Equation (1) ddeped on the type of noise trace-class (Tr (Tr(R) [t] < oo) cylindrical (R π) used in the FDP (Da Prato & Zabezyk (2014). Hypothesis 7.1 / or Hypothesis 7.2 for trace-class and cylindrical noise, respectively).
The measure associated with Equation (1) is indicated with Q The law induced at time E 00, 2] by the canonical process on the measure Q is indicated with Pr E P(H), where p; (5) = Q(w E Ω (w) E 5))), and S is any element of F. Notice, that in infinite dimensional spaces there is not an equivalent of the Lebesgue measure to get densities from measures. In our case we consider however, when it exists, the single dimensional density (d) (f ( ), defined implicitly through dp (x*)///+) = p² (x'( (ellaiiidd'' being dx² the Lebesgue measure. To avoid cluttering the notation, in this work we simply shorten pf (1*) (a:*fi with pf (x) whenever unambiguous. In Appenddx B we provide additional details on the time-varying measure pe(dr)dt. Before proceeding, it is useful to notice that [qqatiinn (1) can also be expressed as an (infinite) system of stochastic differential
equations in terms of X (Xe: e*) as:
dx = () == I,
where we introduced the projection (X,, (AX.+J(X, (), et). Moreover, dW? (dW,, et) with covariance given by [B] = 8(k j)r min(s, t), 8 in Kroenecker sense, and n is the projection on the base of R.
2.2 The reverse diffusion process
We now derive the reverse time dynamics for FDPs of the form defined in Equation (1) We require that the time reversal of the canonical process, X, XT-,, is again a diffusion process, with distribution given by the path-reversed measure @(w), along with the reversed filtration F. Note that the time reversal of an infinite dimensional process is more involved than for the finite dimensional case (Anderson) 1982, Föllmer 1985). There are two major approaches to guarantee the existence of the reverse diffusion process. The first approach (Föllmer & Wakolbinger] 1986) is applicable only when R = 7 (the case of cylindrical Wiener processes) and it relies on a finite local entropy condition. The second approach, which is valid in the case of trace class noise Tr [2] < oo, is based on stochastic calculus of variations (Millet et al. 1989). The full technical analysis of the necessary assumptions for the two approaches is involved, and we postpone formal details to Appendixx A
Theorem 1. Consider Eqqation (1), R I, suppose Assumption 7 in Appendix A. holds; else, (R + 1) suppose Assumption 5 anndd Assumption 6 Appendix A.2 hold. Then Xt. corresponding to the path measure Q(w), has the following SDE representation:
3

---

dxx, (-AX; T(X,,T--) -t RD, log ( dt aw, No PT.
(3)
where W is Q PR-Wiener process, and the notation D2 log Pi that, when projected, satisfies (Da log (x)) e²)
(()) (x) stands for the mapping H - H
oz log Pi (d)
8
( 1
By projecting onto the eigenbasis, have infinite system of SDEs:
- - (X T - t) rk Oz log(p/
log pr (f)=)(())) dt dw ==.....,o0 (4)
The methodology proposed in this work requires to operate on proper Wiener processes, with Tr(R) oc, which implies, intuitively, that the considered noise has finite variance. We now discuss a Corollary, in which Assumption is replaced by stricter conditions, that we use to check the validity of the practical implementation of FDPS.
Corollary 1. Suppose Assumption 6frrmm Appendix A.2 holds. Assume that i) Tr[RR = =E,'' co. ti) 6' (z, t) b'x', Vi, i.e. the drift term is linear and only depends x through its projection onto the corresponding basis and iii) the drift is bounded, such that 3K > 0: K < b < 0, Vi. Then, the reverse process evolves according to Equation (4)
Theorem stipulates that, given some conditions, the reverse time dynamics for FDPs of the form defined in Equation (1) exist. Our analysis provides theoretical grounding to the observations in concurrent work (Lim et al., 2023) where, empirically, it is observed that the cylindrical class of noise is not suitable. We argue that, when R I, the difficulty stems from designing the coefficients of the SDEs such that the forward (see requirement (5.3) in Da Prato & Zabczyk (2014)) as well as the backward processes Assumption T exxsst. The work by Bond-Taylor & Willcocks (2023) uses cylindrical (white) noise, but we are not aware of any theoretical justification, since the model architecture only partially suited for the functional domain.
As an addendum, we note that the advantages of projecting the forward and backward processes on the eignenbasis of the Hilbert space H, in Equation (2) and Equation (4) become evident when discussing about the implementation of FDPs, specifically when we derive practical expressions for training and the simulation of the backward process, discussed in 5 and in a fully expanded toy example in Appendix D
2.3 A Girsanov formula for the ELBO
Direct simulation of the backward FDP described by Equation (3) is not possible. Indeed, we have no access to the true score of the density p: induced time T E [0, T7. To solve the problem, we introduce a parametric score function И x [0, x 1n - H. We consider the dynamics:
ddx.= = (-AA- f(x,, T - t) + Rsg ( T - +)) dt + dW,, 120~x0 XT E P(H).
(5)
with path measure fXx and dWV, being jixr R Wiener process. To emphasize the connection between Equation (3) and Equation (5) we define initial conditions with the subscript T. instead of O.
In principle, we should have Xr = Pr. as it will evident from the BL in |Equation (8) However, Pr has a simple and eas-le--sample- from-ffrmmmoo distribution only for T - 0o, which is not compatible with a realistic implementation. The analysis of the discrepancy when is finite outside of the scope of this work, and the interested reader can refer to Franzese et al. (2022) for an analysis on standard diffusion models. The final measure of the new process at time is indicated by Xo. i.e. xo(S)= = pxx (fw E Ω Xr(w) E S}).
Next, we quantify the discrepancy between Xo and the true data measure Po through an ELBO. Thanks to an extension of Girsanov theorem infinite dimensional SDES (Da Prato Zabezyk 2014), is possible relate the path measures (Q and pxr, respectively) of the process X, induced by different drift terms in Equation (3) and different initial conditions.

---

Starting from the score function se, we define:
7e (x, R s0 T 1) D2 log PT pr)(())
(d)
(6)
Under loose regularity assumptions (see Condition 2 in Appendix A.4) W. W, ro (X ds is
Per R- Wiener process (Theorem 10. 14 in Da Prato & Zabezyk (014)), where Girsanov Theorem
also tells us that the measure per satisfies the Radon-Nikodym derivative:
7. 170 /
dier dQ
t).d.)) RŽH /
| dt
= exp
(7)
n
By virtue of the disintegration theorem, dQ = ddpd and similarly deer = dPpdpr, being Qo, Po the measures of the processes when considering particular initial value. Then, pxr satisties
dpxr = dl -dp dxz for any measure XT P(H). Consequently, the canonical process X, has an SDE dpr
representation according to Equation (5) under the new path measure fexr. Then (see Appendix A.5 for the derivation) we obtain the ELBO:
BG / (xx 11 H dt + KL [L ll XT).
KL Po 11
xol S
(8)
Provided that the required assumptions in Theorem I are met, the validity of (8)is general. Our goal, however, is to set the stage for practical implementation of FDPs, which calls for design choices that easily enable satisfying the required assumptions for the theory to hold, Then, for the remainder of the paper, we consider the particular the dynamics as follows:
where f= 0 in Equation (1) This simplifies
dX,, = AX,dt + ddW. Xo ~ Pa E P(H)
(9)
dx,, (-AX; + RsØ (8, T -t) dt + aw,, X X
(10)
Since the only drift component in Equation (9) is the linear term A, the projection b' will be linear as well. Such a design choice, although not necessary from a theoretical point of view, carries several advantages. The design of a drift term satisfying the conditions of Corollary Ieecmee straightforward, where such conditions naturally aligns with the requirements of the existence of the forward process (Chapter 5 of Da Prato & Zabczyk 2014)). Moreover, the forward process conditioned on given initial conditions admits known solutions, which means that simulation of SDE paths is cheap and straightforward, without the need for performing full numerical integration. Finally, it is possible to claim existence of the true score function and even provide its analytic expression (full derivation in in/Appendix A.7 as:
D2 log Pi (d) (x) = S(() -1 (x ex (Xol X, x)),
(11)
where S(1) = ( exp((t Rexp((( (()) This last aspect is particularly useful
= when considering the conditional version of Equation (6) through (Dæ log ((0) (x mo))
0 Dzh log (150)² (xr | (eitH,)))). as:
to (or 1) R
- (R<(el (so(x, -1 1) D2 log (1)((1) (x| |200)
(12)
where, similarly to the unconditional case, we have D, log P. (1) (or (m)) = -S(() exp((A)ao)
Then, Equation (12) can be used to rewrite Equation (8)
/ 11 (X,, 1) 2
I [Iio(X., Xo, t)] dt + 1,
Ee
(13)
where 7 is quantity independent of 0. Knowledge of the conditional true score D,, log (d) (ar am) and cheap simulation of the forward dynamics, allows for easier numerical optimization than the more general case of f * O.
5

---

3
Sampling theorem for FDPs
The theory of FDPs developed far is valid for real, separable Hilbert spaces. Our goal now is to specify for which subclass of functions it is possible to perfectly reconstruct the original function given only its evaluation in countable set of points. We present a generalization of the sampling theorem (Shannon] 1949), which allows us to move from generic Hilbert spaces to domain which amenable a practical implementation of FDPs, and their application to common functional representation of data such as images, data on manifolds, and more. We model these functions as objects belonging to the set of square integrable functions over C homogeneous manifolds M (such as R.N. S.V etc...), i.e., the Hilbert space H (.2 Then, exact reconstruction implies that all the relevant information about the considered functions is contained in the set of sampled points.
First, we define functions that are band-limited:
Definition 1. A function æ in H = (MM a spectral entire function of exponential type v (SE-v) if A < va 1z), k E N. Informally, the "Fourier Transform" of is contained in the interval [0, v)
Pesenson 2000
Second, we define grids that cover the manifold with balls, without too much overlap. Those grids will be used to collect the function samples. Their formal definition is as follows:
Definition 2. (r, X) denotes the set of all sets of points Z {p=} such that: i) distspiiss pt) > O and ii) balls (p,, >) form a cover of M with multiplicity < r.
Combining the two definitions, we can state the key result of this Section. As long as the sampled function is lindedimited, if the samples grid is sufficiently fine, exact reconstruction is possible:
Theorem 2. For any set Z E (r, A), any SE-v function x is uniquely determined by its set of values in Z (i.e (x(pp(}) as long as À < d, that is
X xlp.lmp.
(14)
P:EZ
where mpp -M- + H are known polynomial Q and the notation xpp) indicates that the function x is evaluated at point p.
A precise definition of the value of the constant dl and its interpretation is outside the goal of this work, and we refer the interested reader to Pesenson (2000) for additional details. For our purposes, is sufficient to interpret the condition in Theorem 2 as generalization of the classical Shannon-Nyquis sampling theorem (Shannon 1949). Under this light. Theorrmm 2 has practical relevance, because gives the conditions for which the sampled version of functions contains all the information of the original functions. Indeed, given the set of points p: on which function x is evaluated, is possible to reconstruct exactly æ(p) for arbitrary p.
The uncertainty principle. It is not always possible to define Hilbert spaces of square integrable functions that are simultaneously homogeneous and separable, for all the manifolds M of interest. In other words, is difficult practice to satisfy both the requirements for FDPs exist, and for the sampling theorem to be valid (see an example in (Appendix C). Nevertheless, it is possible to quantify the reconstruction error, and realize that practical applications of FDPs are viable. Indeed, given compactly supported function I, and set of points Z with finite cardinality, we can upper- apper-bound the
reconstruction error
Epezz 2lp.[mp.
x
with:
H
(zc - x [p:)) mp
+
Σ PLEZ
x" [pilmpp aalmp x"
+
+||0" xxln
(15)
P.EZ
where x² is the SE-v bandlimited version of x. obtained by filtering out in the frequency domain any component larger than v. The error 61 is due to x + x². The term cg is the reconstruction error
Preccisely they are the limits of spline polynomials that form Riesz basis for the Hilbert space of polyharmonic functions with singularities in 2 (Pesenson 2000).

---

due to finiteness of 1Z; : the sampling theorem applies to z", but the corresponding sampling grid has infinite cardinality. Finally, the term <3 quantifies the energy omitted by filtering out the frequency components of x" larger than v. This (loose) upper bound allows us to understand quantitatively the degree to which the sampling theorem does not apply for the cases of interest. Although deriving tighter bounds is possible, this is outside the scope of this work. What suffices is that in many practical cases, when functions are obtained from natural sources, it has been observed that functions are nearly time and bandwidth limited (Slepian 1983). Consequently, as long as the sampling grid is sufficiently fine, the reconstruction error e is negligible.
We now hold all the ingredients to formulate generative functional diffusion models using the Hilbert space formalism and implement them using a finite grid of points, which what we do next.
4 Score Network Architectural Implementations
We are now equipped with the ELBO (Equation (8) and a score function sg that implements the T|
mapping H x [o, x Rm - H. We could then train the score by optimizing the ELBO and produce samples arbitrary close to the true data measure Pa. However, since the domain of the score function is the infiiie-dimensionall Hilbert space, such mapping cannot be implemented in practice. Indeed, having access to samples of functions on finite grid of points is, in general, not sufficient. However, when the conditions for Theorem 2 hold,
can substitute with no information loss x: E H with its collection of samples (a(pt). P This allows considering score network architectures that receive as input a collection of points, and not abstract functions. Such architectures should be flexible enough to work with an arbitrary number of input samples at arbitrary grid points, and produce as outputs functions in H.
4.1 Implicit Neural Representation
The first approach we consider in this work based on the idea of Implicit Neural Representations (INRs) (Sitzmann et al. 2020). These architectures can receive as inputs functions sampled at arbitrary, possibly irregular points, and produce output functions evaluated at any desired point. Unfortunately, the encoding of the inputs is not as straightforward as in the Neural Fourier Operator (NFO) case, and some form of autoencoding is necessary. Note, however, that in traditional score- diffusson models (Song al.) 2021), the parametric score function can be thought of as denoising autoencoder. This is a valid interpretation also in our case, as it is evident by observing the term E X₂ X, = ø] of the true score function in [Equation (I1) Since INRs are powerful denoisers (Kim et al.) (022a)), combined with their simple design and small number of parameters, in this Section we discuss how to implement the score network of FDPS using INRs.
We define a valid INR as a parametric family (1b, 1, 0) of functions in H, i.e., mappings T x [0, 7) x RT + H. A valid INR is the central building block for the implementation of the parametric score function, and relies on two sets of parameters: 0, which are the parameters of the score function that we optimize according to Equation (8) and th which serve the purpose of building mapping from H into a finite dimensional space. More formally:
Definition 3. Given manifold M, valid Implicit Neural Representation (INR) is an element of H defined by a family of parametric mappings n(zb, t, 0,), with t E [0, 0. sp E Rm That is, for p E M, we have ne(bb, t, [p) E R. Moreover, we require n f, 0) E L2(M).
A valid INR defined in Definition 3
is not sufficiently flexible to implement the parametric score function sg, as it cannot accept input elements from the
Hilbert space H: indeed, the score function is defined as mapping over H [0, T) m H. whereas the valid INR is a mapping defined over RT x [0, T x Rm + H. Then, we use the second set of parameters 2/ to condensate all the information of generic E H into knite-dimensionn vector. When the conditions for Theorem hohll,, we can substitute - with no information loss - E H with its collection of samples (sr(px), Pi... Then, we can construct an implicitly defined mapping H x [0, T) x Rmm - IRm as:
g ((<=lp..
arg min +
Σ(n(si t, 0) - =(pp.)²
(16)
Pt:
In this work, we consider the modulation approach to INRs. The set of parameters &b are obtained by minimizing Equation (16) using few steps of gradient descent on the objective
7

---

Ep. (n((b, t. 0).pp =fpp.)² starting from the zero initialization of Φb This approach, also ex- plored by Dupont et al. (2022b), is based on the concept of meta-learning (Finn et al. (017). In summary, our method constructs mappings H [0, 1" 1 + H. where the same INR is used first to encode x into 1 and subsequently to output the value functions for any desired input point p. thus
implementing the following score network:
so(2, 1) - (S(()) (s exp(AA)lo((eb))))))..)0
(17)
4.2 Transformers
As an alternative approach, we consider implementing the score function 5e using transformer architectures Vaswani et (2017), by interpreting them mappings between Hilbert spaces (Cao) 202I). We briefly summarize here such a perspective, focusing on a single attention layer for simplicity, and adapt the notation used throughout the paper accordingly.
Consider the space (2)) with the usual collection of samples {x(pi]. pr). As a first step, both the "features" {ar[pi]} and positions {p:} are embedded into some higher dimensional space and summed together, to obtain a sequence of vectors {yi}. Then, three different (learnable) matrices ((K), BCV) are used to construct the linear transformations of the vector sequence {yi} as
ÿr(Q) = [U: (Q) yi (K( = { (K) = g(k)y, y;), y = (1/ (V) = y;). Finally, the three matrices (Q. are multiplied together, according to any variant of the attention mechanism. Indeed, different choices for the order of multiplication and normalization schemes in the products and in the matrices correspond to different attention layers Vaswani et al. (0117). In practical implementations, these operations can repeated multiple times (multiple attention layers) and can be done in parallel according to multiple projection matrices (multiple heads).
The perspective explored in (Cao, 2021) is that it is possible to interpret the sequences yi (Q.KV) as learnable basis functions in some underlying latent Hilbert space, evaluated the set of coordinates {pt). Furthermore, depending the type of attention mechanism selected, the operation can be interpreted as a different mapping between Hilbert spaces, such as Fredholm equations of the second-kind or Petrov-Galerkin-typp projections (Cao (021) Eqs. 9 and 14).
While complete treatment of such an interpretation is outside the scope of this work, what suffices is that it is possible to claim that transformer architectures are a viable candidate for the implementation of the desired mapping H x [0, x 1 + H, a possibility that we explore experimentally in this work. It is worth noticing that, compared to the approach based on INRs, resolution invariance is only learned, and not guaranteed, and that the number of parameters generally higher compared to an INR. Nevertheless, learning the parameters of transformer architectures does not require meta-learning, which a practical pain-point of INRs used in our context. Additional details for the transformer-bast based implementation of the score network are available in Appendix F.
Finally, for completeness, it is worth mentioning that related class of architectures, the Neural Operators and NFOs (Kovachki et al. |021 Li et al. 2020), are also valid alternatives. However, such architectures require the input grid to be regularly spaced (Li et al. 2020), and their output function is available only at the same points of the input, which would reduce the flexibility of FDPs.
5 Training and sampling of FDPs
Given the parametric score function ag from Equation (17) by simulating the reverse FDP, we generate samples whose statistical measure Xo is close in KL sense to pn- Next, we explain how to numerically compute of the quantities in Equation (13) which is part the ELBO Equation (8)
and how to generate new samples from the trained FDP (simulation of Equatton (10).
ELBO Computation. Equation (8) involves Equation (13) which requires the computation of the Hilbert space norm. The grid of points r(p.] is interpolated in H as Σ, x pp]k'. Then, the norm of interest can be computed as:
Σ xlpsse ct ))
-R
Σ slp.lss, R Σ alpp)")) = ((+))-)
RŽH
(18)
8

---

Depending on the choice of ć², e', the sum w.r.t the index & is either naturally truncated or it needs to be further approximated by selecting cutoff index value. Finally, training can then be performed by minimizing:
/
Es
(1 dt R2
= B.R.
Σ
(
Σ
ro E Xe(mils', () 5¹
dt
(19)
Numerical integration. Simulation of infinite dimensional SDEs is a well studied domain (Debuss
che 201T), including finite difference schemes (Gyöngy 1998 1999 Yoo 2000), finite element methods and/or Galerkin schemes (Hausenblas) 2003a bj Shardlow 1999). In this work, we adopt
a finite element approximate scheme, and introduce the interpolation operator, from RI to H, i.e. E; spp (Hausenblas) 20036). Notice that, in general, the functions ç differ from the basis et. In addition, the projection operator maps functions from H into R2,, as (xr, ()), Ç H. Usually, L= 1.). When Ç ç* the scheme is referred to as the Galerkin scheme. We consider instead a point matching scheme (Hausenblas) 2003b), in which Ç 8lp p.] with in Dirac sense, and consequently (²) x[pt]. Then. the infinite dimensional SDE of the forward process from Equation (9) approximated by the finite ([Z)) dimensional SDE:
dx.[px) = [ (t) 1)
di (p.,, k 1==,...,12 .
(20)
Similarly, the reverse process described by Equation (10) corresponds the following SDE:
(- (-(11 A Σ Xeluvle' (*)
(+)) Rso Σ & -t).c* )) dt + dW. (21)
[p.] =
k- = /
Equation (21) is a finite dimensional SDE, and consequently we can use any known numerical integrator to simulate its paths. In Appenddix D we provide a complete toy example to illustrate our approach in a simple scenario, where we emphasize practical choices.
Experiments
Despite a rather involved theoretical treatment, the implementation of FDPs is simple. We imple- mented our approach JAX (Bradbury et al. 2018), and use WANDB (Biewald 2020) for our experimental protocol. Additional details on implementation, and experimental setup, well as more experiments are available in Appenddx E
We evaluate our approach on image data, using the CELEBA 64 x 64 (Liu et al. 2015) dataset, Our comparative analysis with the state-of-the-ar art includes generative quality, using the FID score (Heusel et al.) 2017), and parameter count for the score network. We also discuss (informally) the complexity of the network architecture, as measure of the engineering effort in exploring the design space of the score network. We compare against vanilla Score Based Diffusion (SBD) (Song al. 2021), From Data To Functa (FD2F) (Dupont et al. 2022a) which diffuses latent variables obtained from an INR, Infinite Diffusion (DO-DIFF) (Bond- Taylor & Willcocks 2023), which is recent approach that only partially suited for the functional domain, as it relies on the combination of Fourier Neural Operators and classical convolutional U-NET backbone. Our FDP method is implemented using either MLP or Transformers. In the first case, we consider a score network implemented as simple MLP with 15 layers and 256 neurons in each layer. The activation function is Gabor wavelet activation function (Saragadam et al. 2023). In the latter case, our approach is built upon the UVIT backbone detailed by Bao al. (2022). The architecture comprises 7 layers, with each layer composed of self- atfenttention mechanism with attention heads and a feedforward layer.
We present quantitative results in Table showing that our method FDP(MLP) achieves an impres- sively low FID score, given the extremely low parameter count, and the simplicity of the architecture. FD2F obtains a worse (larger) FID score, while having many more parameters, due the complex

---

parametrization of their score network. As a reference we report the results of SBD, where the price to be pay to achieve an extremely low FID is to have many more parameters and a much more intricate architecture. Finally, the very recent OD-DIFF method, has low FID-CCIIP score (Kynkäänniemi et al. 2022). but requires a very complex architecture and more than orders of magnitude more parameters than our approach. Showcasing the flexibility of the proposed methodology, we consider more complex architecture based on Vision Transformers (FDP(UVIT)) These corresponding results indicate improvements in terms of image quality (FID score= 1) and do not require meta-learning steps, but require more parameters (O(20M)) than the INR variant. To the best of our knowledge, none of related work in the purely functional domain (Lim al. 2023 Hagemann et al. 2023 Dutordoir et al. 2022 Kerrigan et al. 2022) provides results going beyond simple data-sets. Finally, present some qualitative results in Figures and cleearly showing that the proposed methodology
is capable of producing diverse and detailed images.
Methods
FID (L)
FID-CLIP (L)
Params
FDP(MLP)
35.00
12.44
O(1 M)
FDP(UViT)
11.00
6.55
O(20 M)
FD2F
40.40
OC10 M)
SBD
3.30
OC100 M)
OO-DIFF
4.57
OC100 M)
Table 1: Quantitative results, CELEBA data-set. (FID-CLIP
Kynkäänniemi et al.
2022
Figure 1: Qualitative results with MLP.
Figure 2: Qualitative results with UVTT.
7 Conclusion, Limitations and Broader Impact
We presented theoretical framework to define functional diffusion processes for generative modeling. FDPs generalize traditional score-based diffusion models to infiniteddimensional function spaces, and in this context we were the first to provide full characterization of forward and backward dynamics, together with formal derivation of an ELBO that allowed the estimation the parametric score function driving the reverse dynamics.
To use FDPs in practice, we carefully studied for which subset of functions it was possible to operate on a countable set of samples without losing information. We then proceeded to introduce series of methods to jointly model using only simple INR or a Transformer an approximate functional representation of data on discrete grids, and an approximate score function. Additionally, we detailed practical training procedures of FDPs, and integration schemes to generate new samples.
The implementation of FDPs for generative modeling was simple. We validated the viability of FDPs through a series of experiments on real images, where show, while only using simple MLP for learning the score network, extremely promising results in terms of generation quality.
Like other works in the literature, the proposed method can have both positive (c.g., synthesizing new data automatically) and negative (c.g., deep fakes) impacts on society depending on the application.
Acknowledgments
GF gratefully acknowledges support from Huawei Paris and the European Commission (ADROrT6G Grant agreement ID: 101095363), MF gratefully acknowledges support from the AXA Research Fund and the Agence Nationale de Recherche (grant ANR- LAN--18-EELN 18-CE46-0002 and ANR-19-P31 11R119-31AA-002,,
10

---

References
Brian D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-33 32, 1982.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint 2rXi::2208.0939,, 2022.
Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: vit backbone for score-based diffusion models. arXiv preprint 22Xv:2209.121522 2022.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https:: http::://wwwwaaatt com/ Software available from wandb.com.
Vladimir Bogachev, Giuseppe Da Prato, and Michael Röckner, Existence and uniqueness of solutions for fokker-planck equations on hilbert spaces. Evol. Equ., 10, 07 2009.
Vladimir Bogachev, Giuseppe Da Prato, and Michael Röckner. Uniqueness for solutions of fokker- planck equations on infinite dimensional spaces. Communications Partial Differential Equations. 36(6);925-939, 2011.
Sam Bond-Taylor and Chris G Willcocks. co-diff: Infinite resolution diffusion with subsampled mollified states. arXiv preprint 230..:22...,,2.,,, 2023.
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miine et al. Jax: compos- able transformations of python+ numpy programs. 5:14-24, 2018.
Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 34:24924-24940, 2021.
Giuseppe Da Prato and Jerzy Zabezyk. Stochastic equations in infinite dimensions. Cambridge university press, 2014.
Arnaud Debussche. Weak approximation of stochastic partial differential equations: the nonlinear case. Mathematics of Computation, 101273):99-117,200
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794. Curran Associates, Inc., 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In International Conference on Learning Representations, 2022.
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data functa: Your data point is function and you treat it like one. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5694-5725. PMLR, 17-23 Jul 2022a.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Teh, and Arnaud Doucet. COIN+ Neural compression across modalities. Transactions of Machine Learning Research, 2022b.
Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes. arXiv preprint arXiv:2206.03399 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In International conference on machine learning, pp. 1116-111 1135. PMLR, 2017.
Hans Föllmer. An entropy approach to the time reversal of diffusion processes. Stochastic Differential Systems Filtering and Control, pp. 156-163. Springer, 1985.

---

Hans Föllmer. Time reversal on wiener space. In Stochastie processes mathematesss and physics, pp. 119-129. Springer. 1986.
Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filippone, and Pietro Michiardi. How much is enough? study diffusion times in score-based generative models. arXiv preprint arXiv:2206.05/73 2022.
H. Föllmer and A. Wakolbinger. Time reversal of infinite-dimensionall diffusions. Stochastic Processes and their Applications, 22(1):59-77, 1986. ISSN 0304-4149. doi: https://oi..or//I
101600300-41149886990111 14-6. httphhttt//wwwwt wwwp:w/wwwwwwnnre.ttccoco cow/coo////o///o http///ttt..ccocccccc////rrtttat p:1/0300414986900146
István Gyöngy. Lattice approximations for stochastic quasi-linear parabolic partial differential equations driven by space-time white noise i. Potential Analysis, 9(1):1-25, 1998.
István Gyöngy. Lattice approximations for stochastic quasi-linear parabolic partial differential equations driven by space-time white noise ii Potential Analysis, 11(1):1-37, 1999.
Paul Hagemann, Sophie Mildenberger, Lars Ruthotto, Gabriele Steidl, and Nicole Tianjiao Yang. Multilevel diffusion: Infinite dimensional score-based diffusion models for image generation, 2023.
Erika Hausenblas. Approximation for semilinear stochastic evolution equations. Potential Analysis, 18(2):141-186. 2003a.
Erika Hausenblas. Weak approximation for semilinear stochastic evolution equations. In Stochastic analysis and related topics VIII, pp. 111-128. Springer, 2003b.
Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXh::2211.132211 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 3 Is International Conference Neural Information Processing Systems, NIPS 17, pp. 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020.
Jonathan Ho, Chitwan Saharia, William Chan, J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation, 2021.
org/
abs/2106. 15282
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022. URL /ttpp:://axii..ooto wrwp.vioo///b///1111 2bs/ 02303
Emiel Hoogeboom, Víctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3D. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 8867-8887 PMLR, 17-23 Jul 2022.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models in infinite dimensions, 2022. htttt::ts/
org/ abs 2212. 00886
Chaewon Kim, Jaeho Lee, and Jinwoo Shin. Zero-shot blind image denoising via implicit neural representations. arXiv preprint arXi::2204.024005 2022a.
12

---

Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11001-11228, PMLR, 17-23 Jul 2022b.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In A. Beygelzimer, Y. Dauphin, Liang. and J. Wortman Vaughan (eds.), Advances in Neural Infor- mation P Processing Systems, 2021. httpp//s..ppppppeeeeeeeee/////er
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations. 2021.
Nikola Kovachki, Zongyi Li. Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2 084881 2021.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. arXiv preprint arXiv:2203.06026 2022.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. 2, 2010.
Available: htlp://yann. http://ww.ecco..c//////////////a wtt.c/o/o////.s.......mm
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint ar (t.ivv:01000088000 2020.
Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, al. score-based diffusion models in function space. arXiv preprint arXiv:2302.07400, 2023.
Jinglin Liu, Chengxi Li, Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. Proceedings of the AAAI Conference on Artificial Intelligence, 36( 10000 1028, Jun. 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17Xiv:1711.05101 2017.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2837-2845, June 2021.
Annie Millet, David Nualart, and Marta Sanz. Time reversal for infinite-dimennionall diffusions. Probability theory and related fields, 82(3)311-347, 1989.
Sarthak Mittal, Guillaume Lajoie, Stefan Bauer, and Arash Mehrjou. From points to functions: dimenii-dd representations in diffusion models. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
Isaac Pesenson. A sampling theorem on homogeneous manifolds. Transactions of the American Mathematical Society, 352 2000.
Angus Phillips, Thomas Seror, Michael Hutchinson, Valentin De Bortoli, Arnaud Doucet, and Emile Mathieu. Spectral diffusion processes. arXiv preprint arXiv:2209.144 2022.
Jakiw Pidstrigach, Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinit-diimensional diffusion models for function spaces. arXiv preprint arXiv: 2013.1 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URI. https::tt////... /tttpi:///rxv.... wereiiioorrgeee wtrg/bbs/2204 06125
13

---

Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipa- tion. arXiv preprint arXiv:2206.1333 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo-Lopes Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Vishwanath Saragadam, Daniel LeJeune. Jasper Tan, Guha Balakrishnan, Ashok Veeraragha- van, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. arXiv preprint 20X1::2301.05187, 2023.
C.E. Shannon. Communication the presence of noise. Proceedings of the IRE, 37(1):10-21, 1949.
Tony Shardlow. Numerical methods for stochastic parabolic pdes. Numerical functional functiioa analysis and optimization, 20(1-2):121-144 1999.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7462-7473. Curran Associates, Inc., 2020.
David Slepian. Some comments on fourier analysis, uncertainty and modeling. SIAM review, 25(3): 379-393, 1983.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256-2265. Lille, France, 07-09 Jul 2015. PMLR.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, Larochelle, Beygelzimer, d' Alché-Bue, F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12438-12444 12448. Curran Associates, Inc., 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021.
Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem, 2022. URL hhttts://aaritii http://wwwvvvo...aaaa. /ttaa///200..... 04119
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Paseal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- tion, 1361):16,111774 2011.
14

---

Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. International Conference on Learning Representations (ICLR), 2022.
Hyek Yoo. Semi-discretization of stochastic partial differential equations on r by a finite-difference method. Mathematics of computation, 69(230):653- 66666666666666666 2000.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806, 2020.
Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M. Susskind, and Miguel Ángel Bautista. Diffusion probabilistic fields. In The Eleventh Internationnl Conference on Learning Representations, 2023. URL https: http:s///pparreeess..e/ffooooooo
2GN
15

---

Supplementary Material: Continuous-Time Functional Diffusion Processes
A Reverse Functional Diffusion Processes
In this Section, we review the mathematical details to obtain the backward FDP discussed in Theo- rem [] Depending the considered class of noise, different approaches are needed. First, we present in Appendix A. the conditions to ensure existence of the backward process, which we use if the C operator is an identity matrix, C I. Then we move different approach in Appendix A.2 for the case C + I.
A.1 Follmer Formulation
The work in Föllmer 1986) is based on finite entropy condition, which report here Condition One simple way to ensure that the condition is satisfied is to assume:
Condition 1. For given k. define (())) be the path measure corresponding to the (infinite) system
[xx; (X,, ()dt + +d;, dx;; dw?, ==
i k
(22)
We say that Q satisfies the finite local entropy condition if KL [Q Q(4) < vk,
Define F.O) o X3 [X, 5
sttj #
Assumption 1.
Σ Lo (B E (xxx0))))(( dtj < oc, Q(1ass. (23)
Notice that Assumptiio is true, then Condition holld (Föllmer (186)) Thm. 2.23)
Theorem 3. If KL [Q [(x)) < co, then KL
Q Q(1))
< co.
Proof. The proof can be obtained by adapting the result of Lemma 3.6 of Föllmer Wakolbinger (986)..
This Theorem states that if the forward FDP path measure satisfies the finite local entropy condition, then also the reverse FDP path measure Q satisfies the finite local entropy condition. Theorem 4. Let Q be finite entropy measure. Then:
dx 1y* (X,,, t)dt + dwt, under Q dxt = 8 (Xt, (Xt, t) + dw,,
(24)
under
where:
log
Pt (x* 1 I j + kt)
(z (z, T + (2,t)
(25)
drk
Proof. For the proof, we refer Theorem 3.14 Föllmer & Wakolbinger (1866).
A.2 Millet Formulation
Let (R) = fa E H: (at)) < oc). For simplicity, we overload the notation of the letter K, and use it for generic constants, that might be different case by case basis. Assumption 2.
5K1+ Vo E I2(( supt H (1 01) + SKI+ Er
Vz, y E (2(()),
Σ -v,, < > r'a' - 1/)2
<
16

---

This assumption is simply the translation of HI from Millet et al. (1989) to our notation. Assumption 3. There exists an increasing sequence of finite subsets J(n), n E N. Un)((n) N such that Vn E N, > there exists a constant K ((M, n) such that the following holds:
)
sup
sup iEJ(n))
sup (x,
sup jiJ(m)
|
S M
+ EErr
= kM,
3
Again, this assumption is simply the translation of H5 from Millet et al. 1989) to our notation. Assumption 4. Either i):
Vx, y E (R), ()
K - y/)
or it): Vi, (x) is function of x for most M coordinates and Vx, y E (2) sup
(rr)) -3 22)
This corresponds satisfying either H3 or jointly H2 and H4 of Millet et al. (1989). For simplicity. we can combine together the different assumptions into
Assumption 5. Let Assumption 2. Assumption 3 and Assumption 4 hold.
Finally, we state required assumptions about the density: Assumption 6. Suppose that the initial condition is Xo E (R).
Assume that the conditional law of given xt . # i has density Pi (d)) (x* |x jj # i) w.r.t Lebesgue measure on R.
* Assume thut Jto S., Ir* d (d) du (p: (af |x & i)) Pt (dw.t)ddddddd )dl < for fixed subset
J N,to and D, [[]TEJ K2) x (I] RR. K5 compact in R. I,2 (R).
We reported in our notation the content of Theorem 4.3 of Millet et al. 1989). ). This can be used to prove the existence of the backward process.
A.3 Proof of Theorem
If R I, then assume Assumption I Consequently, Q is finite entropy measure. Then Theorem 4 holds, from which the desired result. If, instead R of 7, then we require Assumption 5|As sumption 6 Application of Thm 4.3 of Millet et al. (1989) allows to prove the validity of Theorem also in this case.
A.3.1 Proof of Corollary
Assumption 5 is required directly. We need to show that with the considered restrictions Assumption valid.
Since E,r" rf < 0o, then E = Ka < co. Moreover, (bi ,))2 < Kz Then,
dx E (R)... the following holds sup, (b) Err) < (r + K, S
max max(Ka, K2) (1+ (x))))) Similarly, Vx, y E L²(R) we have sup, (b,
(y, 1) < E K3(2' )² Thus Assumption 2 is satisfied.
Since b' (x. bounded and independent on Assumption i satisfied, explicitly discussed in Millet et al. 1989).
Finally, since ('(x) is a function of for M coordinate, and sup, sup, ((r)))(())))))) (y, 2) < 22 )² Assumption is satisfied.
Then, combined toghether Assumption 5 holds.

---

A.4
Girsanov Regularity
Condition 2. Assume that (2 t) is an F measurable process and that either:
2.
I
Ihe (Xe, dt ]
7 t)
exp
2
Rt
dt
= Eq
exp
2
<0, (26)
or
exp 20 8) n+ dt )
36 > O: B
00.
(27)
This is equivalent the regularity condition in eq. 10.23 Da Prato & Zabezyk 2014) or Proposition 10.17 in Da Prato & Zabczyk (204)).
A.5
Proof of KL divergence expression
We leverage Equation (7) to express the Kullback-Leibler divergence as:
doo
dpr
dQo
KL.
|| [pxr] = Re
log
log
= Eag
log
+ KL. ppr |1 XT] =
dP
dxr
dP.
1 T
/ [1o t)
Eg
(ro (To(XX,,)) ();
Rt + 2
dt
+ KL. (pr 11 Xx| =
1o Xx,,, /)
EQ / ()) dt
E
dt
KL
[pr xrl
+ KL For || xx)-
Moreover, since
dQr dFy
dpp dxo
KL
+ || ppr
= B2
log
log
2 KL [Po Xol
we can combine the two results and obtain Equation (8)
A.6
Conditional score matching
In this subsection we prove the equality in Equation (13):
j [IooXx
/
Es
t)1 R dt
110 (2)(, dtdpp(x)=
0
/
Da log (x) s0(x, t) R dtdp((x)
JH
De log (1) (x) so(x, ntH
dtdp (2,
HHH
T'
[
Dz log (x) Dr log pt (4) (x xo) + D log pt (4) mv) - so(x,t)
dtdpi (x, xo)
|HXH
D2 log log (x) D₂ log pi (rr xx0) 2 nh + ||| Dr log (x xw)) - sx
HXH
2 Dx log pt
(x) (x)
Da log (2) (Ir xo), Da log pr
(d) (x xo) se(z, deddp(2z,
18

---

To simplify the equality, we need to notice that:
(x (x'la)++)dd' = dpe( (1 =f (x = / dpe(x', xo0l2971)
(zo (zo (z) dx' / Pe a1 (x' xf (dtttddpppolllll (
Then, computing
S. dæ log ol ( x0) = Sf
der pla) (It looti xn)
d pli) (x (oft 200 dpc (z)
f plu (x |iti (2 mD)
dp: | zo ddp rlti) = I
2o do (x 2o) dp (Tv. ,87)
(x0 vo)
dx (1 (at) xo so)ddi ) de (x(
d
to dæy (x xo dor ( (x) (w)
dp dx Pi (1) (1 dd (x)
d
d log (4)
d dlog (x (o0ttt) dpe(x)) (1) (ar ( spt (xf#i)-
dr Pi
dx
Consequently:
SH*H DD Do (1) (x) D2 log (4) (x [2)) so(x,t) t)) dpe(x, =
Combining together and rearranging the terms, get the desired Equation (13)
A.7 Explicit expression of score function
As mentioned in the text, we consider the case f 0. this case, there exists a weak solution to Equation (1) as:
X, exp(tA)Xo + exp((( A)d)..
(28)
0
Consequently, the true score function has expression:
19

---

d dx.
log (1) ( (zf1x0 = de d:
a
( (a01)o)tt) der Smo Pi pla) (xx) (x') 20. wit )dpe (20 (2sti)
Pi (d) (z))))
Pi (x0)))it1
(d)
- -sz (0+) (at exp(((0) (1b) )rr) pi (/d) (x* (ro, witi )d (
Pi (d) (2sttit
-(at)- (xtp(") (x* (z=)) Szp exp (tb' ) (x' (zo, wi + (200 ((o)zt
03 (1
-(st)-1 pptpp ((dd)((( (xd)
Se exp(1b) (op(1))=rpp (00) (2r Lan, اf )d (1 (
0b (0) (ot()))
-(s)-- (()pp()) (Iدج) (1'
Sag exp(tb") x x (x | و ddp (at outtt)
03 (000 (2f
-(st)-1 (1-) - Sag exp (tb' whi (2)) (x ( )l (d) (w (eott)))ddn
(1) (xt [ト)
-(st)-1 (xtp) (z*)a0))) - Sag exp (tb) zpppl (c)) (x² =y (
Pi (d) (I+(z)
-(st)-1 (0* (2"pp(((( (I - Sag expl (tb Jobp (d)) (#g1)))rr) pl (ze ((t()
Pi _ (x*) miti)
-(st)-1 x²
exp(tt") (d) (at (atla (r)d
where 8f = p. px00p (xp((2bt)-1 This is exactly the desired Equation (11). Similar calculations allow to 26
prove Dr log Pi (xt aо) = S(() (a exp(tA)zo).
(d)
B Fokker Planck equation
In this Section discuss the infinite dimensional generalization of the classical Fokker Planck equation. We can associate to Eq. the differential operator:
Lou(x, () = Deu(E, () + 2 Tr(RDuu(E, () + (Ax + f(x, 1), Dzule, 0), a E H,EE T],
Cu(a,)
(29)
where D. is the time derivative, Day D2 are first and second order Fréchet derivatives in space. The domain of the operator Lo is D(Lo), the linear span of real parts of functions Ughh ((t) exp(i(x, h(t))), E H. [o, 7] where ő E C* ((), (TT) = 0, h E C* ([0. D(A))) where i indicates adjoint. Provided appropriate conditions are satisfied, see for example Bogachev ct al. 2009 2011), the time varying measure Pe (drr)dt exists, is unique, and solves the Bokker-Plance equation L.p: = O.
C Uncertainty principle
We here clarify that Hilbert spaces of square integrable functions that are not, in general, simultane- ously homogeneous and separable. For example, while IR is homogeneous, the set of square integrable functions over not separable, since the basis is the uncountable set cos cos(2mup), sin(2xvp), v E TR. Then, FDP requirements not met, we need countable basis. Moreover, would need in
20

---

general an infinite number of samples (grid over the whole R) to reconstruct the functions. Conversely, set like the interval [0, 1] C R has countable basis cos(2Ttp), sin(t))))((2 Z (and thus is separable) and, considering to be band-limited, a sampling grid with finite cardinality would allow to reconstruct of the function. However, / is not homogeneous as no isometry group exists. Consequently. [Theorem 2 not applicable. To fix the issue, one could naively think of extending any function defined over / to the whole IR by considering ølp] = x[p],P | E I and ø[p] = O, & 1. Obvi- ously, if L2(4) then E L2(R). However, since I has finite support, it cannot be bandlimited, making such an approach not a viable solution. In classical signal processing literature, the problem is usually referred to as the uncertainty principle (Slepian 1983).
D A complete example
We present an example in which we cast Equation (20) for square integrable functions over the interval / = 0. 1]. (T). In this case, one natural selection for the basis is the Fourier basi e = (x_.,,,x((_j2222) expr=2p),xpppp))) exp(-i2rp)) 1, exp(j2rp), (xpp)rr))))
Assume the operator A to be a pseudo-differantial operator, such that (Ax, e²) = 'tt Also, assume that bc, r are selected such that conditions of Corollary aare met, and consequently the backward process exists. Since we are working with samples collected on the grid x [Yx] we first map the samples to the frequency domain, and then build Fourier-like representation with finite set of sinusoids. We then define the mapping (= def En= 2² eexp(-j2ri x) and its inverse ((=))& N Σ 2f exp(j2nk ))
This suggests to consider the following expression for the interpolating functions:
s'=
N-1
Σ K=
c exp (-j2mk N
N-1
Σ exp j2 - 1 k=
-N)) 2))
Those functions are indeed nothing but frequency truncated version of the sine function, which is the classical reconstruction function of the sampling theorem I-D signals. Moreover (E', <*) 8(i - k). We are now ready to show i) the expression of the forward process, ii) the expression the parametric score function sg and T/o iii) the computation of the ELBO and finally iv) the expression for the backward process.
The forward process defined in Equation (20) has expression:
dx, [+/N]= = 3 (f3((x)//))))) dt + dw, [*/N], k == 1,.....,
(30)
where dW, [+/N] = <dW:).. Simple calculations show that X, [4/]] is equivalent distribution to V
X, |//N| = (xxp)
(31)
where s¹ - (S()), = - p (2)t))))))
2 and e¹ ~N(0,) 1). allowing simulation of the forward process in single step. The parametric score function can be approximated as:
s0
Σ x.lINNIss, [//N=
(32)
3 ((x, -xx/N/()) (x6(0*))(((((((x( (16 sh
3
Similarly:
Σ x.|%/Nss E No xo1v/NES.) [/N]= [
70
(33)
(x+) sk
i
(3 (n(g(X [/N)), Xollm))
J
We stress that although we should consider real Hilbert space, we select the complex exponential avoid cluttering the notation. It is possible to select [cos( (2mp), sin(2rp), cos (2m22))) sin(k)p)) as a basis, and redoing the calculations in this Section we can obtain functionally equivalent scheme as the one with the real basis.
21

---

CombiningEquatiion (31) and Equation (33) we can fully characterize the training objective defined in
Equation (19) Then, it possible to optimize the value of the parameters e with any gradient- based optimizer.
Finally, the backward process approximation is expressed as:
dX, [t/N] -J (1'3((X(((/1) + r'3 se Σ X KeIENNEE
dt dw, [//]]
(34)
1......... Z),
from which new samples can be generated.
D.1 Proofs
We start by proving Equation (30)
Starting from the drift term of Equation (20) we have the
following chain of equalities
N
N-1 Σ
N-1
X.WNEE
X.
Σ
è'
exp
-j2ml .5
N
N
1=0
1=0
100
N-2
N-1
b'd' exp
-j2al N
5*
N
N-1
x, 1//N
Σ
D' eexp(j2ml//N) (xpp)))xpppppp2ppppp
N
Σ b' ((X
((/(())
The noise term dW, [t/N] is approximated as:
aw, [//]] (dw, <*) awie', ) - 20
dwi exp (j2T N
3(dw)))
where we are truncating the sum. The score has expression:
X (1NEE -(S(t))
Σ Xi 1//N/ 1/))
50
(E (0000000000((((110)0)0,,00,00,
cb
gk
Ch exp(b*t) (n(g(XX [/N]), ()x
ck
sk
Ch exp(b*1) (N-1 n(g( (VN)), 0)
exp j2 x))
4
22

---

where the approximation is due to the substitution of explicit scalar product with the discretized version trough S. When evaluated on the grid of interest:
Σ Xx11/N55 [YN]= (C+ exp(b (N-1 ( [ exp( (xxpl-jmmk: k))) exp(j2nkl/N)
3 ((X,|/N) exp(U"t) (n(g(X, [/)),, [/w]]
The value of 7o. Equation (33) and the expression of the backward process, obtained similarly, considering the above results.
Equation (34) are
E Implementation Details and Additional Experiments
In all experiments extends to the 2-dimensional case what we described in Appendix D.1] As stated in the main paper, our practical implementation sets f 0: then, we only need specify the value for the parameters by , In our implementation we consider an extended class of SDEs that include time-varying multiplying coefficients in front of the drift and diffusion terms, as done for example the Variance Preserving SDE originally described by SSng & Ermon (2020). This can be simply interpreted as the time-rescaled version of autonomous SDEs.
the the complex Fourier basis for the Hilbert spaces, indexed by k. This
E.1 Architectural details
INR-based score network. In our implementation, we use the original INR architecture (Sitzmann) al., (2020). For the specific denoising task we consider in our model, we extend the input of the network architecture to include the corrupted version of the input sample and the diffusion time t, in addition to the spatial coordinates. We emphasize that our architectural design simple, and does not require self-attention mechanisms (Song & Ermon, 2020). The non-linearity we use in our network is Gabor wavelet activation function (Saragadam al. 2023). Furthermore, we found beneficial the inclusion of skip connections.
As stated in the main paper, we consider the modulation approach INRs. In particular, implement the meta-learning scheme described by Dupont et al.) (2022b); Finn et al. (2117). The outer loop is dedicated learning the base parameters of the model, while the inner loop focuses on refining the base parameters for each input sample. In the outer loop. the optimization algorithm AdaBelicf (Zhuang et al. 2020), sweeping the learning rate over Ie-4, 1c-5, 1c-6. We found the use of a cosine warm-up schedule to be beneficial for avoiding training instabilities and convergence to sub-optimal solutions. The inner loop implemented by using three steps of stochastic gradient descent (SGD).
Transformer-basee score network. In our experiments with the Transformer architecture for score modeling, we employed the UViT backbone [aoo et al. (0022). This backbone processes all inputs, be they temporal or noisy image patches, as tokens. Rather than utilizing UViT's default learned positional embeddings, we adapted to integrate 2D sinusoidal positional encodings. For the noisy input images, patch embeddings transform them into sequence tokens. Notably, we chose a patch size of to fully harness the functional properties of our framework. Time embeddings are computed based on the time and then concatenated with the image tokens.
Our chosen transformer architecture comprises layers, with each layer composed a self-attention mechanism with attention heads and feedforward layer. Furthermore, between the shallower and deeper layers, as outlined by Bao et al. (2022).
long skip connections
For optimization during our training, we utilized the AdamW/Loshchin & Hutter (017) algorithm with weight decay of 0.03. We employed a cosine warm-up schedule for the learning rate, which ends at a value of 2c-4.
23

---

-12
14
Figure Left: real (red) and generated samples (bluc). Center and Right: Samples diffused for times 0.2 and 1.0 respectively.
E.2
Additional results
E.2.1 A Toy example.
We here present some qualitative examples on a synthetic data-set of functions E L([-1, 1)), and therefore consider the settings described in Appendix D. The Quadratic data is generated in (Phillips et al., 2022). i.e. Xolp] = qp² + c, where с ~ 0.1) and is a binary random variable that take values { 1- 1, 1} with equal probability. Concerning the design of the forward SDE, select 6* min( Vk, 10) and p - (thus satisfying Corollary 1p. The real data generated considering a grid of 100 equally spaced points. We can see in Figure 3|some qualitative results. On the left real (red) and generated through FDP (blue) samples show good agreement. Center and right plots depict some example of diffused samples for times 0.2 and 1.0 respectively.
E.2.2 MNIST data-set
We evaluate approach simple data-set, using MNIST 32 32 (LeCun al. 2010 In this experiment, we compare our method against the baseline score-based diffusion models Song al. (202I). which we take from the official code repository hatps http//ggttttb.be com www.yang--aaeeeeeeeeeeeeoo score acere_sde The baseline implements the score network using U-NET with self-attention and skip connections, as indicated by current best practices, which amounts to O(10)) parameters.
Instead, our method uses a score-network//nn implemented as a simple MLP with 8 layers and 128 neurons in each layer. The activation function is sinusoidal non-linearity (Sitzmann et al. 2020). Our model counts O(10)) parameters. We consider an SDE with parameters p
k 4 42 and
176
fl.m min ( + m² + 0.3) 0.3)
k.m
3.6).. These values have been determined empirically
33
by observing the power spectral density of the data-set, to ensure a well-behaved Signal to Noise ratio evolution throughout the diffusion process for all frequency components.
"Strictly speaking, the sum of the series p.mn is not convergent. We experimented changing the decay to ensure convergence, but we observed no numerical difference with the settings we the setting we used. It is an interesting avenue for future work study this approximation has impact for higher-resolutio data-sets.

---

1
6
8
0
80
6
5
4
8
1
9
6
9
9
8
5
/
9
9
I
9
8
5
/
9
O
Figure 6: Example of super-resolution of Mnist images. From left to right: initial (training) resolution higher
resolutions.
6
8
7
O
0
J
9
9
7
5
/ O
0
7
8
1
6|
7
O
6
8
5
9
4
3
7
8 3 8 3 2
5
3
1
8
6
6
2
3
8
U
2
7
0
2
2
9
3
7
6
O
3
7
2
4
6
2
6
6
0
0 1
4
7
9
2
2
2
0
7
0
4
6
7
1
4
9
8
/
8
0
9
6
4
0
6
2
0
7 9
7
5
6
2
4 6
8
0
/
8
O
0
3
7
9
8
5
A
6
3
2 1
4
/
0
/
O
2
q
O
4/
9
3
5
3
9
2
3
6 6 8
0
8
3
/
3
5
2
6
4
T 5 4
2
9
3
1
7
6
7
0
3
9
4 9 8
7
7
6
8
7
6
2
7
8
8
0
9 7
7
8
2
0
7
8
7 3 3
O
7
9
7
7
3
8
8
9
3
9
6
S
5 Z
3
5
6
3
/
O
9
O
4
6
4
2
O
7
8
6
O 9
8
/
6
3
9
/ 4
4
0
O
8
6
6
&
9
/
O
6
2
9 7
/
q
6
7
8
9
5
3
2
Figure S: Top right: MNIST real samples. Top Left:
8
Q
/
6 9
2
3
0
Each sample is diffused for a given random time. Bot- tom: output of INR for corresponding input noisy im- age.
Figure 4: MNIST samples generated according to our proposed FDPs.
In Figure 4 we report un un-curated samples generated according to our FDP. In InFiugee 5 we present instead various "intermediate" noisy versions of the training data, to illustrate the kind of noise we use to train the score network, and the output of the denoising INR. We also report the Fréchet Inception Distance (FID) score computed using 16k samples (lower is better). For the baseline we obtain FID=0.05, whereas for the proposed method we obtain #1D=0.43. Although the FID score is in favor of the baseline, we believe that our results obtained with simple MLP - are very promising, as further corroborated by experiments a more complex dataset, which we show next.
Super resolution. We demonstrate how an INR trained at a 32x32 resolution on MNIST can be seamlessly applied to increase the resolution of the generated data points. Given that the INR establishes a mapping between grid and its corresponding value, we initiated the diffusion process from a grid at 32x32 resolution. The diffusion process continued until the final step, where we used the last learned parameters to extrapolate outputs at a higher resolution. significant advantage of using INRs is the ability to conduct the instrcce sampling at lower resolution and then effortlessly transition between resolutions with just a single forward call to the model. Figure 6 shows results at different resolutions.
E.2.3 CELEBA data-set
For the CELEBA data-set we considered the same SDE for the MNIST experiment. Results reported in the main paper have been obtained using a numerical integration scheme variant of the predictor-corrctor scheme of (Song & Ermon 2020), which we adapted to the SDEs we consider in our work. In Figure and Figure 9we report additional un-curated samples obtained with the INR and Transformer respectively. We proceed to describe further experiments in the following section.
Conditional generation. the following, we consider three use-cases for conditional generation: in in-painting, de-blurring, and colorization, which we describe next. All these additional experiments
25

---

were completed using the same architecture and configuration of the unconditional generation described above.
In-painting. We perform in-painting experiments by adopting the same approach described by Song & Ermon (020), and report results in Figure 10] Original images (left-column of Figure 10p masked (center-column Figure 108, where we set the value corresponding to the missing pixels to The right column Figure shlshows the results of the in-painting scheme where, qualitatively, is possible to observe that the conditional generation is able to fill the missing portion of the images while maintaining good semantic coherence.
De-blurring. Our FDPs are naturally suited for the de- blurring case as shown in Figure 12. In this experiment, take the original images (left column of Figure 12) and filter them with a low pass filter (center column of Figure 12). The de-blurring scheme is implemented as the in-painting approach described by [Sngg& & Ermon (2200, where the only difference that the masking each update is applied in the frequency domain. The right column of Figure 12 shows that our technique gracefully recovers missing details and is capable of producing high quality images conditioned on the distorted inputs.
Colorization. In this use-case, adapt the approach from (Song & Ermon 2020) to our setting. Figure IT depicts qualitative results of the colorization experiment, confirming the flexibility of the proposed scheme.
E.2.4 SPOKEN DIGIT data-set
To demonstrate the versatility of our framework, we conducted preliminary experiments on an audio dataset, specifically the Spoken Digit Dataset. This dataset comprises recordings of spoken digits, stored as wav files at an 8kHz sampling rate, with each recording trimmed to minimize silence at the beginning and end. The dataset features five speakers who have contributed recordings, providing 50 recordings of each digit per speaker. The dataset is publicly available on the TensorFlow Dataset Catalog. As preprocessing, each sample was either padded with zeros or truncated to a maximum duration of one second. Subsequently, the data was normalized using function from the pydub library, and each sample scaled range of the [-1,+1] +1] by dividing it by the dataset's maximum intensity. For the audio experiments, we fed the log mel spectrograms, which is common practice in audio processing tasks. The transformer model was configured with patch size of 2, an embedding size of 512. 13 layers, and heads, We employed the AdamW optimizer with a weight decay of 0.03 and cosine warmup schedule that decays at value of 1e-5. The preliminary examination of the audio generated by our model reveals its ability effectively generate spoken digits. Upon listening to the model's generated samples, we were able to recognize the digits accurately, showcasing the model's potential in audio generation tasks. FFiuure pprovides comparative analysis of the waveforms generated by our model against real examples from the dataset.
total of 2,500
raw audio data directly into the transformer model, without converting
Waveform
Weveefom
3.0
-270
Time
Time 50
(a) Real samples
(b) Generated samples
Figure Comparison of real and generated waveforms.
26

---

Figure 8: Uncurated CELEBA samples generated by the INR.
27

---

Figure 9: Uncurated CELEBA samples generated by the Transformer.
28

---

2
5
2
5
0 9 5 5 5
7 5
4
0 9 9 7
7
0
4
3 6 7
7
3
1
9
3
7
O
7
8
2
9
Figure 10: In-painting experiment using INR. Left:: real samples, Center: Masked samples, Right: Reconstructed samples.
29

---

Figure 11: Colorization experiment using INR. Left: real samples, Center: Gray-scale samples, Right: Recon-
structed samples.
30

---

Figure 12: De-blurring experiment using INR. Left: real samples, Center: blurred samples, Right: Reconstructed samples.
31


Reviews:
[{"review_id": "owt6EBNqCE", "reviewer": "Reviewer_GA6N", "summary": "The authors introduce a framework for infinite dimensional diffusion models in Hilbert spaces, including deriving a reverse process and novel ELBO loss objective. The authors introduce a novel score model network architecture.", "questions": "- The implicitly defined INR encoder, g, detailed in equation 16, considers gradient descent in an inner layer. How many steps are taken? Is the gradient through the score network taken through these gradient descent steps? Is this not quite slow? Have the authors considered other point-wise encoders?", "limitations": "The authors have a brief paragraph on limitations regarding perform, more challenging experiments and comparisons to other approaches like NFO, but could be further discussed. See questions and weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is a timely paper with a lot of interest in the community. There are many concurrent works also looking into similar topics e.g. [1] (I do not expect detailed comparison given they were only public recently).\n- Although performance is not great compared to standard diffusion methods in terms of generative modelling, the results are better than Neural Processes and similar point wise methods (with the exception of [2], concurrent so cannot criticise for worse performance)\n- The use of implicit optimization based encoder, g, in equation 16, is interesting. I have some concerns detailed below but nevertheless it is interesting.\n- Theoretical results seem correct but have not been checked thoroughly\n\n[1] Infinite-Dimensional Diffusion Models for Function Spaces, Pidstrigach et al, 2023\n[2] Infinite resolution diffusion with subsampled mollified states., Bond-Taylor et al, 2023", "weaknesses": "- The implicitly defined INR encoder, g, detailed in equation 16, considers gradient descent in an inner layer. Will this not be quite slow and hence generation / sampling would also be quite slow?\n- Empirical performance / FID scores are quite bad in comparison to other standard methods. Although interesting, is there an application where this method would be more useful than standard methods? \n- Functional diffusions could be interesting for dimension agnostic diffusions, super resolution, irregular data. There are limited experiments looking into these other applications.\n- Complexity in table 1 is quite subjective. I would argue parameter could does not matter if it can be ran on a single GPU in reasonable wall clock time. The implicit layer, g, with gradient descent for equation 16 is sequential so I imagine this slows down generation significantly.\n- Calling other concurrent / prior work heuristic without details. \"However, these works do not formally prove the existence of a backward process and their score approximation and score matching optimization objective is heuristic\". Does not [1] prove existence in Theorem 1? (I do not expect detailed comparison given they were only public recently, but given the authors claim this conccurrent work is heuristic I would be interested to know more).\n\n[1] Infinite-Dimensional Diffusion Models for Function Spaces, Pidstrigach, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Weaknesses:\nThe implicitly defined INR encoder, g, detailed in equation 16, considers gradient descent in an inner layer. Will this not be quite slow and hence generation / sampling would also be quite slow?*\n\nPlease refer to the common remark for all reviewers, above.\n\n*Empirical performance / FID scores are quite bad in comparison to other standard methods. Although interesting, is there an application where this method would be more useful than standard methods?*\n\nIndeed, we obtain a FID score that is larger than the state of the art, but with several orders of magnitude fewer parameters! In addition, we have now a set of new results where the FID score is much lower (see Fig. 2, additional pdf). These new results have been obtained with an alternative implementation of the score network, based on a simple Transformer architecture. In general, the FID metric should be taken with a grain of salt, as it is not necessarily  representative of the true underlying quality of the generated data [Gaurav]. \nDespite a rather involved mathematical formulation, the functional approach is interesting in particular for practitioners. Indeed, in principle, it allows working on any data modality using the same architecture for the score network. Standard methods, on the other hand, require careful design and fine tuning, which can be lengthy and expensive tasks. We have additional preliminary results on audio data, whereby FDPs use the same score network as for image data. \n\n*Functional diffusions could be interesting for dimension agnostic diffusions, super resolution, irregular data. There are limited experiments looking into these other applications.*\n\nIndeed, our goal in this paper is to develop in detail a new methodology: this requires a  careful, rigorous and lengthy analysis, which sacrifices space for empirical validation.\nPlease refer also to the common remark above for additional comments and results.\n\n*Complexity in table 1 is quite subjective. I would argue parameter could does not matter if it can be ran on a single GPU in reasonable wall clock time. The implicit layer, g, with gradient descent for equation 16 is sequential so I imagine this slows down generation significantly.*\n\nFirst, we would like to clarify that by “complexity”, we refer to the (subjective) effort of a practitioner to design the score network architecture. Parameter count is listed separately from “complexity”. For example, the work InftyDiff, requires the cascade of multiple models (NFO, knn interpolators, Convolutional Unets), whereas the architectures we  consider in our work are much simpler (INRs are literally vanilla MLPs with sinusoidal activation functions, the new transformer based score network has no convolutional layers, etc…).\nAs a reminder, we consider the implementation of the score network as an engineering choice: INR-based score nets are small and simple, but require meta-learning, the newly introduced Transformer-based score nets are larger, but do not require meta-learning.\n\n*Calling other concurrent / prior work heuristic without details. \"However, these works do not formally prove the existence of a backward process and their score approximation and score matching optimization objective is heuristic\". Does not [1] prove existence in Theorem 1? (I do not expect detailed comparison given they were only public recently, but given the authors claim this concurrent work is heuristic I would be interested to know more).*\n\nWe apologize if the message was not clear. In the concurrent work [1], the existence of the backward process is proven, where the score functional is introduced. We are however, at the time of the submission of the manuscript, and to the best of our knowledge, the first one to show that the score matching objective is not only an heuristic but a sound variational bound. We achieve this thanks to the infinite dimensional generalization of the Girsanov theorem.\nAs it has also been requested by another reviewer, we will expand more how our work differs from related works, clarifying such details.   \n[1] Infinite-Dimensional Diffusion Models for Function Spaces, Pidstrigach, 2023\n\n*Questions:\nThe implicitly defined INR encoder, g, detailed in equation 16, considers gradient descent in an inner layer. How many steps are taken? Is the gradient through the score network taken through these gradient descent steps? Is this not quite slow? Have the authors considered other point-wise encoders?*\n\nPlease refer to the common remark above. In short, yes INR requires meta-learning which can be challenging, and yes, we considered different implementations of the score nets using simple Transformer architectures.\n\n\n\nParmar, Gaurav, Richard Zhang, and Jun-Yan Zhu. \"On aliased resizing and surprising subtleties in gan evaluation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n"}, {"review_id": "KJZ8ztbbAk", "reviewer": "Reviewer_jQYa", "summary": "The authors propose a model approach called Functional Diffusion Processes (FDPs) that generalizes score-based diffusion model to infinite-dimensional function space. The authors derive the reverse time dynamics and sampling theorems to find a subset of functions on a countable set of samples without losing information. The authors demonstrate their approach on a multi-layered perception.\n", "questions": "The theoretical parts of the paper is quite dense and I cannot say I have fully understood the paper to come up with any meaningful questions.", "limitations": "N/A", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theoretical parts of the paper appears to be well structured. However, the paper itself does not appear to be self contained and has many references to the appendices to understand. For example, the paper makes many references to the assumptions being made, but they are never mentioned in the main paper and are only mentioned in the appendix.\n", "weaknesses": "2. The authors claim their approach is to create a new breed of generative models in function space, which do not require specialized network architectures, and that can work with any kind of continuous data. This itself is an interesting idea. However, after reading the introduction and looking at the experimental results. I cannot see what applications this may benefit and how much of an improvement it has to have a purely functional domain. I believe this could be better motivated.\n3. This may be because I didn't understand the experiment in Section 6 and does not appear to be self contained. I can understand that there is a page limitation, however, I believe that the experiment section is a bit compressed and difficult to understand. Having said that, the authors do have additional experiments in the appendix which appears to be much better structured.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*The theoretical parts of the paper appears to be well structured. However, the paper itself does not appear to be self contained and has many references to the appendices to understand. For example, the paper makes many references to the assumptions being made, but they are never mentioned in the main paper and are only mentioned in the appendix.*\n\nThank you for raising your concern about our paper being self-contained. We understand the importance of making the paper accessible to readers.\nWhile we aimed to maintain a concise presentation within the given page limit, we acknowledge that striking a balance between brevity and comprehensiveness is challenging.  To improve the readability of our paper, we plan to incorporate a summary of the key assumptions in the main paper. This will provide readers with a more coherent understanding of the underlying concepts without relying heavily on appendices.\n\n*The authors claim their approach is to create a new breed of generative models in function space, which do not require specialized network architectures, and that can work with any kind of continuous data. This itself is an interesting idea. However, after reading the introduction and looking at the experimental results. I cannot see what applications this may benefit and how much of an improvement it has to have a purely functional domain. I believe this could be better motivated.*\n\nThe motivations for adopting a functional perspective are multiple. First, we claim that working in a functional domain allows greater parameter efficiency and simpler architectural design, as already shown in the submitted version of the manuscript. Moreover, the approach allows using the same architectures for different modalities. We performed some preliminary experiments on audio datasets, which we will include in the camera ready, using the same exact architectures. \nFinally, the functional approach allows the design (and use) of architectures which are dimension agnostic. This claim is supported in the rebuttal by the new set of experiments concerning super-resolution tasks. \n      \n*This may be because I didn't understand the experiment in Section 6 and does not appear to be self contained. I can understand that there is a page limitation, however, I believe that the experiment section is a bit compressed and difficult to understand. Having said that, the authors do have additional experiments in the appendix which appears to be much better structured.*\n\nWhile it is true that we left out the experimental details (like selection of parameters b and r) for the appendix, we clearly stated the considered datasets, the competitors and the metrics on which the different methods were compared. We are convinced that the information in the main paper and the appendix to be sufficient to understand our experiments. Moreover, we will release, if the paper is accepted, the full source code of the software implementation of our work, with instructions to reproduce the experiments we did, and additional details for practitioners who would be interested in using our approach.\n"}, {"review_id": "grrhUDDtJn", "reviewer": "Reviewer_p4Hy", "summary": "The paper introduces a novel continuous-time diffusion-based generative model on function space. Unlike previous works in this area, which primarily focused on discrete-time formulations, the authors concentrate on stochastic differential equations (SDE) in their approach.\n\nThe paper begins by defining the forward process, also known as the noising process, as an stochastic differential equations (SDE) on function spaces. The paper subsequently demonstrates the existence of corresponding backward processes under certain conditions, similar to the formulation of finite-dimensional score-based generative models. Notably, two constraints in the forward process are required to guarantee the existence of the reverse diffusion process, which distinguishes the current work from its finite-dimensional counterparts. Firstly, the perturbation noises should be cylindrical Wiener processes, including $R$-Wiener process with a trace-class covariance operator $R$. Secondly, an operator $\\mathcal{A}$ can be present in the drift term of the forward process, which generates a strongly continuous semigroup; the semigroup will serve as a contraction map.\n\nThe paper highlights the novelty of its results by emphasizing that previous approaches have not fully explored the limits of discretization. Therefore, the provided proof offers theoretical support for the utilization of various types of integrators, unlike previous approaches.\n\nFurthermore, the authors introduce an evidence lower bound (ELBO) on the reverse diffusion process, leveraging an extension of the Girsanov theorem. This result provides a theoretical foundation for optimal guarantees, in contrast to previous approaches that relied on heuristics to minimize score matching objectives on function space.\n\nThe paper also extends the sampling theorem, enabling the reconstruction of the original function from its evaluations at a countable set of points. This extension justifies the training of function-space models with finite observations. Consequently, the authors exploit the sampling theorem to employ implicit neural representations for the model parameterizations, unlike previous methods that predominantly relied on neural operators.\n\nFinally, the paper demonstrates the effectiveness of the proposed method on various image generation benchmark datasets. In the experiment, they use Fourier basis functions inversely proportional to the square of the basis index for the perturbation noise.", "questions": "Unlike neural operators, implicit neural implementation is powerful, but achieving discretization invariance is challenging. How do you think is the best way to approach this?", "limitations": "Please refer to the comments provided in the weaknesses section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I believe that the paper's theoretical results will make significant contributions to the machine learning community for several reasons.\n\nFirst, the existence of the reverse process ensures the optimality of common model parameterizations (Markovian), specifically those that model only the drift term of the reverse process, including score modeling.\n\nSecondly, the SDE formulation sheds light on the utilization of various integration methods for sample generation, such as DDIM or exponential integrators.\n\nFurthermore, the derivation of the evidence lower bound (ELBO) using Girsanov results in Equations 12 and 13. In these equations, the covariance operator $R$ is presented in the norm of the $R^{1/2}(\\mathcal{H})$ space. The inclusion of this additional $R$ helps address the problem encountered when defining the Kullback-Leibler (KL) divergence for previous generative models based on discrete-time diffusion in function space.\n\nLastly, the sampling theorem introduces intriguing open questions regarding the parameterizations of score-based generative models in function space. This opens up the possibility of employing new network architectures beyond neural operators.", "weaknesses": "Most of the content in the paper is good. However, some improvements are needed in the experiment.\n\nIn particular, the characteristic of the function-spaced model is its resolution invariance. However, the analysis of this aspect is missing in the paper's results. I believe that some analysis regarding the resolution-invariant property of the learned models should be included, even if it's just a small amount. In such analysis, I don't think it is necessary to strictly rely on an implicit neural representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Weaknesses:\nMost of the content in the paper is good. However, some improvements are needed in the experiment.\nIn particular, the characteristic of the function-spaced model is its resolution invariance. However, the analysis of this aspect is missing in the paper's results. I believe that some analysis regarding the resolution-invariant property of the learned models should be included, even if it's just a small amount. In such analysis, I don't think it is necessary to strictly rely on an implicit neural representation.*\n\nPlease refer to the common remark above.\n\n*Questions:\nUnlike neural operators, implicit neural implementation is powerful, but achieving discretization invariance is challenging. How do you think is the best way to approach this?*\n\nIn principle, INRs and meta learning are — by design — resolution invariant. Indeed, the modulations are obtained by minimizing the average squared distance between the reconstructed and original image. This operation is by construction agnostic to the resolution or the spacing of the grid at which points are evaluated. We include in the rebuttal new experiments showcasing how the INRs can be used to produce higher resolution images.\nIn practice, one can expect the meta learning procedure to present instabilities when tested at resolutions different from the training ones, as the various numerical approximation errors can cumulate and push the meta learning procedure into instability-prone regions. \n"}, {"review_id": "tLrtoSqQFb", "reviewer": "Reviewer_XJB7", "summary": "This work proposes *functional diffusion processes (FDPs)*, which generalizes the SDE-based continuous-time framework for diffusion models for data living in Hilbert spaces. The work builds on the theory of infinite-dimensional SDEs and their time reversals, as well as an application of the infinite-dimensional Girsanov theorem to derive an ELBO objective. A practical implementation, via discretization and implicit neural representations, is proposed and evaluated on the Celeb-A dataset.", "questions": "- The work restricts its attention to classes of functions which can be reconstructed exactly via finite sets of samples (Sect. 3). If my understanding is correct, the observation points may vary across functions. Does the proposed practical implementation allow for this? If so, how would you expect performance of the model to change if one were to e.g. train on one discretization and sample on another, such as an image super-resolution task?", "limitations": "The limitations of the work are adequately addressed in the paper. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is generally clear and well-written throughout.\n- This work adds to the growing literature on function-space diffusion models (a quite active area) and will likely be of significant interest to the machine learning community. The main theoretical portion of this work places continuous-time, SDE-based functional diffusion models on solid theoretical grounds. \n- All theoretical claims throughout the paper are precisely stated and rigorously justified. To the best of my knowledge, the proofs of the claims (contained in the appendix) are correct.", "weaknesses": "- The experiments (section 6) are the weakest part of the paper. \n\t- The proposed methodology obtains significantly worse FID scores than standard (Euclidean) diffusion models. However, note that this is achieved with far fewer parameters.\n\t- It was also somewhat unclear how the \"complexity\" column in Table 1 was derived, as this seems like a highly subjective measurement\n\t- Also in Table 1, it was unclear why FID scores were being compared against FID-CLIP scores (for the model of Bond-Taylor & Willcocks, 2023).\n\t- It was unclear if the parameter counts in Table 1 also included the parameters for the INRs, or merely the parameters for the score network. \n- The relationship between this work and the various other concurrent function-space diffusion model works could be better clarified. \n\n\n#### Minor comments\n- The notation on line 169 was somewhat unclear to me on a first read, particularly what $\\hat{\\mathbb{Q}}_0, \\hat{\\mathbb{P}}_0$ represent\n- There may be a small typo on line 169: should it read $d \\hat{\\mathbb{P}}^{\\chi_T} = d \\hat{\\mathbb{P}}^{\\rho_T} \\frac{d \\chi_T}{d \\rho_T}$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*The experiments (section 6) are the weakest part of the paper.\nThe proposed methodology obtains significantly worse FID scores than standard (Euclidean) diffusion models. However, note that this is achieved with far fewer parameters.*\n\nIndeed, even if FID scores obtained with FDPs are not SOTA, the variant of the score network implemented as an INR achieves remarkably good image quality with several orders of magnitude fewer parameters than SOTA methods, which can be important in some application scenarios (e.g. limited resource devices). Moreover, our new set of experimental results obtained with Transformer-based score nets substantially improve FID scores, at the expense of a larger number of parameters. \nNote also that the FID metric is not sufficiently reliable, despite being widely adopted in the literature . Indeed, it is prone to being heavily influenced by small perturbations that are imperceptible to humans [Gaurav]. We include some examples of non curated samples obtained with a different functional architecture (Transformers), showing that the quality of generated images is higher than what the simple FID analysis would suggest. \n\n*It was also somewhat unclear how the \"complexity\" column in Table 1 was derived, as this seems like a highly subjective measurement*\n\nWe do agree that a single metric about complexity could be interpreted as subjective. We will include in the camera ready version of the paper details about the architectures of all the considered competitors instead of simply using general adjectives, because this is what we meant by “complexity”. For example, the work discussed in “Bond-Taylor & Willcocks, 2023” requires multiple blocks, like classical UNeT architectures.\n\n*Also in Table 1, it was unclear why FID scores were being compared against FID-CLIP scores (for the model of Bond-Taylor & Willcocks, 2023)*\n\nClassical FID score is not available for these competitors, so we just wanted to point that out for the reader.\n\n*It was unclear if the parameter counts in Table 1 also included the parameters for the INRs, or merely the parameters for the score network.*\n\nWe clarified this in the shared answer: the implicit neural representation network IS the score network. So, this is the total number of parameters we have in our implementation. Using an INR to implement the score network is possible due to the alignment of the goal of score matching (denoising task) and the intrinsic denoising capabilities of INRs (Kim et al., 2022a)\nThe relationship between this work and the various other concurrent function-space diffusion model works could be better clarified.\nWe do agree, see also the answer to  reviewer GA6N. We will dedicate more space in the camera ready version of the paper to such analysis.\n\n*Minor comments\nThe notation on line 169 was somewhat unclear to me on a first read*\n\nWe apologize for the confusion, we added some extra english wording in the paper to clarify this\n\n*There may be a small typo on line 169*\n\nYes, thank you for spotting the error.\n\n*The work restricts its attention to classes of functions which can be reconstructed exactly via finite sets of samples (Sect. 3). If my understanding is correct, the observation points may vary across functions. Does the proposed practical implementation allow for this? If so, how would you expect performance of the model to change if one were to e.g. train on one discretization and sample on another, such as an image super-resolution task?*\n\nAs explained in the shared part of the answer, we have many new results on this theme, with a positive answer to the raised questions.\n \n\nParmar, Gaurav, Richard Zhang, and Jun-Yan Zhu. \"On aliased resizing and surprising subtleties in gan evaluation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022."}, {"review_id": "U9VLL8oGb0", "reviewer": "Reviewer_vNhS", "summary": "In this work the authors propose an extension to the diffusion models framework for function spaces.\nIn particular, given a Hilbert space $H$, they introduce a noising process based on a Wiener process on $H$. Depending on some assumptions on the type of noise, they then show the existence of the time-reversal process. Then they derive an ELBO via an extension of the Girsanov theorem.\nLater, akin to the the classical Shannon-Nyquist sampling theorem, they show that for a subclass of functions and grids for which the function is uniquely determined by its values on this grid.\nEventually, the process is discretised at the evaluation points, yielding finite dimensional noising and denoising processes.\nThey apply their model to model the CelebA dataset, and show that with relatively few parameters their are able to obtain low FID.", "questions": "- line 98: Why does $R$ has to be diagonal?\n- line 105: Is it correct to interpret the 'trace-class' as the following: we have $\\mathrm{Tr}(R) = \\sum_i \\lambda_i = Var(W_t)$ that is the trace of the operator is given by the arithmetic series of its eigenvalue which itself is the variance of the $R-Wiener$ process, so this can be seen as a finite variance constraint? Which seems pretty natural.\n- Equation 3: How can we define a density $\\rho_t$ since there is no 'Lebesgue-like' (translation invariant) measure on infinite dimensional Hilbert space? Is it w.r.t. a Gaussian measure?\n- Table 1: It is nice to see that FDP can be parameter efficient. Is the reason for not trying larger architecture computational? If so what's the bottleneck? Would $\\inf$ - DIFF with 1M parameter be performing as well?\n- line 333: The MLP architecture seems very deep, have you tried something perhaps a bit shallower yet wider (e.g. 512 neurons)?\n- Section 6: What noise $R$ was used for this experiment? From a Karhunen-Loeve perspective would make sense to use the correlation of the data process see [Angus et al 2022].\n\n\nSpectral Diffusion Processes, Phillips, Angus and Seror, Thomas and Hutchinson, Michael and De Bortoli, Valentin and Doucet, Arnaud and Mathieu, Emile, 2022", "limitations": "- Something that has not been explored, yet can be left , is conditional sampling which I believe is often of interest e.g. looking at the neural processes literature.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- I found it quite easy to follow the submission, especially Section 2 which I think is well fleshed out.\n- The proposed methodology is rigorous and sound.\n- Theorem 2 is nice and potentially useful as it gives sufficient conditions to recover the original function.", "weaknesses": "- Perhaps the main weakness is the limited experimental setups and lack of ablation studies. As such it is hard to really understand the practical benefits of this specific approach.\n- In particular, one advantage of working with functional data is to be able to handle data at discretised at arbitrary resolutions or on varying and irregular grid, yet this work practically focuses on the CelebA dataset. It is interesting to see the parameter efficiency by representing as the data as function, but indeed there is no hope to beat method that assume this fixed grid setting and can rely on specialised architectures (e.g. U-net). It may be of interest to tackle tasks such as upscaling, or data with missing values, irregular time-series etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Perhaps the main weakness is the limited experimental setups and lack of ablation studies. As such it is hard to really understand the practical benefits of this specific approach.\nIn particular, one advantage of working with functional data is to be able to handle data at discretised at arbitrary resolutions or on varying and irregular grid, yet this work practically focuses on the CelebA dataset. It is interesting to see the parameter efficiency by representing as the data as function, but indeed there is no hope to beat method that assume this fixed grid setting and can rely on specialised architectures (e.g. U-net). It may be of interest to tackle tasks such as upscaling, or data with missing values, irregular time-series etc.*\n\nWe do agree with the reviewer that additional experimental validation could strengthen the work. Please refer to the shared answer, as well as the additional pdf, where extra experiments are provided.\n\n*Questions: line 98... has to be diagonal?*\n\nWe do apologize for the imprecision: to define a “diagonal” operator one must specify a basis. There is no requirement on the operator R to be diagonal, it is just a choice that simplifies the design of the SDEs  coefficients. The only restriction we consider when dealing with Hilbert spaces of integrable functions is to have trace-class noise, which, as correctly guessed by the reviewer, just implies a finite variance condition\n\n*line 105: Is it correct to interpret the 'trace-class' as the following... Which seems pretty natural.*\n\nThis is indeed correct (see also point above) and is an important requirement which if not satisfied can hinder existence of valid diffusion processes, a matter not explored by some of our competitors. We will clarify in the text of the revised manuscript.\n\n*Equation 3: How can we define a density since there is no 'Lebesgue-like' (translation invariant) measure on infinite dimensional Hilbert space? Is it w.r.t. a Gaussian measure?*\n\nWe apologize for the technical confusion. It is indeed true that on infinite dimensional spaces there is not an equivalent of the Lebesgue measure, and this is the reason why e.g. [Lim2023] it is only possible to define the ratio w.r.t. some reference measure (like the Gaussian one).\nIn our case, our compact notation indicates the conditional density $\\rho^{(d)}_t(x^i | x^{ j\\neq i})$. This on the contrary is a single dimensional density which exists (it is the same object considered in [Pidstrigach2023]). In the submitted version of the manuscript (line 110-111) we explicit this “To avoid cluttering the notation, we shorten ...” We do understand however that this is an important technical point, and we will expand the notation in the final version of the manuscript\n\n*Table 1: It is nice to see that FDP can be parameter efficient. Is the reason for not trying larger architecture computational? If so what's the bottleneck? Would inf- DIFF with 1M parameter be performing as well?*\n\nWe explored deeper MLP architectures and found no substantial benefits. Concerning Infty-Diff, as stressed in the main paper, the considered architectures are complex combinations of functional and non functional blocks (like convolutional UNet transformers). Consequently, it is not straightforward to reduce the number of parameters without the need to completely re-design the whole architecture.\n\n*line 333: The MLP architecture seems very deep, have you tried something perhaps a bit shallower yet wider (e.g. 512 neurons)?*\n\nYes, and we did not obtain better results. See also the shared answer above.\n\n*Section 6: What noise \n was used for this experiment? From a Karhunen-Loeve perspective would make sense to use the correlation of the data process see [Angus et al 2022].*\n\n We thank for the interesting question. All details about the actual values of b and r are reported in the supplementary material. Our procedure for the selection of coefficients of b and r is indeed intimately related to the observation of the power spectral density of the data. The design should satisfy the following requirements: Signal to Noise ratio (on a frequency by frequency basis) should decrease gracefully up to a low snr regime (which corresponds, roughly speaking, to the uninformative gaussian steady state) and the selection of coefficients must satisfy the requirements of corollary 1. We think it will be an interesting complement to the paper to express the connection between our methodology and the one described in [Angus2022].\n\n\n\nSpectral Diffusion Processes, Phillips, Angus and Seror, Thomas and Hutchinson, Michael and De Bortoli, Valentin and Doucet, Arnaud and Mathieu, Emile, 2022\n\n*Limitations:\nSomething that has not been explored, yet can be left , is conditional sampling which I believe is often of interest e.g. looking at the neural processes literature.*\n\n In the submitted version of the manuscript we included some conditional sampling experiments like deblurring, inpainting and colorization. We moreover include in this rebuttal some super-resolution experiments conditioned on lower resolution images. Class-conditional experiments, possible with the proposed method with minor modification to the considered architectures, are an interesting proposal for future works.\n"}]

Author rebuttal:
We sincerely thank the reviewers for their thorough analysis of our work and for their very positive feedback on our paper.
We here provide some general comments which are common to all reviewers, and clarify the minor technical points directly in the messages to individual reviewers.

*Implicit Neural Representation: clarifications, discussion of limitations and considered alternatives*

First, we clarify that the considered INR effectively implements the score network, they are the same object (see lines 292-293)
In our experiments, we tried a variety of architectural choices for the INR (deep and narrow vs. shallow and wide architectures) and found, in accordance with literature results [Dupont2022b], that the former design choice consistently provides better results. 
The reviewers correctly point out a possible drawback of the meta-learning approach for INRs, which require local adaptation steps. It is however important to stress that our design goal is to use as few inner steps as possible (we obtain our results with only 3 inner steps). This is in line with the approach described in [Dupont2022b].
Note that we view the design of the score network as an “engineering” choice. In the submitted paper, we list possible alternatives and present results with an INR, which has the clear benefit of requiring an extremely small number of parameters. In addition, we have new original results where the score network is implemented using a simple Transformer architecture (interpreted as a mapping between Hilbert spaces, as discussed in [Cao2021]). These new results, which we will add to a possible camera ready version of the paper, indicate improvements in terms of image quality (FID score=11, fig 2 in the  additional pdf) and do not require meta-learning steps, but require more parameters (O(40M)) than the INR variant. In our experiments with Transformers, we adopted the UViT backbone[Bao]. This backbone treats all inputs, whether time or noisy image patches, as tokens. Instead of using UViT's learned positional embeddings, we modified it to incorporate 2D sinusoidal positional embeddings as described in [Fan]. 

*Resolution invariance/ Different modalities*

A shared request among the reviewers is an experiment to illustrate the benefits of FDPs for an example task which can leverage the intrinsic dimensionality agnostic property of a functional representation of data.
We present new experimental results on the task of super resolution. We demonstrate how the same INR trained in the submitted version of the paper can be seamlessly applied to increase the resolution of the generated data points. (Refer to additional pdf fig 1). This is a practical approach that leverages the properties of INRs.
Moreover, we have preliminary results on different data modalities (audio waveforms) using the very same Transformer architectures considered for image data, grounding our claim that FDPs allow to simplify the design of the score network for a variety of application domains.

In principle, nothing prevents using FDPs on datasets where data has been collected on irregular grids. Indeed, the sampling theorem described in Section 3 does not require regularity of the grid but only that the covering is “sufficiently fine grained”. Moreover, score networks based on either INRs or Transformers have no problems in dealing with irregularly spaced data (to the best of our knowledge, NFOs instead require regular spacing). The major technical problem to address in case of irregular data, is the numerical simulation of the infinite dimensional SDEs, for which efficient integration schemes are elusive. In our future work, we plan to investigate such peculiar integration schemes.

*Bao, Fan, et al. "All are worth words: A vit backbone for diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.*

*Fan, Haoqi, et al. "Multiscale vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.*


Meta-review: The authors present a new framework for infinite-dimensional diffusion models valued in a Hilbert space. In particular, this framework encompasses the derivation of the reverse process of an infinite-dimensional diffusion, which generalizes the finite-dimensional setting. Then, the authors introduces a  Evidence Lower Bound (ELBO) loss objective to learn the drift of this reverse process to generate new samples. Finally, the paper introduces a novel score model network architecture.

The reviewers and I think that this paper is an interesting contribution to the diffusion models literature. As a result, we all agree that the paper should be accepted. 

Please take the reviewers' comments into account when submitting the final version of your work.
Decision: Accept (poster)