Title: Continuous-Time Functional Diffusion Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Hilbert spaces

Diffusion models

Stochastic Partial Differential Equations
Abstract: We introduce Functional Diffusion Processes (FDPs), which generalize score-based diffusion models to infinite-dimensional function spaces. FDPs require a new mathematical framework to describe the forward and backward dynamics, and several extensions to derive practical training objectives. These include infinite-dimensional versions of Girsanov theorem, in order to be able to compute an ELBO, and of the sampling theorem, in order to guarantee that functional evaluations in a countable set of points are equivalent to infinite-dimensional functions. We use FDPs to build a new breed of generative models in function spaces, which do not require specialized network architectures, and that can work with any kind of continuous data.
Our results on real data show that FDPs achieve high-quality image generation, using a simple MLP architecture with orders of magnitude fewer parameters than existing diffusion models.

Continuous-Time Functional Diffusion Processes

Abstract

We introduce Functional Diffusion Processes (FDPs), which generalize scorebased diffusion models to infinite-dimensional function spaces. FDPs require a new mathematical framework to describe the forward and backward dynamics, and several extensions to derive practical training objectives. These include infinitedimensional versions of Girsanov theorem, in order to be able to compute an ELBO, and of the sampling theorem, in order to guarantee that functional evaluations in a countable set of points are equivalent to infinite-dimensional functions. We use FDPs to build a new breed of generative models in function spaces, which do not require specialized network architectures, and that can work with any kind of continuous data. Our results on real data show that FDPs achieve high-quality image generation, using a simple MLP architecture with orders of magnitude fewer parameters than existing diffusion models. Code available here.

1 Introduction

Diffusion models have recently gained a lot of attention both from academia and industry. The seminal work on denoising diffusion (Sohl-Dickstein et al., 2015) has spurred interest in the understanding of such models from several perspectives, ranging from denoising autoencoders (Vincent, 2011) with multiple noise levels (Ho et al., 2020), variational interpretations (Kingma et al., 2021), annealed (Song & Ermon, 2019) and continuous-time score matching (Song & Ermon, 2020; Song et al., 2021). Several recent extensions of the theory underpinning diffusion models tackle alternatives to Gaussian noise (Bansal et al., 2022; Rissanen et al., 2022), second order dynamics (Dockhorn et al., 2022), and improved training and sampling (Xiao et al., 2022; Kim et al., 2022b; Franzese et al., 2022).

Diffusion models have rapidly become the go-to approach for generative modeling, surpassing GANs (Dhariwal & Nichol, 2021) for image generation, and have recently been applied to various modalities such as audio (Kong et al., 2021; Liu et al., 2022), video (Ho et al., 2022; He et al., 2022), molecular structures and general 3D shapes (Trippe et al., 2022; Hoogeboom et al., 2022; Luo & Hu, 2021; Zeng et al., 2022). Recently, the generation of diverse and realistic data modalities (images, videos, sound) from open-ended text prompts (Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022) has projected practitioners into a whole new paradigm for content creation.

A common trait of diffusion models is the need to understand their design space (Karras et al., 2022), and tailor the inner working parts to the chosen application and data domain. Diffusion models require specialization, ranging from architectural choices of neural networks used to approximate the score (Dhariwal & Nichol, 2021; Karras et al., 2022), to fine-grained details such as an appropriate definition of a noise schedule (Dhariwal & Nichol, 2021; Salimans & Ho, 2022), and mechanisms to deal with resolution and scale (Ho et al., 2021). Clearly, the data domain impacts profoundly such design choices. As a consequence, a growing body of work has focused on the projection of data

modalities into a latent space (Rombach et al., 2022), either by using auxiliary models such as a VAEs (Vahdat et al., 2021), or by using a functional data representation (Dupont et al., 2022a). These approaches lead to increased efficiency, because they operate on smaller dimensional spaces, and constitute a step toward broadening the applicability of diffusion models to general data.

The idea of modelling data with continuous functions has several advantages (Dupont et al., 2022a): it allows working with data at arbitrary resolutions, it enjoys improved memory-efficiency, and it allows simple architectures to represent a variety of data modalities. However, a theoretically grounded understanding of how diffusion models can operate directly on continuous functions has been elusive so far. Preliminary studies apply established diffusion algorithms on a discretization of functional data by conditioning on point-wise values (Dutordoir et al., 2022; Zhuang et al., 2023). A line of work that is closely related to ours include approaches such as Kerrigan et al. (2022), who consider a Gaussian noise corruption process in Hilbert space and derive a loss function formulated on infinite-dimensional measures to approximate the conditional mean of the reverse process. Within this line of works, Mittal et al. (2022) consider diffusion of Gaussian processes. We are aware of other concurrent works that study diffusion process in Hilbert spaces (Lim et al., 2023; Pidstrigach et al., 2023; Hagemann et al., 2023). However, differently from us, these works do not formally prove that the score matching optimization is a proper evidence lower bound (ELBO), but simply propose it as an heuristic. None of these prior works discuss the limits of discretization, resulting in the failure of identifying which subset of functions can be reconstructed through sampling. Finally, the parametrization we present in our work merges how functions and score are approximated using a single, simple model.

The main goal of our work is to deepen our understanding of diffusion models in function space. We present a new mathematical framework to lift diffusion models from finite-dimensional inputs to function spaces, contributing to a general method for data represented by continuous functions.

In § 2, we present Functional Diffusion Processs (FDPs), which generalize diffusion processes to infinite-dimensional functional spaces. We define forward (§ 2.1) and backward (§ 2.2) FDPs, and consider generic functional perturbations, including noising and Laplacian blurring. Using an extension of Girsanov theorem, we derive in § 2.3 an ELBO, which allows defining a parametric model to approximate the score of the functional density of FDPs. Given a FDP and the associated ELBO, we are one-step closer to the definition of a loss function to learn the parametric score. However, our formulation still resides in an abstract, infinite-dimensional Hilbert space.

Then, for practical reasons, in § 3, we specify for which subclass of functions we can perfectly reconstruct the original function given only its evaluation in a countable set of points. This is an extension of the sampling theorem, which we use to move from the infinite-dimensional domain of functions to a finite-dimensional domain of discrete mappings.

In § 4, we discuss various options to implement such discrete mappings. In this work, we explore in particular the usage of implicit neural representations (INRs) (Sitzmann et al., 2020) and Transformers Vaswani et al. (2017) to jointly model both the sampled version of infinite-dimensional functions, and the score network, which is central to the training of FDPs, and is required to simulate the backward process. Our training procedure, discussed in § 5, involves approximate, finite-dimensional Stochastic Differential Equations (SDEs) for the forward and backward processes, as well as for the ELBO.

We complement our theory with a series of experiments to illustrate the viability of FDPs, in § 6. In our experiments, the score network is a simple multilayer perceptron (MLP), with several orders of magnitude fewer parameters than any existing score-based diffusion model. To the best of our knowledge, we are the first to show that a functional-space diffusion model can generate realistic image data, beyond simple data-sets and toy models.

2 Functional Diffusion Processes (FDPs)

We begin by defining diffusion processes in Hilbert Spaces, which we call functional diffusion processes (FDPs). While the study of diffusion processes in Hilbert spaces is not new (F¨ ollmer & Wakolbinger, 1986; Millet et al., 1989; Da Prato & Zabczyk, 2014), our objectives depart from prior work, and call for an appropriate treatment of the intricacies of FDPs, when used in the context of generative modeling. In § 2.1 we introduce a generic class of diffusion processes in Hilbert spaces. The key object is Equation (1), together with its associated path measure Q and the time varying

measure ρ t , where ρ 0 represents the starting (data) measure. In § 2.2 we derive the reverse FDP with the associated path-reversed measure ˆ Q , and in § 2.3 we use an extension of Girsanov theorem for infinite-dimensional SDEs to compute the ELBO. The ELBO is a training objective involving a generalization of the score function (Song et al., 2021).

2.1 The forward diffusion process

Weconsider H to be a real, separable Hilbert space with inner product ⟨· , ·⟩ , norm ∥·∥ H , and countable orthonormal basis { e k } ∞ k =1 . Let L ( H ) be the set of bounded linear operators on H , B ( H ) be its Borel σ -algebra, B b ( H ) be the set of bounded B ( H ) -measurable functions H → R , and P ( H ) be the set of probability measures on ( H,B ( H )) . Consider the following H -valued SDE:

where t ∈ [0 , T ] , W t is a R -Wiener process on H defined on the quadruplet (Ω , F , ( F t ) t ≥ 0 , Q ) , and Ω , F are the sample space and canonical filtration, respectively. The domain of f is D ( f ) ∈ B ([0 , T ] × H ) , where f : D ( f ) ⊂ [0 , T ] × H → H is a measurable map. The operator A : D ( A ) ⊂ H → H is the infinitesimal generator of a C 0 -semigroup exp( t A ) in H ( t ≥ 0) , and ρ 0 is a probability measure in H . We consider Ω to be C 1 ([0 , T ]) , that is the space of all continuous mappings [0 , T ] → H , and X t ( ω ) = ω ( t ) , ω ∈ Ω to be the canonical process. The requirements on the terms A , f that ensure existence of solutions to Equation (1) depend on the type of noise trace-class ( Tr { R } < ∞ ) or cylindrical ( R = I ) - used in the FDP (Da Prato & Zabczyk (2014), Hypothesis 7.1 or Hypothesis 7.2 for trace-class and cylindrical noise, respectively).

̸

The measure associated with Equation (1) is indicated with Q . The law induced at time τ ∈ [0 , T ] by the canonical process on the measure Q is indicated with ρ τ ∈ P ( H ) , where ρ τ ( S ) = Q ( { ω ∈ Ω : X τ ( ω ) ∈ S } ) , and S is any element of F . Notice, that in infinite dimensional spaces there is not an equivalent of the Lebesgue measure to get densities from measures. In our case we consider however, when it exists, the single dimensional density ρ ( d ) τ ( x i | x j = i ) , defined implicitly through d ρ τ ( x i | x j = i ) = ρ ( d ) τ ( x i | x j = i )d x i , being d x i the Lebesgue measure. To avoid cluttering the notation, in this work we simply shorten ρ ( d ) τ ( x i | x j = i ) with ρ ( d ) τ ( x ) whenever unambiguous. In Appendix B we provide additional details on the time-varying measure ρ t (d x )d t . Before proceeding, it is useful to notice that Equation (1) can also be expressed as an (infinite) system of stochastic differential equations in terms of X k t = ⟨ X t , e k ⟩ as:

̸

where we introduced the projection b k ( X t , t ) = ⟨A X t + f ( X t , t ) , e k ⟩ . Moreover, d W k t = ⟨ dW t , e k ⟩ with covariance given by E [ W k t W j s ] = δ ( k -j ) r k min( s, t ) , δ in Kroenecker sense, and r k is the projection on the base of R .

2.2 The reverse diffusion process

We now derive the reverse time dynamics for FDPs of the form defined in Equation (1). We require that the time reversal of the canonical process, ˆ X t = X T -t , is again a diffusion process, with distribution given by the path-reversed measure ˆ Q ( ω ) , along with the reversed filtration ˆ F . Note that the time reversal of an infinite dimensional process is more involved than for the finite dimensional case (Anderson, 1982; F¨ ollmer, 1985). There are two major approaches to guarantee the existence of the reverse diffusion process. The first approach (F¨ ollmer & Wakolbinger, 1986) is applicable only when R = I (the case of cylindrical Wiener processes) and it relies on a finite local entropy condition. The second approach, which is valid in the case of trace class noise Tr { R } < ∞ , is based on stochastic calculus of variations (Millet et al., 1989). The full technical analysis of the necessary assumptions for the two approaches is involved, and we postpone formal details to Appendix A.

̸

Theorem 1. Consider Equation (1) . If R = I , suppose Assumption 1 in Appendix A.1 holds; else, ( R = I ) suppose Assumption 5 annd Assumption 6 in Appendix A.2 hold. Then ˆ X t , corresponding to the path measure ˆ Q ( ω ) , has the following SDE representation:

̸

̸

̸

where ˆ W is a ˆ Q R -Wiener process, and the notation D x log ρ ( d ) t ( x ) stands for the mapping H → H that, when projected, satisfies ⟨ D x log ρ ( d ) t ( x ) , e k ⟩ = ∂ ∂x k log ( ρ ( d ) t ( x k | x i = k ) ) .

By projecting onto the eigenbasis, we have an infinite system of SDE s:

The methodology proposed in this work requires to operate on proper Wiener processes, with Tr { R } < ∞ , which implies, intuitively, that the considered noise has finite variance. We now discuss a Corollary, in which Assumption 5 is replaced by stricter conditions, that we use to check the validity of the practical implementation of FDPs.

Corollary 1. Suppose Assumption 6 from Appendix A.2 holds. Assume that i) Tr { R } = ∑ i r i < ∞ , ii) b i ( x, t ) = b i x i , ∀ i , i.e. the drift term is linear and only depends on x through its projection onto the corresponding basis and iii) the drift is bounded, such that ∃ K > 0 : -K < b i < 0 , ∀ i . Then, the reverse process evolves according to Equation (4) .

Theorem 1 stipulates that, given some conditions, the reverse time dynamics for FDPs of the form defined in Equation (1) exist. Our analysis provides theoretical grounding to the observations in concurrent work (Lim et al., 2023) where, empirically, it is observed that the cylindrical class of noise is not suitable. We argue that, when R = I , the difficulty stems from designing the coefficients b i of the SDEs such that the forward (see requirement (5.3) in Da Prato & Zabczyk (2014)) as well as the backward processes Assumption 1 exist. The work by Bond-Taylor & Willcocks (2023) uses cylindrical (white) noise, but we are not aware of any theoretical justification, since the model architecture is only partially suited for the functional domain.

As an addendum, we note that the advantages of projecting the forward and backward processes on the eignenbasis of the Hilbert space H , as in Equation (2) and Equation (4), become evident when discussing about the implementation of FDPs, specifically when we derive practical expressions for training and the simulation of the backward process, as discussed in § 5, and in a fully expanded toy example in Appendix D.

2.3 A Girsanov formula for the ELBO

Direct simulation of the backward FDP described by Equation (3) is not possible. Indeed, we have no access to the true score of the density ρ ( d ) τ induced at time τ ∈ [0 , T ] . To solve the problem, we introduce a parametric score function s θ : H × [0 , T ] × R m → H . We consider the dynamics:

with path measure ˆ P χ T , and d ˜ W t being a ˆ P χ T R -Wiener process. To emphasize the connection between Equation (3) and Equation (5), we define initial conditions with the subscript T , instead of 0. In principle, we should have χ T = ρ T , as it will be evident from the ELBO in Equation (8). However, ρ T has a simple and easy-to-sample-from distribution only for T → ∞ , which is not compatible with a realistic implementation. The analysis of the discrepancy when T is finite is outside of the scope of this work, and the interested reader can refer to Franzese et al. (2022) for an analysis on standard diffusion models. The final measure of the new process at time T is indicated by χ 0 , i.e. χ 0 ( S ) = ˆ P χ T ( { ω ∈ Ω : ˆ X T ( ω ) ∈ S } ) .

Next, we quantify the discrepancy between χ 0 and the true data measure ρ 0 through an ELBO. Thanks to an extension of Girsanov theorem to infinite dimensional SDEs (Da Prato & Zabczyk, 2014), it is possible to relate the path measures ( ˆ Q and ˆ P χ T , respectively) of the process ˆ X t induced by different drift terms in Equation (3) and different initial conditions.

Starting from the score function s θ , we define:

Under loose regularity assumptions (see Condition 2 in Appendix A.4) ˜ W t = ˆ W t -t ∫ 0 γ θ ( X s , s ) ds is a ˆ P ρ T R -Wiener process (Theorem 10.14 in Da Prato & Zabczyk (2014)), where Girsanov Theorem also tells us that the measure ˆ P ρ T satisfies the Radon-Nikodym derivative:

By virtue of the disintegration theorem, d ˆ Q = d ˆ Q 0 d ρ T and similarly d ˆ P ρ T = d ˆ P 0 d ρ T , being ˆ Q 0 , ˆ P 0 the measures of the processes when considering a particular initial value. Then, ˆ P χ T satisfies d ˆ P χ T = d ˆ P d χ T d ρ T , for any measure χ T ∈ P ( H ) . Consequently, the canonical process ˆ X t has an SDE representation according to Equation (5), under the new path measure ˆ P χ T . Then (see Appendix A.5 for the derivation) we obtain the ELBO:

Provided that the required assumptions in Theorem 1 are met, the validity of Equation (8) is general. Our goal, however, is to set the stage for a practical implementation of FDPs, which calls for design choices that easily enable satisfying the required assumptions for the theory to hold. Then, for the remainder of the paper, we consider the particular case where f = 0 in Equation (1). This simplifies the dynamics as follows:

Since the only drift component in Equation (9) is the linear term A , the projection b j will be linear as well. Such a design choice, although not necessary from a theoretical point of view, carries several advantages. The design of a drift term satisfying the conditions of Corollary 1 becomes straightforward, where such conditions naturally aligns with the requirements of the existence of the forward process (Chapter 5 of Da Prato & Zabczyk (2014)). Moreover, the forward process conditioned on given initial conditions admits known solutions, which means that simulation of SDE paths is cheap and straightforward, without the need for performing full numerical integration. Finally, it is possible to claim existence of the true score function and even provide its analytic expression (full derivation in Appendix A.7) as:

where S ( t ) = ( t ∫ s =0 exp(( t -s ) A ) R exp ( ( t -s ) A † ) d s ) . This last aspect is particularly useful

̸

when considering the conditional version of Equation (6), through ⟨ D x log ρ ( d ) t ( x | x 0 ) , e k ⟩ = ∂ ∂x k log ( ρ ( d ) t ( x k | x i = k , x 0 ) ) , as:

where, similarly to the unconditional case, we have D x log ρ ( d ) t ( x | x 0 ) = -S ( t ) -1 ( x -exp( t A ) x 0 ) . Then, Equation (12) can be used to rewrite Equation (8):

̸

where I is a quantity independent of θ . Knowledge of the conditional true score D x log ρ ( d ) t ( x | x 0 ) and cheap simulation of the forward dynamics, allows for easier numerical optimization than the more general case of f = 0 .

3 Sampling theorem for FDPs

The theory of FDPs developed so far is valid for real, separable Hilbert spaces. Our goal now is to specify for which subclass of functions it is possible to perfectly reconstruct the original function given only its evaluation in a countable set of points. We present a generalization of the sampling theorem (Shannon, 1949), which allows us to move from generic Hilbert spaces to a domain which is amenable to a practical implementation of FDPs, and their application to common functional representation of data such as images, data on manifolds, and more. We model these functions as objects belonging to the set of square integrable functions over C ∞ homogeneous manifolds M (such as R N , S N , etc...), i.e., the Hilbert space H = L 2 ( M ) . Then, exact reconstruction implies that all the relevant information about the considered functions is contained in the set of sampled points.

First, we define functions that are band-limited :

Definition 1. A function x in H = L 2 ( M ) is a spectral entire function of exponential type ν (SEν ) if | ∆ k 2 x | ≤ ν k | x | , k ∈ N . Informally, the 'Fourier Transform' of x is contained in the interval [0 , ν ] (Pesenson, 2000).

Second, we define grids that cover the manifold with balls, without too much overlap. Those grids will be used to collect the function samples. Their formal definition is as follows:

̸

Definition 2. Y ( r, λ ) denotes the set of all sets of points Z = { p i } such that: i) inf j = i dist ( p j , p i ) > 0 and ii) balls B ( p i , λ ) form a cover of M with multiplicity < r .

Combining the two definitions, we can state the key result of this Section. As long as the sampled function is band-limited, if the samples grid is sufficiently fine, exact reconstruction is possible:

Theorem 2. For any set Z ∈ Y ( r, λ ) , any SEν function x is uniquely determined by its set of values in Z (i.e. { x [ p i ] } ) as long as λ < d , that is

where m p i : M → H are known polynomials 2 , and the notation x [ p ] indicates that the function x is evaluated at point p .

Aprecise definition of the value of the constant d and its interpretation is outside the goal of this work, and we refer the interested reader to Pesenson (2000) for additional details. For our purposes, it is sufficient to interpret the condition in Theorem 2 as a generalization of the classical Shannon-Nyquist sampling theorem (Shannon, 1949). Under this light, Theorem 2 has practical relevance, because it gives the conditions for which the sampled version of functions contains all the information of the original functions. Indeed, given the set of points p i on which function x is evaluated, it is possible to reconstruct exactly x [ p ] for arbitrary p .

The uncertainty principle. It is not always possible to define Hilbert spaces of square integrable functions that are simultaneously homogeneous and separable, for all the manifolds M of interest. In other words, it is difficult in practice to satisfy both the requirements for FDPs to exist, and for the sampling theorem to be valid (see an example in Appendix C). Nevertheless, it is possible to quantify the reconstruction error, and realize that practical applications of FDPs are viable. Indeed, given a compactly supported function x , and a set of points Z with finite cardinality, we can upper-bound the ∥ ∥

̸

where x ν is the SEν bandlimited version of x , obtained by filtering out - in the frequency domain any component larger than ν . The error ϵ 1 is due to x = x ν . The term ϵ 2 is the reconstruction error

due to finiteness of | Z | : the sampling theorem applies to x ν , but the corresponding sampling grid has infinite cardinality. Finally, the term ϵ 3 quantifies the energy omitted by filtering out the frequency components of x ν larger than ν . This (loose) upper bound allows us to understand quantitatively the degree to which the sampling theorem does not apply for the cases of interest. Although deriving tighter bounds is possible, this is outside the scope of this work. What suffices is that in many practical cases, when functions are obtained from natural sources, it has been observed that functions are nearly time and bandwidth limited (Slepian, 1983). Consequently, as long as the sampling grid is sufficiently fine, the reconstruction error ϵ is negligible.

We now hold all the ingredients to formulate generative functional diffusion models using the Hilbert space formalism and implement them using a finite grid of points, which is what we do next.

4 Score Network Architectural Implementations

We are now equipped with the ELBO (Equation (8)) and a score function s θ that implements the mapping H × [0 , T ] × R m → H . We could then train the score by optimizing the ELBO and produce samples arbitrary close to the true data measure ρ 0 . However, since the domain of the score function is the infinite-dimensional Hilbert space, such a mapping cannot be implemented in practice. Indeed, having access to samples of functions on finite grid of points is, in general, not sufficient. However, when the conditions for Theorem 2 hold, we can substitute - with no information loss x ∈ H with its collection of samples { x [ p i ] , p i } . This allows considering score network architectures that receive as input a collection of points, and not abstract functions. Such architectures should be flexible enough to work with an arbitrary number of input samples at arbitrary grid points, and produce as outputs functions in H .

4.1 Implicit Neural Representation

The first approach we consider in this work is based on the idea of Implicit Neural Representations (INRs) (Sitzmann et al., 2020). These architectures can receive as inputs functions sampled at arbitrary, possibly irregular points, and produce output functions evaluated at any desired point. Unfortunately, the encoding of the inputs is not as straightforward as in the Neural Fourier Operator (NFO) case, and some form of autoencoding is necessary. Note, however, that in traditional score-based diffusion models (Song et al., 2021), the parametric score function can be thought of as a denoising autoencoder. This is a valid interpretation also in our case, as it is evident by observing the term E [ X 0 | X t = x ] of the true score function in Equation (11). Since INRs are powerful denoisers (Kim et al., 2022a), combined with their simple design and small number of parameters, in this Section we discuss how to implement the score network of FDPs using INRs.

We define a valid INR as a parametric family ( ψ , t, θ ) of functions in H , i.e., mappings R m × [0 , T ] × R m → H . A valid INR is the central building block for the implementation of the parametric score function, and it relies on two sets of parameters: θ , which are the parameters of the score function that we optimize according to Equation (8), and ψ , which serve the purpose of building a mapping from H into a finite dimensional space. More formally:

Definition 3. Given a manifold M , a valid Implicit Neural Representation ( INR ) is an element of H defined by a family of parametric mappings n ( ψ , t, θ ) , with t ∈ [0 , T ] , θ , ψ ∈ R m . That is, for p ∈ M , we have n ( ψ , t, θ )[ p ] ∈ R . Moreover, we require n ( ψ , t, θ ) ∈ L 2 ( M ) .

A valid INR as defined in Definition 3 is not sufficiently flexible to implement the parametric score function s θ , as it cannot accept input elements from the infinite-dimensional Hilbert space H : indeed, the score function is defined as a mapping over H × [0 , T ] × R m → H , whereas the valid INR is a mapping defined over R m × [0 , T ] × R m → H . Then, we use the second set of parameters ψ to condensate all the information of a generic x ∈ H into a finite-dimensional vector. When the conditions for Theorem 2 hold, we can substitute - with no information loss x ∈ H with its collection of samples { x [ p i ] , p i } . Then, we can construct an implicitly defined mapping g : H × [0 , T ] × R m → R m as:

In this work, we consider the modulation approach to INRs. The set of parameters ψ are obtained by minimizing Equation (16) using few steps of gradient descent on the objective

∑ p i ( n ( ψ , t, θ )[ p i ] -x [ p i ]) 2 , starting from the zero initialization of ψ . This approach, also explored by Dupont et al. (2022b), is based on the concept of meta-learning (Finn et al., 2017). In summary, our method constructs mappings H × [0 , T ] × R m → H , where the same INR is used first to encode x into ψ , and subsequently to output the value functions for any desired input point p , thus implementing the following score network:

4.2 Transformers

As an alternative approach, we consider implementing the score function s θ using transformer architectures Vaswani et al. (2017), by interpreting them as mappings between Hilbert spaces (Cao, 2021). We briefly summarize here such a perspective, focusing on a single attention layer for simplicity, and adapt the notation used throughout the paper accordingly.

Consider the space L 2 ( M ) , with the usual collection of samples { x [ p i ] , p i } . As a first step, both the ' features ' { x [ p i ] } and positions { p i } are embedded into some higher dimensional space and summed together, to obtain a sequence of vectors { y i } . Then, three different (learnable) matrices θ ( Q ) , θ ( K ) , θ ( V ) are used to construct the linear transformations of the vector sequence { y i } as ˆ Y ( Q ) = { ˆ y i ( Q ) = θ ( Q ) y i } , ˆ Y ( K ) = { ˆ y i ( K ) = θ ( K ) y i } , ˆ Y ( V ) = { ˆ y i ( V ) = θ ( V ) y i } . Finally, the three matrices ˆ Y ( Q,K,V ) are multiplied together, according to any variant of the attention mechanism. Indeed, different choices for the order of multiplication and normalization schemes in the products and in the matrices correspond to different attention layers Vaswani et al. (2017). In practical implementations, these operations can be repeated multiple times (multiple attention layers) and can be done in parallel according to multiple projection matrices (multiple heads).

The perspective explored in (Cao, 2021) is that it is possible to interpret the sequences ˆ y i ( Q,K,V ) as learnable basis functions in some underlying latent Hilbert space, evaluated at the set of coordinates { p i } . Furthermore, depending on the type of attention mechanism selected, the operation can be interpreted as a different mapping between Hilbert spaces, such as Fredholm equations of the second-kind or Petrov-Galerkin-type projections (Cao (2021) Eqs. 9 and 14).

While a complete treatment of such an interpretation is outside the scope of this work, what suffices is that it is possible to claim that transformer architectures are a viable candidate for the implementation of the desired mapping H × [0 , T ] × R m → H , a possibility that we explore experimentally in this work. It is worth noticing that, compared to the approach based on INRs, resolution invariance is only learned , and not guaranteed, and that the number of parameters is generally higher compared to an INR. Nevertheless, learning the parameters of transformer architectures does not require meta-learning, which is a practical pain-point of INRs used in our context. Additional details for the transformer-based implementation of the score network are available in Appendix E.

Finally, for completeness, it is worth mentioning that a related class of architectures, the Neural Operators and NFOs (Kovachki et al., 2021; Li et al., 2020), are also valid alternatives. However, such architectures require the input grid to be regularly spaced (Li et al., 2020), and their output function is available only at the same points p i of the input, which would reduce the flexibility of FDPs.

5 Training and sampling of FDPs

Given the parametric score function s θ from Equation (17), by simulating the reverse FDP, we generate samples whose statistical measure χ 0 is close in KL sense to ρ 0 . Next, we explain how to numerically compute of the quantities in Equation (13), which is part of the ELBO in Equation (8), and how to generate new samples from the trained FDP (simulation of Equation (10)).

ELBO Computation. Equation (8) involves Equation (13), which requires the computation of the Hilbert space norm. The grid of points x [ p i ] is interpolated in H as ∑ i x [ p i ] ξ i . Then, the norm of interest can be computed as:

Depending on the choice of ξ i , e i , the sum w.r.t the index k is either naturally truncated or it needs to be further approximated by selecting a cutoff index value. Finally, training can then be performed by minimizing:

Numerical integration. Simulation of infinite dimensional SDEs is a well studied domain (Debussche, 2011), including finite difference schemes (Gy¨ ongy, 1998, 1999; Yoo, 2000), finite element methods and/or Galerkin schemes (Hausenblas, 2003a,b; Shardlow, 1999). In this work, we adopt a finite element approximate scheme, and introduce the interpolation operator, from R | Z | to H , i.e. ∑ i x [ p i ] ξ i (Hausenblas, 2003b). Notice that, in general, the functions ξ i differ from the basis e i . In addition, the projection operator maps functions from H into R L , as ⟨ x, ζ j ⟩ , ζ j ∈ H . Usually, L = | Z | . When ζ i = ξ i the scheme is referred to as the Galerkin scheme. We consider instead a point matching scheme (Hausenblas, 2003b), in which ζ i = δ [ p -p i ] with δ in Dirac sense, and consequently ⟨ x, ζ i ⟩ = x [ p i ] . Then, the infinite dimensional SDE of the forward process from Equation (9) is approximated by the finite ( | Z | ) dimensional SDE:

Similarly, the reverse process described by Equation (10) corresponds to the following SDE:

Equation (21) is a finite dimensional SDE, and consequently we can use any known numerical integrator to simulate its paths. In Appendix D we provide a complete toy example to illustrate our approach in a simple scenario, where we emphasize practical choices.

6 Experiments

Despite a rather involved theoretical treatment, the implementation of FDPs is simple. We implemented our approach in JAX (Bradbury et al., 2018), and use WANDB (Biewald, 2020) for our experimental protocol. Additional details on implementation, and experimental setup, as well as more experiments are available in Appendix E.

We evaluate our approach on image data, using the CELEBA 64 × 64 (Liu et al., 2015) dataset. Our comparative analysis with the state-of-the-art includes generative quality, using the FID score (Heusel et al., 2017), and parameter count for the score network. We also discuss (informally) the complexity of the network architecture, as a measure of the engineering effort in exploring the design space of the score network. We compare against vanilla Score Based Diffusion (SBD) (Song et al., 2021), From Data To Functa (FD2F) (Dupont et al., 2022a) which diffuses latent variables obtained from an INR, Infinite Diffusion ( ∞ -DIFF) (Bond-Taylor & Willcocks, 2023), which is a recent approach that is only partially suited for the functional domain, as it relies on the combination of Fourier Neural Operators and a classical convolutional U-NET backbone. Our FDP method is implemented using either MLP or Transformers. In the first case, we consider a score network implemented as a simple MLP with 15 layers and 256 neurons in each layer. The activation function is a Gabor wavelet activation function (Saragadam et al., 2023). In the latter case, our approach is built upon the UViT backbone as detailed by Bao et al. (2022). The architecture comprises 7 layers, with each layer composed of a self-attention mechanism with 8 attention heads and a feedforward layer.

We present quantitative results in Table 1, showing that our method FDP(MLP) achieves an impressively low FID score, given the extremely low parameter count, and the simplicity of the architecture. FD2F obtains a worse (larger) FID score, while having many more parameters, due to the complex

parametrization of their score network. As a reference we report the results of SBD, where the price to be pay to achieve an extremely low FID is to have many more parameters and a much more intricate architecture. Finally, the very recent ∞ -DIFF method, has low FID-CLIP score (Kynk¨ a¨ anniemi et al., 2022), but requires a very complex architecture and more than 2 orders of magnitude more parameters than our approach. Showcasing the flexibility of the proposed methodology, we consider a more complex architecture based on Vision Transformers ( FDP(UViT) ). These corresponding results indicate improvements in terms of image quality (FID score=11) and do not require meta-learning steps, but require more parameters (O(20M)) than the INR variant. To the best of our knowledge, none of related work in the purely functional domain (Lim et al., 2023; Hagemann et al., 2023; Dutordoir et al., 2022; Kerrigan et al., 2022) provides results going beyond simple data-sets. Finally, we present some qualitative results in Figures 1 and 2 clearly showing that the proposed methodology is capable of producing diverse and detailed images.

29

7 Conclusion, Limitations and Broader Impact

We presented a theoretical framework to define functional diffusion processes for generative modeling. FDPs generalize traditional score-based diffusion models to infinite-dimensional function spaces, and in this context we were the first to provide a full characterization of forward and backward dynamics, together with a formal derivation of an ELBO that allowed the estimation of the parametric score function driving the reverse dynamics.

To use FDPs in practice, we carefully studied for which subset of functions it was possible to operate on a countable set of samples without losing information. We then proceeded to introduce a series of methods to jointly model - using only a simple INR or a Transformer - an approximate functional representation of data on discrete grids, and an approximate score function. Additionally, we detailed practical training procedures of FDPs, and integration schemes to generate new samples.

The implementation of FDPs for generative modeling was simple. We validated the viability of FDPs through a series of experiments on real images, where we show, while only using a simple MLP for learning the score network, extremely promising results in terms of generation quality.

Like other works in the literature, the proposed method can have both positive (e.g., synthesizing new data automatically) and negative (e.g., deep fakes) impacts on society depending on the application.

8 Acknowledgments

GF gratefully acknowledges support from Huawei Paris and the European Commission (ADROIT6G Grant agreement ID: 101095363). MF gratefully acknowledges support from the AXA Research Fund and the Agence Nationale de la Recherche (grant ANR-18-CE46-0002 and ANR-19-P3IA-0002).

References

Brian D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982.

Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392 , 2022.

Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: a vit backbone for score-based diffusion models. arXiv preprint arXiv:2209.12152 , 2022.

Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb. com/ . Software available from wandb.com.

Vladimir Bogachev, Giuseppe Da Prato, and Michael R¨ ockner. Existence and uniqueness of solutions for fokker-planck equations on hilbert spaces. J. Evol. Equ. , 10, 07 2009.

Vladimir Bogachev, Giuseppe Da Prato, and Michael R¨ ockner. Uniqueness for solutions of fokkerplanck equations on infinite dimensional spaces. Communications in Partial Differential Equations , 36(6):925-939, 2011.

Sam Bond-Taylor and Chris G Willcocks. ∞ -diff: Infinite resolution diffusion with subsampled mollified states. arXiv preprint arXiv:2303.18242 , 2023.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: composable transformations of python+ numpy programs. 5:14-24, 2018.

Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems , 34:24924-24940, 2021.

Giuseppe Da Prato and Jerzy Zabczyk. Stochastic equations in infinite dimensions . Cambridge university press, 2014.

Arnaud Debussche. Weak approximation of stochastic partial differential equations: the nonlinear case. Mathematics of Computation , 80(273):89-117, 2011.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems , volume 34, pp. 8780-8794. Curran Associates, Inc., 2021.

Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with criticallydamped langevin diffusion. In International Conference on Learning Representations , 2022.

Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pp. 5694-5725. PMLR, 17-23 Jul 2022a.

Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Teh, and Arnaud Doucet. COIN++: Neural compression across modalities. Transactions of Machine Learning Research , 2022b.

Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes. arXiv preprint arXiv:2206.03992 , 2022.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pp. 1126-1135. PMLR, 2017.

Hans F¨ ollmer. An entropy approach to the time reversal of diffusion processes. In Stochastic Differential Systems Filtering and Control , pp. 156-163. Springer, 1985.

Hans F¨ ollmer. Time reversal on wiener space. In Stochastic processes-mathematics and physics , pp. 119-129. Springer, 1986.

Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filippone, and Pietro Michiardi. How much is enough? a study on diffusion times in score-based generative models. arXiv preprint arXiv:2206.05173 , 2022.

F¨ ollmer and A. Wakolbinger. Time reversal of infinite-dimensional diffusions. Stochastic Processes and their Applications , 22(1):59-77, 1986. ISSN 0304-4149. doi: https://doi.org/10. 1016/0304-4149(86)90114-6. URL https://www.sciencedirect.com/science/article/ pii/0304414986901146 .

Istv´ an Gy¨ ongy. Lattice approximations for stochastic quasi-linear parabolic partial differential equations driven by space-time white noise i. Potential Analysis , 9(1):1-25, 1998.

Istv´ an Gy¨ ongy. Lattice approximations for stochastic quasi-linear parabolic partial differential equations driven by space-time white noise ii. Potential Analysis , 11(1):1-37, 1999.

Paul Hagemann, Sophie Mildenberger, Lars Ruthotto, Gabriele Steidl, and Nicole Tianjiao Yang. Multilevel diffusion: Infinite dimensional score-based diffusion models for image generation, 2023.

Erika Hausenblas. Approximation for semilinear stochastic evolution equations. Potential Analysis , 18(2):141-186, 2003a.

Erika Hausenblas. Weak approximation for semilinear stochastic evolution equations. In Stochastic analysis and related topics VIII , pp. 111-128. Springer, 2003b.

Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221 , 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, pp. 6629-6640, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 6840-6851. Curran Associates, Inc., 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation, 2021. URL https://arxiv.org/ abs/2106.15282 .

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022. URL https://arxiv.org/abs/2210. 02303 .

Emiel Hoogeboom, V´ ıctor Garcia Satorras, Cl´ ement Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3D. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pp. 8867-8887. PMLR, 17-23 Jul 2022.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusionbased generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems , 2022.

Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models in infinite dimensions, 2022. URL https://arxiv.org/abs/2212.00886 .

Chaewon Kim, Jaeho Lee, and Jinwoo Shin. Zero-shot blind image denoising via implicit neural representations. arXiv preprint arXiv:2204.02405 , 2022a.

Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pp. 11201-11228. PMLR, 17-23 Jul 2022b.

Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=2LdBqxc1Yv .

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations , 2021.

Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481 , 2021.

Tuomas Kynk¨ a¨ anniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr´ echet inception distance. arXiv preprint arXiv:2203.06026 , 2022.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 , 2020.

Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, et al. Score-based diffusion models in function space. arXiv preprint arXiv:2302.07400 , 2023.

Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. Proceedings of the AAAI Conference on Artificial Intelligence , 36(10):11020-11028, Jun. 2022.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2837-2845, June 2021.

Annie Millet, David Nualart, and Marta Sanz. Time reversal for infinite-dimensional diffusions. Probability theory and related fields , 82(3):315-347, 1989.

Sarthak Mittal, Guillaume Lajoie, Stefan Bauer, and Arash Mehrjou. From points to functions: Infinite-dimensional representations in diffusion models. In ICLR Workshop on Deep Generative Models for Highly Structured Data , 2022.

Isaac Pesenson. A sampling theorem on homogeneous manifolds. Transactions of the American Mathematical Society , 352(9):4257-4269, 2000.

Angus Phillips, Thomas Seror, Michael Hutchinson, Valentin De Bortoli, Arnaud Doucet, and Emile Mathieu. Spectral diffusion processes. arXiv preprint arXiv:2209.14125 , 2022.

Jakiw Pidstrigach, Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinite-dimensional diffusion models for function spaces. arXiv preprint arXiv: 2302.10130 , 2023.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical textconditional image generation with clip latents, 2022. URL https://arxiv.org/abs/2204. 06125 .

Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. arXiv preprint arXiv:2206.13397 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 10684-10695, June 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems , 2022.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations , 2022.

Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. arXiv preprint arXiv:2301.05187 , 2023.

C.E. Shannon. Communication in the presence of noise. Proceedings of the IRE , 37(1):10-21, 1949.

Tony Shardlow. Numerical methods for stochastic parabolic pdes. Numerical functional analysis and optimization , 20(1-2):121-145, 1999.

Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 7462-7473. Curran Associates, Inc., 2020.

David Slepian. Some comments on fourier analysis, uncertainty and modeling. SIAM review , 25(3): 379-393, 1983.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alch´ e-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 12438-12448. Curran Associates, Inc., 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motifscaffolding problem, 2022. URL https://arxiv.org/abs/2206.04119 .

Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation , 23(7):1661-1674, 2011.

Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations (ICLR) , 2022.

Hyek Yoo. Semi-discretization of stochastic partial differential equations on r by a finite-difference method. Mathematics of computation , 69(230):653-666, 2000.

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems , 33:18795-18806, 2020.

Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M. Susskind, and Miguel ´ Angel Bautista. Diffusion probabilistic fields. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=ik91mY-2GN .

Supplementary Material: Continuous-Time Functional Diffusion Processes

A Reverse Functional Diffusion Processes

̸

In this Section, we review the mathematical details to obtain the backward FDP discussed in Theorem 1. Depending on the considered class of noise, different approaches are needed. First, we present in Appendix A.1 the conditions to ensure existence of the backward process , which we use if the C operator is an identity matrix, C = I . Then we move to a different approach in Appendix A.2 for the case C = I .

A.1 Follmer Formulation

The work in F¨ ollmer (1986) is based on a finite entropy condition, which we report here as Condition 1. One simple way to ensure that the condition is satisfied is to assume:

Condition 1. For a given k , define Q ( k ) to be the path measure corresponding to the (infinite) system

̸

We say that Q satisfies the finite local entropy condition if KL [ Q ∥ Q ( k ) ] < ∞ , ∀ k .

̸

Define F ( i ) t = σ ( X i 0 , X j s , 0 ≤ s ≤ t, j = i ) .

Assumption 1.

̸

Notice that if Assumption 1 is true, then Condition 1 holds (F¨ ollmer (1986), Thm. 2.23)

Proof. The proof can be obtained by adapting the result of Lemma 3.6 of F¨ ollmer & Wakolbinger (1986).

This Theorem states that if the forward FDP path measure Q satisfies the finite local entropy condition, then also the reverse FDP path measure ˆ Q satisfies the finite local entropy condition.

Theorem 4. Let Q be a finite entropy measure. Then:

̸

Proof. For the proof, we refer to Theorem 3.14 of F¨ ollmer & Wakolbinger (1986).

A.2 Millet Formulation

Let L 2 ( R ) = { x ∈ H : ∑ r i ( x i ) 2 < ∞} . For simplicity, we overload the notation of the letter K , and use it for generic constants, that might be different on a case by case basis.

Assumption 2.

where:

This assumption is simply the translation of H1 from Millet et al. (1989) to our notation.

Assumption 3. There exists an increasing sequence of finite subsets J ( n ) , n ∈ N , ∪ n J ( n ) = N such that ∀ n ∈ N , M > 0 there exists a constant K ( M,n ) such that the following holds:

Again, this assumption is simply the translation of H5 from Millet et al. (1989) to our notation.

Assumption 4. Either i):

or ii): ∀ i, b i ( x ) is a function of x for at most M coordinates and

This corresponds to satisfying either H3 or jointly H2 and H4 of Millet et al. (1989). For simplicity, we can combine together the different assumptions into

Assumption 5. Let Assumption 2, Assumption 3, and Assumption 4 hold.

Finally, we state required assumptions about the density:

Assumption 6. Suppose that the initial condition is X 0 ∈ L 2 ( R ) .

̸

Assume that the conditional law of x i given x j , j = i has density ρ ( d ) t ( x i | x j , j = i ) w.r.t Lebesgue measure on R .

̸

̸

Assume that ∫ 1 t 0 ∫ D J | r i d d x i ( ρ ( d ) t ( x i | x j , j = i )) | d x i ρ t (d x j = i )d t < ∞ , for fixed subset J ⊂ N , t 0 > 0 and D J = { ( ∏ j ∈ J K j ) × ( ∏ j / ∈ J R ) , K j compact in R } ∩ L 2 ( R ) .

̸

We reported in our notation the content of Theorem 4.3 of Millet et al. (1989). This can be used to prove the existence of the backward process.

A.3 Proof of Theorem 1

If R = I , then we assume Assumption 1. Consequently, Q is a finite entropy measure. Then Theorem 4 holds, from which the desired result. If, instead R = I , then we require Assumption 5,Assumption 6. Application of Thm 4.3 of Millet et al. (1989) allows to prove the validity of Theorem 1 also in this case.

A.3.1 Proof of Corollary 1

Assumption 5 is required directly. We need to show that with the considered restrictions Assumption 6 is valid.

Since ∑ i r i < ∞ , then ∑ i ( r i ) 2 = K a < ∞ . Moreover, ( b i ( x i , t )) 2 < K 2 b ( x i ) 2 . Then, ∀ x ∈ L 2 ( R ) , the following holds sup t { ∑ r i ( b i ( x, t )) 2 } + ∑ ( r i ) 2 ≤ ∑ r i K 2 b ( x i ) 2 + K a ≤ max( K a , K 2 b ) ( 1 + ∑ r i ( x i ) 2 ) . Similarly, ∀ x, y ∈ L 2 ( R ) we have sup t { ∑ r i ( b i ( x, t ) -b i ( y, t )) 2 } ≤ ∑ r i K 2 b ( x i -y i ) 2 . Thus Assumption 2 is satisfied.

Since b i ( x, t ) is bounded and independent on t , Assumption 3 is satisfied, as explicitly discussed in Millet et al. (1989).

Finally, since b i ( x ) is a function of x for M = 1 coordinate, and sup t { ∑ ( r i ) 2 ( b i ( x, t ) -b i ( y, t )) 2 } ≤ ∑ ( r i ) 2 K 2 b ( x i -y i ) 2 , Assumption 4 is satisfied.

Then, combined toghether Assumption 5 holds.

̸

A.4 Girsanov Regularity

Condition 2. Assume that γ θ ( x, t ) is an ˆ F measurable process and that either:

This is equivalent to the regularity condition in eq. 10.23 of Da Prato & Zabczyk (2014) or Proposition 10.17 in Da Prato & Zabczyk (2014).

A.5 Proof of KL divergence expression

We leverage Equation (7) to express the Kullback-Leibler divergence as:

we can combine the two results and obtain Equation (8)

A.6 Conditional score matching

In this subsection we prove the equality in Equation (13):

To simplify the equality, we need to notice that:

̸

Then, computing

̸

̸

Consequently:

Combining together and rearranging the terms, we get the desired Equation (13).

A.7 Explicit expression of score function

As mentioned in the text, we consider the case f = 0 . In this case, there exists a weak solution to Equation (1) as:

Consequently, the true score function has expression:

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

where s i = r i exp ( 2 b i t ) -1 2 b i . This is exactly the desired Equation (11). Similar calculations allow to prove D x log ρ ( d ) t ( x | x 0 ) = -S ( t ) -1 ( x -exp( t A ) x 0 ) .

B Fokker Planck equation

In this Section we discuss the infinite dimensional generalization of the classical Fokker Planck equation. We can associate to Eq. (1) the differential operator:

where D t is the time derivative, D x , D 2 x are first and second order Fr´ echet derivatives in space. The domain of the operator L 0 is D ( L 0 ) , the linear span of real parts of functions u ϕ,h = ϕ ( t ) exp( i ⟨ x, h ( t ) ⟩ ) , x ∈ H,t ∈ [0 , T ] where ϕ ∈ C 1 ([0 , T ]) , ϕ ( T ) = 0 , h ∈ C 1 ([0 , T ]; D ( A † )) , where † indicates adjoint. Provided appropriate conditions are satisfied, see for example Bogachev et al. (2009, 2011), the time varying measure ρ t (d x )d t exists, is unique, and solves the Fokker-Planck equation L † 0 ρ t = 0 .

C Uncertainty principle

We here clarify that Hilbert spaces of square integrable functions that are not, in general, simultaneously homogeneous and separable. For example, while R is homogeneous, the set of square integrable functions over R is not separable, since the basis is the uncountable set cos(2 πνp ) , sin(2 πνp ) , ν ∈ R . Then, FDP requirements are not met, as we need a countable basis. Moreover, we would need in

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

general an infinite number of samples (grid over the whole R ) to reconstruct the functions. Conversely, a set like the interval I = [0 , 1] ⊂ R has countable basis cos(2 πtp ) , sin(2 πtp ) , t ∈ Z (and thus is separable) and, considering x to be band-limited, a sampling grid with finite cardinality would allow to reconstruct of the function. However, I is not homogeneous as no isometry group exists. Consequently, Theorem 2 is not applicable. To fix the issue, one could naively think of extending any function defined over I to the whole R by considering ¯ x [ p ] = x [ p ] , p ∈ I and ¯ x [ p ] = 0 , p / ∈ I . Obviously, if x ∈ L 2 ( I ) then ¯ x ∈ L 2 ( R ) . However, since ¯ x has finite support, it cannot be bandlimited, making such an approach not a viable solution. In classical signal processing literature, the problem is usually referred to as the uncertainty principle (Slepian, 1983).

D A complete example

We present an example in which we cast Equation (20) for square integrable functions over the interval I = [0 , 1] , L 2 ( I ) . In this case, one natural selection for the basis is the Fourier basis 3 e k = { . . . , exp( -j 2 π 2 p ) , exp( -j 2 πp ) , 1 , exp( j 2 πp ) , exp( j 2 π 2 p ) , . . . } . Assume the operator A to be a pseudo-differantial operator, such that ⟨A x, e k ⟩ = b k x k . Also, assume that b k , r k are selected such that conditions of Corollary 1 are met, and consequently the backward process exists. Since we are working with samples collected on the grid x [ i / N ] we first map the samples to the frequency domain, and then build a Fourier-like representation with a finite set of sinusoids. We then define the mapping F ( z i ) k def = ∑ N -1 i =0 z i exp ( -j 2 πk i N ) and its inverse I ( z i ) k def = N -1 ∑ N -1 i =0 z i exp ( j 2 πk i N ) . This suggests to consider the following expression for the interpolating functions:

Those functions are indeed nothing but a frequency truncated version of the sinc function, which is the classical reconstruction function of the sampling theorem on 1-D signals. Moreover ⟨ ξ i , ζ k ⟩ = δ ( i -k ) . We are now ready to show i) the expression of the forward process, ii) the expression of the parametric score function s θ and γ θ , iii) the computation of the ELBO and finally iv) the expression for the backward process.

The forward process defined in Equation (20) has expression:

where d W t [ k / N ] ≃ F (d W i t ) k . Simple calculations show that X t [ k / N ] is equivalent in distribution to

where s l = 〈 S ( t ) , e l 〉 = r l exp ( 2 b l t ) -1 2 b l and ϵ l ∼ N (0 , 1) , allowing simulation of the forward process in a single step. The parametric score function can be approximated as:

Similarly:

Combining Equation (31) and Equation (33) we can fully characterize the training objective defined in Equation (19). Then, it is possible to optimize the value of the parameters θ with any gradient-based optimizer.

Finally, the backward process approximation is expressed as:

from which new samples can be generated.

D.1 Proofs

We start by proving Equation (30). Starting from the drift term of Equation (20), we have the following chain of equalities:

The noise term d W t [ k / N ] is approximated as:

where we are truncating the sum. The score term has expression:

where the approximation is due to the substitution of explicit scalar product with the discretized version trough F . When evaluated on the grid of interest:

The value of ˜ γ θ , Equation (33) and the expression of the backward process, Equation (34), are obtained similarly, considering the above results.

E Implementation Details and Additional Experiments

In all experiments we use the the complex Fourier basis for the Hilbert spaces, indexed by k . This extends to the 2-dimensional case what we described in Appendix D.1. As stated in the main paper, our practical implementation sets f = 0 : then, we only need to specify the value for the parameters b k , r k . In our implementation we consider an extended class of SDEs that include time-varying multiplying coefficients in front of the drift and diffusion terms, as done for example in the Variance Preserving SDE originally described by Song & Ermon (2020). This can be simply interpreted as the time-rescaled version of autonomous SDEs.

E.1 Architectural details

INR-based score network. In our implementation, we use the original INR architecture (Sitzmann et al., 2020). For the specific denoising task we consider in our model, we extend the input of the network architecture to include the corrupted version of the input sample and the diffusion time t , in addition to the spatial coordinates. We emphasize that our architectural design is simple, and does not require self-attention mechanisms (Song & Ermon, 2020). The non-linearity we use in our network is a Gabor wavelet activation function (Saragadam et al., 2023). Furthermore, we found beneficial the inclusion of skip connections.

As stated in the main paper, we consider the modulation approach to INRs. In particular, we implement the meta-learning scheme described by Dupont et al. (2022b); Finn et al. (2017). The outer loop is dedicated to learning the base parameters of the model, while the inner loop focuses on refining the base parameters for each input sample. In the outer loop, the optimization algorithm is AdaBelief (Zhuang et al., 2020), sweeping the learning rate over 1e-4, 1e-5, 1e-6. We found the use of a cosine warm-up schedule to be beneficial for avoiding training instabilities and convergence to sub-optimal solutions. The inner loop is implemented by using three steps of stochastic gradient descent (SGD).

Transformer-based score network. In our experiments with the Transformer architecture for score modeling, we employed the UViT backbone Bao et al. (2022). This backbone processes all inputs, be they temporal or noisy image patches, as tokens. Rather than utilizing UViT's default learned positional embeddings, we adapted it to integrate 2D sinusoidal positional encodings. For the noisy input images, patch embeddings transform them into a sequence of tokens. Notably, we chose a patch size of 1 to fully harness the functional properties of our framework. Time embeddings are computed based on the time and then concatenated with the image tokens.

Our chosen transformer architecture comprises 7 layers, with each layer composed of a self-attention mechanism with 8 attention heads and a feedforward layer. Furthermore, we use long skip connections between the shallower and deeper layers, as outlined by Bao et al. (2022).

For optimization during our training, we utilized the AdamWLoshchilov & Hutter (2017) algorithm with a weight decay of 0.03. We employed a cosine warm-up schedule for the learning rate, which ends at a value of 2e-4.

1.0

1.0

0.5

0.5

0.0

F00

0.5

0.5

1.0

1.0

-1.5

-1.000.75-0.50-0.250.00

0.25

0.50

0.75

1.00

-1.000.750.50-0.250.00

0.25

0.50

0.75

1.00

1.00-0.750.500.250.00

0.25

0.50

0.75

1.00

E.2 Additional results

E.2.1 A Toy example.

We here present some qualitative examples on a synthetic data-set of functions ∈ L ([ -1 , 1]) , and therefore consider the settings described in Appendix D. The Quadratic data is generated as in (Phillips et al., 2022), i.e. X 0 [ p ] = qp 2 + ϵ , where ϵ ∼ N (0 , 0 . 1) and q is a binary random variable that take values {-1 , 1 } with equal probability. Concerning the design of the forward SDE, we select b k = min( √ k, 10) and r k = k -2 (thus satisfying Corollary 1). The real data is generated considering a grid of 100 equally spaced points. We can see in Figure 3 some qualitative results. On the left real (red) and generated through FDP (blue) samples show good agreement. Center and right plots depict some example of diffused samples for times 0.2 and 1.0 respectively.

E.2.2 MNIST data-set

We evaluate our approach on a simple data-set, using MNIST 32 × 32 (LeCun et al., 2010). In this experiment, we compare our method against the baseline score-based diffusion models from Song et al. (2021), which we take from the official code repository https://github.com/yang-song/ score_sde . The baseline implements the score network using a U-NET with self-attention and skip connections, as indicated by current best practices, which amounts to O (10 8 ) parameters.

Instead, our method uses a score-network/INR implemented as a simple MLP with 8 layers and 128 neurons in each layer. The activation function is a sinusoidal non-linearity (Sitzmann et al., 2020). Our model counts O (10 5 ) parameters. We consider an SDE with parameters r k,m = 176 k 2 + m 2 +2 , 4 and

b k,m = min(( k 2 + m 2 +0 . 3) -1 + ( r k,m 33 ) 4 , 3 . 6) . These values have been determined empirically by observing the power spectral density of the data-set, to ensure a well-behaved Signal to Noise ratio evolution throughout the diffusion process for all frequency components.

2

9

B

9

6

3

9

2

3

9

6

80

8

0

y

了

6

h

9

9

3

9

3

D

9

6

D

9

8

5

9

3

8

5

4

0

6

2

2

9

6

3

S

8

8

0

7

9

3

a

8

4

D

3

7

E

8

7

D

9

3

5

3

9

7

8

g

9

9

3

2

8

2

q

9

8

9

9

2

2

7

9

a

8

6

9

5

9

3

b

8

7

S

/

1

8

4

2

9

2

5

2

9

31

1

》

8

9

o

9

6

4

3

0

4

2

3

1

0

8

q

1

a

又

b

6

In Figure 4 we report un-curated samples generated according to our FDP. In Figure 5 we present instead various 'intermediate' noisy versions of the training data, to illustrate the kind of noise we use to train the score network, and the output of the denoising INR. We also report the Fr´ echet Inception Distance (FID) score computed using 16k samples (lower is better). For the baseline we obtain FID= 0 . 05 , whereas for the proposed method we obtain FID= 0 . 43 . Although the FID score is in favor of the baseline, we believe that our results - obtained with a simple MLP - are very promising, as further corroborated by experiments on a more complex dataset, which we show next.

Super resolution. We demonstrate how an INR trained at a 32x32 resolution on MNIST can be seamlessly applied to increase the resolution of the generated data points. Given that the INR establishes a mapping between a grid and its corresponding value, we initiated the diffusion process from a grid at a 32x32 resolution. The diffusion process continued until the final step, where we used the last learned parameters to extrapolate outputs at a higher resolution. A significant advantage of using INRs is the ability to conduct the resource-intensive sampling at a lower resolution and then effortlessly transition between resolutions with just a single forward call to the model. Figure 6, shows our results at different resolutions.

E.2.3 CELEBA data-set

For the CELEBA data-set we considered the same SDE as for the MNIST experiment. Results reported in the main paper have been obtained using a numerical integration scheme of a variant of the predictor-corrector scheme of (Song & Ermon, 2020), which we adapted to the SDEs we consider in our work. In Figure 8 and Figure 9 we report additional un-curated samples obtained with the INR and Transformer respectively. We proceed to describe further experiments in the following section.

Conditional generation. In the following, we consider three use-cases for conditional generation: in-painting, de-blurring, and colorization, which we describe next. All these additional experiments

were completed using the same architecture and configuration of the unconditional generation described above.

In-painting. We perform in-painting experiments by adopting the same approach described by Song & Ermon (2020), and report results in Figure 10. Original images (left-column of Figure 10) are masked (center-column of Figure 10), where we set the value corresponding to the missing pixels to 0. The right column of Figure 10 shows the results of the in-painting scheme where, qualitatively, it is possible to observe that the conditional generation is able to fill the missing portion of the images while maintaining good semantic coherence.

De-blurring. Our FDPs are naturally suited for the de-blurring use-case, as shown in Figure 12. In this experiment, we take the original images (left column of Figure 12) and filter them with a low pass filter (center column of Figure 12). The de-blurring scheme is implemented as the in-painting approach described by Song & Ermon (2020), where the only difference is that the masking at each update is applied in the frequency domain. The right column of Figure 12 shows that our technique gracefully recovers missing details and is capable of producing high quality images conditioned on the distorted inputs.

Colorization. In this use-case, we adapt the approach from (Song & Ermon, 2020) to our setting. Figure 11 depicts qualitative results of the colorization experiment, confirming the flexibility of the proposed scheme.

E.2.4 SPOKEN DIGIT data-set

To demonstrate the versatility of our framework, we conducted preliminary experiments on an audio dataset, specifically the Spoken Digit Dataset. This dataset comprises recordings of spoken digits, stored as wav files at an 8kHz sampling rate, with each recording trimmed to minimize silence at the beginning and end. The dataset features five speakers who have contributed to a total of 2,500 recordings, providing 50 recordings of each digit per speaker. The dataset is publicly available on the TensorFlow Dataset Catalog. As preprocessing, each sample was either padded with zeros or truncated to a maximum duration of one second. Subsequently, the data was normalized using the effects.normalize function from the pydub library, and each sample was scaled to a range of [-1, +1] by dividing it by the dataset's maximum intensity. For the audio experiments, we fed the raw audio data directly into the transformer model, without converting it to log mel spectrograms, which is a common practice in audio processing tasks. The transformer model was configured with a patch size of 2, an embedding size of 512, 13 layers, and 8 heads. We employed the AdamW optimizer with a weight decay of 0.03 and a cosine warmup schedule that decays at a value of 1e-5. The preliminary examination of the audio generated by our model reveals its ability to effectively generate spoken digits. Upon listening to the model's generated samples, we were able to recognize the digits accurately, showcasing the model's potential in audio generation tasks. Figure 7 provides a comparative analysis of the waveforms generated by our model against real examples from the dataset.

(a) Real samples from dataset

(b) Generated samples

Waveform

1.00

0.75

Amplitude

0.50

0.25

0.00

0.25

0.50

0.75

1.00

2

Time (s)

4

Waveform

1.00

0.75

Amplitude

0.50

0.25

0.00

0.25

0.50

0.75

1.00

E

Time (s)

4

6

C

7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors propose an extension to the diffusion models framework for function spaces.\nIn particular, given a Hilbert space $H$, they introduce a noising process based on a Wiener process on $H$. Depending on some assumptions on the type of noise, they then show the existence of the time-reversal process. Then they derive an ELBO via an extension of the Girsanov theorem.\nLater, akin to the the classical Shannon-Nyquist sampling theorem, they show that for a subclass of functions and grids for which the function is uniquely determined by its values on this grid.\nEventually, the process is discretised at the evaluation points, yielding finite dimensional noising and denoising processes.\nThey apply their model to model the CelebA dataset, and show that with relatively few parameters their are able to obtain low FID.\n\nSTRENGTHS:\n- I found it quite easy to follow the submission, especially Section 2 which I think is well fleshed out.\n- The proposed methodology is rigorous and sound.\n- Theorem 2 is nice and potentially useful as it gives sufficient conditions to recover the original function.\n\nWEAKNESSES:\n- Perhaps the main weakness is the limited experimental setups and lack of ablation studies. As such it is hard to really understand the practical benefits of this specific approach.\n- In particular, one advantage of working with functional data is to be able to handle data at discretised at arbitrary resolutions or on varying and irregular grid, yet this work practically focuses on the CelebA dataset. It is interesting to see the parameter efficiency by representing as the data as function, but indeed there is no hope to beat method that assume this fixed grid setting and can rely on specialised architectures (e.g. U-net). It may be of interest to tackle tasks such as upscaling, or data with missing values, irregular time-series etc.\n\nQUESTIONS:\n- line 98: Why does $R$ has to be diagonal?\n- line 105: Is it correct to interpret the 'trace-class' as the following: we have $\\mathrm{Tr}(R) = \\sum_i \\lambda_i = Var(W_t)$ that is the trace of the operator is given by the arithmetic series of its eigenvalue which itself is the variance of the $R-Wiener$ process, so this can be seen as a finite variance constraint? Which seems pretty natural.\n- Equation 3: How can we define a density $\\rho_t$ since there is no 'Lebesgue-like' (translation invariant) measure on infinite dimensional Hilbert space? Is it w.r.t. a Gaussian measure?\n- Table 1: It is nice to see that FDP can be parameter efficient. Is the reason for not trying larger architecture computational? If so what's the bottleneck? Would $\\inf$ - DIFF with 1M parameter be performing as well?\n- line 333: The MLP architecture seems very deep, have you tried something perhaps a bit shallower yet wider (e.g. 512 neurons)?\n- Section 6: What noise $R$ was used for this experiment? From a Karhunen-Loeve perspective would make sense to use the correlation of the data process see [Angus et al 2022].\n\n\nSpectral Diffusion Processes, Phillips, Angus and Seror, Thomas and Hutchinson, Michael and De Bortoli, Valentin and Doucet, Arnaud and Mathieu, Emile, 2022\n\nLIMITATIONS:\n- Something that has not been explored, yet can be left , is conditional sampling which I believe is often of interest e.g. looking at the neural processes literature.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes *functional diffusion processes (FDPs)*, which generalizes the SDE-based continuous-time framework for diffusion models for data living in Hilbert spaces. The work builds on the theory of infinite-dimensional SDEs and their time reversals, as well as an application of the infinite-dimensional Girsanov theorem to derive an ELBO objective. A practical implementation, via discretization and implicit neural representations, is proposed and evaluated on the Celeb-A dataset.\n\nSTRENGTHS:\n- The paper is generally clear and well-written throughout.\n- This work adds to the growing literature on function-space diffusion models (a quite active area) and will likely be of significant interest to the machine learning community. The main theoretical portion of this work places continuous-time, SDE-based functional diffusion models on solid theoretical grounds. \n- All theoretical claims throughout the paper are precisely stated and rigorously justified. To the best of my knowledge, the proofs of the claims (contained in the appendix) are correct.\n\nWEAKNESSES:\n- The experiments (section 6) are the weakest part of the paper. \n\t- The proposed methodology obtains significantly worse FID scores than standard (Euclidean) diffusion models. However, note that this is achieved with far fewer parameters.\n\t- It was also somewhat unclear how the \"complexity\" column in Table 1 was derived, as this seems like a highly subjective measurement\n\t- Also in Table 1, it was unclear why FID scores were being compared against FID-CLIP scores (for the model of Bond-Taylor & Willcocks, 2023).\n\t- It was unclear if the parameter counts in Table 1 also included the parameters for the INRs, or merely the parameters for the score network. \n- The relationship between this work and the various other concurrent function-space diffusion model works could be better clarified. \n\n\n#### Minor comments\n- The notation on line 169 was somewhat unclear to me on a first read, particularly what $\\hat{\\mathbb{Q}}_0, \\hat{\\mathbb{P}}_0$ represent\n- There may be a small typo on line 169: should it read $d \\hat{\\mathbb{P}}^{\\chi_T} = d \\hat{\\mathbb{P}}^{\\rho_T} \\frac{d \\chi_T}{d \\rho_T}$?\n\nQUESTIONS:\n- The work restricts its attention to classes of functions which can be reconstructed exactly via finite sets of samples (Sect. 3). If my understanding is correct, the observation points may vary across functions. Does the proposed practical implementation allow for this? If so, how would you expect performance of the model to change if one were to e.g. train on one discretization and sample on another, such as an image super-resolution task?\n\nLIMITATIONS:\nThe limitations of the work are adequately addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel continuous-time diffusion-based generative model on function space. Unlike previous works in this area, which primarily focused on discrete-time formulations, the authors concentrate on stochastic differential equations (SDE) in their approach.\n\nThe paper begins by defining the forward process, also known as the noising process, as an stochastic differential equations (SDE) on function spaces. The paper subsequently demonstrates the existence of corresponding backward processes under certain conditions, similar to the formulation of finite-dimensional score-based generative models. Notably, two constraints in the forward process are required to guarantee the existence of the reverse diffusion process, which distinguishes the current work from its finite-dimensional counterparts. Firstly, the perturbation noises should be cylindrical Wiener processes, including $R$-Wiener process with a trace-class covariance operator $R$. Secondly, an operator $\\mathcal{A}$ can be present in the drift term of the forward process, which generates a strongly continuous semigroup; the semigroup will serve as a contraction map.\n\nThe paper highlights the novelty of its results by emphasizing that previous approaches have not fully explored the limits of discretization. Therefore, the provided proof offers theoretical support for the utilization of various types of integrators, unlike previous approaches.\n\nFurthermore, the authors introduce an evidence lower bound (ELBO) on the reverse diffusion process, leveraging an extension of the Girsanov theorem. This result provides a theoretical foundation for optimal guarantees, in contrast to previous approaches that relied on heuristics to minimize score matching objectives on function space.\n\nThe paper also extends the sampling theorem, enabling the reconstruction of the original function from its evaluations at a countable set of points. This extension justifies the training of function-space models with finite observations. Consequently, the authors exploit the sampling theorem to employ implicit neural representations for the model parameterizations, unlike previous methods that predominantly relied on neural operators.\n\nFinally, the paper demonstrates the effectiveness of the proposed method on various image generation benchmark datasets. In the experiment, they use Fourier basis functions inversely proportional to the square of the basis index for the perturbation noise.\n\nSTRENGTHS:\nI believe that the paper's theoretical results will make significant contributions to the machine learning community for several reasons.\n\nFirst, the existence of the reverse process ensures the optimality of common model parameterizations (Markovian), specifically those that model only the drift term of the reverse process, including score modeling.\n\nSecondly, the SDE formulation sheds light on the utilization of various integration methods for sample generation, such as DDIM or exponential integrators.\n\nFurthermore, the derivation of the evidence lower bound (ELBO) using Girsanov results in Equations 12 and 13. In these equations, the covariance operator $R$ is presented in the norm of the $R^{1/2}(\\mathcal{H})$ space. The inclusion of this additional $R$ helps address the problem encountered when defining the Kullback-Leibler (KL) divergence for previous generative models based on discrete-time diffusion in function space.\n\nLastly, the sampling theorem introduces intriguing open questions regarding the parameterizations of score-based generative models in function space. This opens up the possibility of employing new network architectures beyond neural operators.\n\nWEAKNESSES:\nMost of the content in the paper is good. However, some improvements are needed in the experiment.\n\nIn particular, the characteristic of the function-spaced model is its resolution invariance. However, the analysis of this aspect is missing in the paper's results. I believe that some analysis regarding the resolution-invariant property of the learned models should be included, even if it's just a small amount. In such analysis, I don't think it is necessary to strictly rely on an implicit neural representation.\n\nQUESTIONS:\nUnlike neural operators, implicit neural implementation is powerful, but achieving discretization invariance is challenging. How do you think is the best way to approach this?\n\nLIMITATIONS:\nPlease refer to the comments provided in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a model approach called Functional Diffusion Processes (FDPs) that generalizes score-based diffusion model to infinite-dimensional function space. The authors derive the reverse time dynamics and sampling theorems to find a subset of functions on a countable set of samples without losing information. The authors demonstrate their approach on a multi-layered perception.\n\nSTRENGTHS:\n1. The theoretical parts of the paper appears to be well structured. However, the paper itself does not appear to be self contained and has many references to the appendices to understand. For example, the paper makes many references to the assumptions being made, but they are never mentioned in the main paper and are only mentioned in the appendix.\n\nWEAKNESSES:\n2. The authors claim their approach is to create a new breed of generative models in function space, which do not require specialized network architectures, and that can work with any kind of continuous data. This itself is an interesting idea. However, after reading the introduction and looking at the experimental results. I cannot see what applications this may benefit and how much of an improvement it has to have a purely functional domain. I believe this could be better motivated.\n3. This may be because I didn't understand the experiment in Section 6 and does not appear to be self contained. I can understand that there is a page limitation, however, I believe that the experiment section is a bit compressed and difficult to understand. Having said that, the authors do have additional experiments in the appendix which appears to be much better structured.\n\nQUESTIONS:\nThe theoretical parts of the paper is quite dense and I cannot say I have fully understood the paper to come up with any meaningful questions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a framework for infinite dimensional diffusion models in Hilbert spaces, including deriving a reverse process and novel ELBO loss objective. The authors introduce a novel score model network architecture.\n\nSTRENGTHS:\n- This is a timely paper with a lot of interest in the community. There are many concurrent works also looking into similar topics e.g. [1] (I do not expect detailed comparison given they were only public recently).\n- Although performance is not great compared to standard diffusion methods in terms of generative modelling, the results are better than Neural Processes and similar point wise methods (with the exception of [2], concurrent so cannot criticise for worse performance)\n- The use of implicit optimization based encoder, g, in equation 16, is interesting. I have some concerns detailed below but nevertheless it is interesting.\n- Theoretical results seem correct but have not been checked thoroughly\n\n[1] Infinite-Dimensional Diffusion Models for Function Spaces, Pidstrigach et al, 2023\n[2] Infinite resolution diffusion with subsampled mollified states., Bond-Taylor et al, 2023\n\nWEAKNESSES:\n- The implicitly defined INR encoder, g, detailed in equation 16, considers gradient descent in an inner layer. Will this not be quite slow and hence generation / sampling would also be quite slow?\n- Empirical performance / FID scores are quite bad in comparison to other standard methods. Although interesting, is there an application where this method would be more useful than standard methods? \n- Functional diffusions could be interesting for dimension agnostic diffusions, super resolution, irregular data. There are limited experiments looking into these other applications.\n- Complexity in table 1 is quite subjective. I would argue parameter could does not matter if it can be ran on a single GPU in reasonable wall clock time. The implicit layer, g, with gradient descent for equation 16 is sequential so I imagine this slows down generation significantly.\n- Calling other concurrent / prior work heuristic without details. \"However, these works do not formally prove the existence of a backward process and their score approximation and score matching optimization objective is heuristic\". Does not [1] prove existence in Theorem 1? (I do not expect detailed comparison given they were only public recently, but given the authors claim this conccurrent work is heuristic I would be interested to know more).\n\n[1] Infinite-Dimensional Diffusion Models for Function Spaces, Pidstrigach, 2023\n\nQUESTIONS:\n- The implicitly defined INR encoder, g, detailed in equation 16, considers gradient descent in an inner layer. How many steps are taken? Is the gradient through the score network taken through these gradient descent steps? Is this not quite slow? Have the authors considered other point-wise encoders?\n\nLIMITATIONS:\nThe authors have a brief paragraph on limitations regarding perform, more challenging experiments and comparisons to other approaches like NFO, but could be further discussed. See questions and weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}