Score-based Generative Modeling through Stochastic Evolution Equations in Hilbert Spaces
Authors: Sungbin Lim
Eunbi Yoon
Taehyun Byun
Taewon Kang
Seungwoo Kim
Kyungjae Lee
Sungjoon Choi
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Generative Model
Score-based Method
Diffusion Model
TL;DR: We propose a unified framework for score-based generative modeling in Hilbert spaces using stochastic evolution equations with time-dependent operators.

Abstract: Continuous-time score-based generative models consist of a pair of stochastic differential equations (SDEs)—a forward SDE that smoothly transitions data into a noise space and a reverse SDE that incrementally eliminates noise from a Gaussian prior distribution to generate data distribution samples—are intrinsically connected by the time-reversal theory on diffusion processes. In this paper, we investigate the use of stochastic evolution equations in Hilbert spaces, which expand the applicability of SDEs in two aspects: sample space and evolution operator, so they enable encompassing recent variations of diffusion models, such as generating functional data or replacing drift coefficients with image transformation. To this end, we derive a generalized time-reversal formula to build a bridge between probabilistic diffusion models and stochastic evolution equations and propose a score-based generative model called Hilbert Diffusion Model (HDM). Combining with Fourier neural operator, we verify the superiority of HDM for sampling functions from functional datasets with a power of kernel two-sample test of 4.2 on Quadratic, 0.2 on Melbourne, and 3.6 on Gridwatch, which outperforms existing diffusion models formulated in function spaces. Furthermore, the proposed method shows its strength in motion synthesis tasks by utilizing the Wiener process with values in Hilbert space. Finally, our empirical results on image datasets also validate a connection between HDM and diffusion models using heat dissipation, revealing the potential for exploring evolution operators and sample spaces.

Score-based Generative Modeling through Stochastic Evolution Equations in Hilbert Spaces
Sungbin Lim 1, 66 Eunbi Yoon Taehyun Byun', Taewon Kang", Seungwoo Kim", Kyungjae Lee", Sungjoon Choi
Department of Statistics, Korea University
DDpparmment of Artificial Intelligence, Korea University
3 Department of Computer Science and Engineering, Korea University
*Artificial Intelligence Graduate School, UNIST
Department of Artificial Intelligence, Chung-Ang University
LG Al Research
SNU-GG AI Research Center
Abstract
Continuous-time score-based generative models consist of a pair of stochastic differential equations (SDEs)-a forward SDE that smoothly transitions data into noise space and a reverse SDE that incrementally climinates noise from a Gaussian prior distribution to generate data distribution samples- are intrinsically connected by the time-reversal theory on diffusion processes. In this paper, we investigate of stochastic evolution equations in Hilbert spaces, which expand the applicability of SDEs in two aspects: sample space and evolution operator, so they enable encompassing recent variations of diffusion models, such generating functional data or replacing drift coefficients with image transformation. To this end, we derive generalized time-reversal formula build bridge between probabilistic diffusion models and stochastic evolution equations and propose a score-based generative model called Hilbert Diffusion Model (HDM). Combining with Fourier neural operator, we verify the superiority of HDM for sampling functions from functional datasets with a power of kernel two-sample test of 4.2 on Quadratic, 0.2 on Melbourne, and 3.6 on Gridwatch, which outperforms existing diffusion models formulated in function spaces. Furthermore, the proposed method shows its strength in motion synthesis tasks by utilizing the Wiener process with values in Hilbert space. Finally, our empirical results on image datasets also validate connection between HDM and diffusion models using heat dissipation, revealing the potential for exploring evolution operators and sample spaces.
the
Introduction
Score-based generative models [52] have shown success various domains, including the generation of images [25], texts [30], videos [26], motions [31]. [52] proposes framework for continuous-tim score-based generative models that use a pair of stochastic differential equations (SDEs); a forward SDE smoothly transitions data into noise space, and reverse SDE incrementally eliminates noise from a Gaussian prior distribution to generate samples from the data distribution. Both SDEs have the same marginal distribution when their coefficients satisfy the time-reversal formula [1, 24), which equation for diffusion processes.
is an ingenious application of the Hainogorov-Fooker
"Corresponding Author. e-mail: sungbin@korea. ac.
This work is done at UNIST.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A Unified Framework with Stochastic Evolution Equations in Hilbert Spaces
Score Operator
0)
dX,- B, (X1) )r G,dW,
dx, (x,) + ) CdW --
P data
Forward Evolution
prior
Reverse Evolution
P data
(a) SDE in Infinite-dimensions (HDM-SDE): B, -b(1)I, G, v2b(1)1
(b) Parabolic Stochastic PDE (HDM-SPDE): B, = (())( ol). G, (2b(1)(o11)) A)
+6
Figure 1: A unified framework with various stochastic evolution equations to yield score-based generative models for different datasets. (a) SDE infinit-dimmensional spaces can generate functions as samples from the function space. (b) Utilizing prior distribution generated from heat dissipation- type forward evolution B, = ( - oI), the same as IHDM [50], we can generate image samples by adding Gaussian noise (6) and executing reverse evolution through the parabolic SPDE.
Recently, there has been an active proposal of variations of diffusion models. such as generating functional data [18, 19, 23, 29, 38, 48, 49] or replacing drift coefficients with image transformation [4, 27, 50), which cannot be covered by the original SDE framework [52] formulated in the Euclidean space. Motivated by this problem, we propose a unified framework for continuous-time score-based generative models by using stochastic evolution equations in Hilbert spaces [6, 14, 34, 35].
dX, (X,)) + G,dW,, dX, = S X,) + Gr-,dW, (I)
where X,,8) is pair of forward and reverse stochastic processes with evolution operators (B,, G.). W, is a Wiener process with values in some Hilbert space H. and S(t,-) [0, T] x H + H score operator, which generalizes the notion of score functions in the Euclidean space. Since can choose H. and (B,, G,) in an abstract setting, stochastic equations (1) naturally extend the use of original SDEs the miniiieeddieniionaal setting and include stochastic partial differential equations (SPDEs), such as parabolic equation driven by white noise (see Figure 1). Consequently, we can expand the SDE framework [52] in two aspects: sample space and evolution operator, enabling it to encompass recent variations of diffusion models in a continuous cittiuuuus-imeeeee score-based generative model.
Contributions An essential key to identifying the relationship between the forward process X, and the reverse process X, in (1) is the derivation of time reversal formula with time-dependent operators B, and GG, which has been a challenging problem [23, 29, 38, 49] when we utilize coefficient schedulings [46, 52]. Contrary to previous approaches using semigroup theory [14], our approach is grounded in Kolmogorov equations with functional derivatives, well-studied in [2, 3, 7, 15-17], so that we can consider wide scope of stochastic evolution equations with time-dependent operators.
We derive a generalized time-reversal formula and unify the SDE framework [52] with stochastic equations in Hilbert spaces. Consequently, we propose a class of continuous -ime score-based generative models, Hilbert Diffusion Model (HDM), which enables using SDEs in cifiniteddimesiinaal settings (HDM-SDE) and parabolic SPDEs (HDM-SPDE).
Combining with neural operators [37], HDM-SDE shows the superiority for sampling functions from functional datasets with multi-modalities compared to GP [55], NP [20], DDPM [25], and other diffusion models formulated in function spaces, NDP [18] and SP-SGM [48]. We also validate that HDM-SDE plays crucial role in successful human motion synthesis compared to DDPM, by utilizing the Wiener process in the Hilbert space.
We build a bridge between parabolic SPDE and diffusion models using heat dissipation [50]. HDM-SPDE improves the performance in image generation as a continuous-time version of IHDM. Also, the proposed method shows the inductive bias in controllable coarse-to- fine image generation, e.g., inpainting, deblurring, and depixelation, without additional training.

---

2 A Foundation on Time-Reversal of Stochastic Evolution Equations
We first present preliminary theory the stochastic evolution equations in Hilbert space and introduce the logarithmic derivatives of probability measures to propose a Hilbert space valued score operator (see Section 2.2) which generalizes the notion of score functions in the Euclidean space. Then we derive the time-reversal formula (Theorem 2.1), which is a key to obtain the time-reversal of the diffusion processes by using the Kolmogorov equations in Hilbert space.
2.1 Preliminaries: Stochastic Evolution Equations in Hilbert Spaces
In preliminaries, we introduce elements of stochastic evolution equations in Hilbert spaces, See B.1 refer [5, 6, 14, 34, 35] for basic setting.
Gelfand Triple and Cameron-Martin Space Let H. be a separable Hilbert space with the H- norm defined by the inner product || I1x = - Let L(H) denotes the set of all bounded linear operators
H. We assume there exists a Radon centered Gaussian measure À the Borel o-field ((H) with the nonnegative, symmetric, self-adjoint, and the finite trace covariance operator E L(H), hence there exists eigensystem {(X(0, .o(e)) 14, X H. E NJ such that A()()) = (()) holds and Tr(A) = -i=1 1(0) < 00. We define Cameron- space by HA == A (H) with the inner product (h1, h2) == (A-1/2(h1). ( A-1/2(h2)) Let H2 C H², be the dual space of H and HA, respectively. By identifying H. with H3 via the Riesz isomorphism, we can consider a continuous inclusion map iRA. 7 + HA such that
(())
(E = (ina (p), = (An, (p)), ( EE
Thus, we can also identify H' with (2/2(H²) (see [8, Example 1.1D. For notational convenience, we write in (v) as A(p) by regarding sp as an element in H. In summary,
H²* (HH,, in, (H") A(H)) in, (H²) C Hx C H.
(3)
We refer to (H" Hx, H) Gelfand triple which provides groundwork for the SDE theory in H. Example 2.1 (Euclidean space). Let H. == _d and À (N Σ) be Gaussian measure with positive-defiite covariance matrix Σ E Rdd Then | < oo for x E IR4, hence HA H. O
Example 2.2 (Reproducing kernel Hilbert space). Let denote the class of square-integrable func- tions finite measure space (xxx with the usual L2-norm, || (Jx (dxr)) (d2 Due to the assumption on A, there exists symmetric and positive definite kernel function KA such that
A()(x)= f (x(xx))((y))aalll
XE S H.
Hence Cameron-Martin space HA is reproducing kernel Hilbert space (RKHS) with Kx.
Wiener Processes and Stochastic Evolution Equations in Hilbert Spaces For T let (We)cefo.r) be H-valued F--adapted A-Wiener process (see Definition B. for details). By the Kosambi- Kosambi-Karhunen theorem, We & has series representation:
W A12(WW)) Σ VXew 600 tE [0,1].
(5)
where W; = (w e EN is white noise of which the components w! are independent real- valued Wiener processes. Then Cov( 1A holds and TE W. Wa'H = min(t, Tr(A) < oo since A has the finite trace. If HA is RKHS with K₃ (Example 2.2), W, preserves spatial correlation,
E (xx)) min (x), X1, E
(6)
Let (?( denote the class of Hilbert-Schmidd operator on H. Following the usual construction of the stochastic integral with respect to W, (e.g., see [14, Section 4] and [35, Section 2)), we can define the following stochastic evolution equation in H.
X = + f' (X, / G.dW., Xo E H. tE [,T),
3

---

where B, 10 x H. - H. and G. [0, + L2(H) are progressively measurable satisfying regular conditions [35, Section 3.2] which guarantee solvability H. Under these assumptions, there exists unique H-valued solution X; to (7) for every / E [0, (), such that Xo E HH and W; E H are independent. We assume G, is constant on H. which still recovers the SDE framework of [52].
2.2
Score Operator and
Formula
Now consider time-reversal X, = XT- of (7) and its corresponding diffusion process,
xo [xB.(XX.ddt+ 1," G.dW.. ~o~N(1 A),
E [0,
(8)
condition for the existence of the time-reversal process (8) is well-studied for diffusson processes in abstract settings (e.g., see [12)), hence we focus on identifying evolutionary operators B and G.
Logarithmic Derivative of Probability Measure and Score Operator A score function in the Euclidean space is defined by gradient of logarithmic probability density V. Po Pr (z). For Hilbert spaces, we employ logarithmic derivatives of Fomin differentiable probability measures [8, 9]. Let us define the class of smooth cylinder functions [10], which dense in the Sobolev space on H. by
FC = [fpr.. = f(( pm) H + R E Cg (R"), E H," m E N}.
(9)
For f.i. E FCG, we compute the Hx- -radddit at u E H. the direction of & E Hx by
(Drrmm (w),5))) = Σ ( ( ((Pi)) E H.
(10)
where Df denotes a Gâteaux derivative. A differentiable measure μ on H along a vector & E Hx has the partial logarithmic derivative P E L1 LLH,, 11) if the following integration by parts holds,
J (D5p.. (u), (( = fpr. (w) (w))(dd), Spr.m E FCC
(11)
If there exists a Borel map PHz (-) H. - H. such that (Pt, (u), E)H,, (u) for & E H), then Pis is called the vector logarithmic derivative of associated with HA. For instance, if is Gaussian measure with mean m E H and covariance operator A, L(H) satisfying A(H) C (P)), then PR is computed by (see [8, Example 2.5] and [14, Proposition 2.26]).
(w) = A o (m u), u H,
(12)
which generalizes the score function Vx log p(x) of Gaussian density p(x) ~ ~N(mp, Ep). For sequence of probability measures (ee)) 70 of the solution to (7) with μ4 = Paaus we define a score operator (t. [0, 7] x + H by multiplying G;G; to the vector logarithmic derivative dr
St u) G.G. (u).
(13)
Hence the score operator (x(,)) is analogous to (1) logP((x) in the reverse SDE of [52].
Kolmogorov Equations and Time-Reves Reve-Reversal Formula Score-based generative model through SDE is grounded on the Fokker-Planche Kolmooooo equations [1, 24). Following [5, 6, 8-10), we define Kolmogorov operator Le the class of smooth cylinder functions FCS by
Lifor (u) TrH, (A, o (e) + (Df (u). (u()))
(14)
where A, = G,G; and D2 denotes the second-order Gâteaux derivative. The Kolmogorov operators (Li) (Li)eejo,r describes evolution of (He)εeoory and becomes generator of Markov diffusion
process [5, 6, 24]. Now we present our main result on the time-reversal formula.
Theorem 2.1 Time-RReersoe Formula). Time reversal X, satisfying (8) has generator
D2 fpr. (u)) + (Df (w). Belu))mx Difrrm (u) Tru, (A, o A D2
(15)
where Ar and Br satisfy the following time-reversal
A, AT- Be(u) -Br-e(u) Sx u).
(16)

---

See Appendix B.1 for the proof of Theorem 2.1. The proof is based on the Kolmogorov forward equation and the Kolmogorov backward equation with functional derivatives show that (15) the generator of the time-reversal diffusion process (8), which is analogous to the proof in the Euclidean space [24]. Due to (16), we can rewrite (8) by
ds+ I. GT- X, = Xo + '." ( SA - -s, syX.)
(17)
Hence we can generate a sample from Xy ~ Panaa by running (17) we obtain Sx(t,-) fort E [0, T].
2.3
Applications of Theorem 2.1
This section presents two important applications of Theorem 2.1; SDEs in and parabolic SPDEs in finite-dimensional space. We utilize these stochastic evolution equations to propose unified framework of continuous-time score-based generative models in Section 3.
space
2.3.1 SDEs in mnfnite-Dimensee
We first generalize the SDE framework [52] in the unfinit-dimensee based on Theorem 2.1. Let {A(e) ol JeEN be an eigensystem of H and consider the following SDE system with values in H,
dx
( by") x di + VA
( aww!! E [0, [0,T),EEN,
(18)
where 61²)
[] R and of [0, R+ are real-valued functions for each & E N. Set
3(00 = exp (5o 6L/ dg) and 2/ (1) = (a! (o!)) (B()))²² Conditioning at X the law of x( follows
Gaussian distribution, hence apply [44, Theorem 5.3] to (18), which induces B (x by
C-660 602 20,00000 10
8/4 x( x1()) J (s)ds
Bx(x) =
(19)
EEN
Indeed, (19) is a special case of (16) (see Appendix C.I).
Example 2.3 (VP-SDE and sub- sub-VP-SUD VP-SDE in Infinite- Dimensions). Set 1b0) (bt) and of Vb(t) for every l E N. Then we have Bg e- à So b(s)da and 21 b(t)e b(w)ds Thus,
X, e S b()dd Xo. A = -5 b(s)da) A.
(20)
Hence we can compute the conditional score function (t, X, (x)) by using (19). As a result, (20) generalizes VP-SDE [52] in fiinite-dimensione Similarly, we can recover the sub-VP-SDE in infnite-dimensions, hence (19) recovers SDEs proposed in [52] when A Id and H. Rd
2.3.2 Parabolic SPDEs
We introduce parabolic SPDEs [34, 35] based on Theorem 2.1. For -1, define
du, b(1)AAu ot dt= + /2b(1)(ol - A)
dwp,
( E [0,7],
(21)
on a bounded domain O with a zero-Neumann boundary condition. By following [50], we use the discrete cosine transformation (DCT) from coordinate vector u, expressed in the standard basis to coordinate vector ü. with the eigendecomposition A VDVT. Hence can rewrite (21)
dů, = b(t) oû + 2b(t) D) dWW, E [0,
(22)
where ú, T u and W. V WW, Since V orthogonal, we have W, d W, for each t, so can regard Wi as standard space-time white noise. Thus, we can apply Theorem 2.1 to (22) with B, = ((t)(DD ol) and G; = 22((t)(ol- -D) in Re (see Appendix C.2). Therefore,
B, = bT ()( DD) + 2b(T - ()I D) D)- PHa ->
6r-
(23)
5

---

If we set b(t) = 1, a 0, and k O, then (21) becomes stochastic heat equation (SHE) and has connection between the generative process of IHDM [50]. Indeed, the mild solution of SHE
Wh A ' ( d
1 E [0,
(24)
where e is Co-semigroup, which induces the mean vector of as e = V expp(DiVV) no- and the remaining stochastic integration part becomes Gaussian process [14, Theorem 5.2]. consider continuous-time version of IHDM with variance scheduling, then (23) induces the appropriate reverse SPDE for sampling. See Section 4.3 for empirical results.
Hilbert Diffusion Models
This section is devoted to describing Hilbert Diffusion Models (HDMs), a class of score-hased generative models based Theorem 2.1 and stochastic evolution equations presented in Section 2.3. We introduce two versions of HDM, HDM-SDE and HDM-SPDE, which utilize SDEs in infinite-dimensions (18) and parabolic SPDEs (21), respectively.
3.1 Estimating Score Operators
To estimate the score operator (t, in Hilbert space, we propose a time-dependent score model sa (t, : [0, 77 x H + H. parameterized with 0 IR", which is trained by the following objective,
0" arg min 1x [l X,) - St, x
(25)
BERP
where denotes the uniform distribution on [0, 7]. Due to [38, 54], note that minimizing (25) is equivalent to minimizing the following objective,
6r arg min Bx, Ex, (t. X,) St, X
(26)
Re
where Me Xo denotes the conditional distribution of X, conditioned at X Paara and S (t, X, (Xo) the corresponding score operator, which is estimated by a time-dependent neural network.
Flement-wise addittio operation
Batch
Lilting Channe's
Dimension of Data
Number of Fourler Layers
Positional Emblecking (P)
Layer (Net)
Fouiner Laler Block
FFT
Reckulioe
JFFT
Activation XeeUD
[I, c]
i. 68]
Not(
Skip Skip-cumaaatin Hoa book
Skip core on tunn block
litting Layer
Fourler
Fouster Layer
Actizttie
Fourier Layer
Activation (GeUUh
Projection Layer
8g(A,
(GeLU)
J C.C.
[8. 0)
(3.
[8,0,0)
[B.E.
[B, (.
Figure 2: The architecture of time-conditioned FNO.
Architectures of HDM We use different architectures for modeling HDM-SDE and HDM-SPDE since the corresponding stochastic evolution equations (18) and (21) have distinet values infinite- dimensions and Fuclidean space, respectively. For HDM-SDE, same as other diffusion models in diiiittedimeesions [23, 29, 38), we modify Fourier Neural Operator (FNO) [37], called time- conditioned FNO, by adding a positional embedding and lincar layer to embed the time variable t (see Figure 2). The spatial variable x first passes through the lifting layer, increasing its dimensions to match the lifting channels. Then, the embedded time variable undergoes the linear layer to adjust its dimensions to match the lifted spatial variable, and is pointwisely added. The Fourier layer takes the input and applies Fast Fourier Transformation (FFT) to transform it to the frequency space, followed by spectral convolution. See Appendix C.3 for more details.
For HDM-SPDE, we use U-Net [51] similar to other diffusion models [25, 52]. mostly varying the number of multipliers of resolutions and the number of residual blocks. Detailed architecture parameters according to datasets are described in Table A.3.

---

3.2 Training and Sampling
Based on (26). both HDM- SDE and HDM-SPDI use the mean-square loss as the training loss function. Fix Xn data and let be generated sample at time t from jr Xo so that Xx (m xxs A pdxx)) Then the loss functions are represented by
Loss Functions
1o (e) (0)l(
LSDE( (f) = (0-21(0077) Ex Exn Ee-N(O.T)
se x,) + B(t) 1/2
(27)
LSPDE (0) 11 Exo- B (1101
So xx) + V (3(1)A
Ke/ba (e)
(28)
Here the cosine beta schedule [46] with ((t) 1+a 2 HDM-SPDE for a small number
tan
for both HDM-SDE and 1+a
[+
>> O.
Solving Reverse Stochastic Evolution Equations After training score model (t, for all E [0, 2", we can generate samples from Fan by running the reverse process (17) with the trained score model instead of S (t, -). Numerically, we utilize the Euler-Marus aarer--aruyama method approximate a solution of (17) starting at X ~ (P(0,A) (see Algorithms and 2).
Xem4a = & + (B (X...) So tm: &em)) At V VAGGT-Gn Nem. (29)
where No i.d. A), and At tm+1 tm for E (O....,MM M} with to and tM T.
4 Experiments
This section presents the experimental settings and results that apply HDM-SDE to generating function and motion and HDM-SPDE to image generation. For more details, see Appendix D.
4.1 Function Generation through HDM-SDE
Datasets and Implementation We validate HDM-SDE for generating functions from 1D functional datasets; Quadratic, Melbourne, and Gridwatch, to follow the setting in [48]. We refer to Section D. for more details about datasets and settings. We tune the kernel hyperparameter of len parameter for 0.8 in Quadratic, 2.0 in Melbourne, and 1.8 in Gridwatch. We commonly use gain parameter 1.0 for all datasets. Grid parameter adjusted 100 in Quadratic, 24 in Melbourne, and 288 in Gridwatch. In the Quadratic and Melbourne datasets, we train our model during IK iterations (1.5K in Gridwatch). For sampling, we set the number of sampling steps as 1,000. For a quantitative evaluation, we calculate the power of a kernel two-sample test [56] based on Maximum Mean Discrepancy (MMD).
POANT:
Fower 4.2
Dataset
DDPM
(b) NDP
HDM-SDE
Figure 3: Comparison of functional samples generated by DDPM (VP-SDE) [25], NDP [18], and HDM-SDE (ours) on Quadratic dataset.
Results Table shows the results of the test power analysis, and HDM-SDE demonstrates excellence compared to Gaussian process [55], Neural process [20], DDPM (VP-SDE) [25], and other diffusion models (NDP [18] and SP-SGM [48]) formulated on function spaces. Our method records power({) % of 4.2 on Quadratic, 0.2 on Melbourne, and 3.6 on Gridwatch. Figures and A. show that HDM-SDE method has outperformed results compared to the diffusion models utilizing the
7

---

GP [55]
NP [20]
DDPM [25]
NDP [18]
SP-SGM [48]
HDM-SDE (Ours)
Quadratic
10..0
8.6
2 99.0
99.0
5.4
4,2
Melbourne 20.1
10.1=1.9
3.3
12.8
5.3
0.2
Gridwatch
29.2+5.5
51.8+15.1
16.641.9
16.321.8
4.7.0.5
3.6
Table I: Power(%) of a kernel two-sample test on ID datasets. Lower is better.
white-noise, due their limited adaptability variable discretizations as discussed in [38]. A trace-class diffusion model using Langevin dynamics [38] records power of 6.9 on Quadratic and of 4.0. on Melbourne, which is comparable to SP-SGM but are less favorable than HDM-SDE.
4.2 Motion Generation through HDM-SDE
DDPM
Frame
HDM-SDE
(Ours)
Figure 4: Generated motion by the baseline (DDPM) [25] and HDM-SDE (ours).
Datasets and Implementation We further investigate the performance of HDM-SDE for motion generation tasks using HumanML3D dataset [22]. The HumanML3D dataset consists of 14,616 motions, where each motion consists of 144-dimensional trajectories, derived from 24 joints and 6D representation and we randomly sample 100 motions for the experiment. The root poses of the skeletons are fixed to the origin, and the motion length is trimmed to 128. We compare method with the baseline (DDPM) [25] utilizing the FNO architecture except the input and output channel sizes of FNO are changed from to 144 to handle 144-dimensional trajectories. Hence, our proposed method and the baseline have identical network architectures and loss functions, and the only difference is the noise sampling procedures in forward and reverse processes. We utilize the squared-exponential kernel for generating the Wiener process in RKHS where the length parameters of each 144-dimensional trajectory are selected using a likelihood- model selection of Gaussian process [55]. See Appendix D.3 for details.
Results The synthesized motions of the proposed method are shown in Figure 4, where we overlay right and left hands trajectories with red and blue, respectively. While the proposed method success- fully generates human motions, the bascline fails to generate meaningful motions. Our proposed method significantly outperforms the baseline in terms of motion synthesis the baseline fails to generate human-like motions. This clearly highlights the advantage of utilizing H-valued Wiener process in modeling multi tittesiiimnii complex trajectories. The quantitative results demonstrate better performance in both hand and foot trajectories within the task space are shown in Table A.2.
4.3 Image Generation through HDM-SPDE
Datasets 2D image experiment includes MNIST (36), CIFARIO [42], LSUN-church [58], AFHQ [13), and FFHQ [28] datasets. We adjust the image resolutions as IHDM [50], LSUN-church images resized into 128x 128; higher resolutions, including AFHQ and FFHQ, have resized into 256x256. See Appendix D.4 for implementation details.
To evaluate the sampling quality, we use the clean FID [47] using Inception V3 [53]. Clean-FID scales generated and reference images into 299x299 resolution via bicubie interpolation and compute statistics. Following IHDM [50], we sample 50,000 images with resolution smaller than 256x 256 and calculate the reference samples as the training set. Otherwise, we sample 10,000 images and use the entire dataset as reference data for evaluating metrics.
Evaluation
8

---

FID: 17.84
FID: 29.09
FID: 17.94
FID: 15.21
9
5
3
3
9
6
4
5
3
0
0
0
8
2
5
2
MNIST
CIFAR1O
LSUN
AFHQ
FFHQ
Figure 5: Uncurated generated samples for different image datasets via HDM-SPDE.
ISUN
APHQ
FFH
Origin
Degraded
NFF: 30
INF:: 7D
NFF: TO0
NFF: 1000
HDM Prior
Super-resolution resolution
IHDM Prior
Depixelate
Gaussian Prior
Debluming
Figure 6:
Sampling from HDM-SPDE prior,
Figure 7: Sample trajectory of HDM-SPDE on super-resolution, depixelate, and deblurring tasks.
IHDM prior, and Gaussian prior 0.01 using trained HDM-SPDE.
Results Figure summarizes uncurated image generation results HDM-SPDE on various image datasets. Our model records FID scores of 17.84 on CIFARIO and 29.09 LSUN-church, which is
better than that of IHDM [50] (CIFAR: 18.96. LSUN: 45.06). Especially, our model records PRDC [45] of (P: 0.696, R: 0.633, D: 0.733, C: 0.727)
CIFAR1O, which is higher than that of IHDM (P: 0.596, R: 0.767, D: 0.580, C: 0.723) except for recall. For AFHQ and FFHQ, HDM-SPDE achieves least twice the better FID score of 17.94 AFHQ and 15.21 on FFHQ than those of IHDM (AFHQ: 41.39, FFHQ: 64.91). Although there remain gaps compared to the state-of-the- results using diffusion models [25, 52], the improvement in sample quality continuous- time version of the IHDM is significant. Thus, we conclude that the proposed framework opens the potential for exploring various evolution operators having inductive bias image generation.
Connection between HDM and IHDM As we addressed in Section 2.3.2, SPDE and IHDM have connection such that the mean trajectory of the SPDE is equivalent to the trajectory of the forward process of IHDM. Hence we hypothesize that the HDM can generate images by running the reverse SPDE from the prior distribution of IHDM. Figure 6 illustrates the generated images by trained HDM-SPD provided with HDM, IHDM, and Gaussian priors. Notably, the reverse SPDE generates similar images from IHDM prior HDM one, contrary to Gaussian prior. On AFHQ and FFHQ, HDM prior results in FID of 17.94 and 19. 9,40, better than 19.40 and 20.89 of IHDM prior, still outperform the results reported by [50], 43.39 and 64.91, respectively.
Controllable Coarse-to-Fiie Fine Generation Motivated from [4], we examine whether HDM-SPDE can leverage the learned inductive bias controllable coarsc-to-fine image generation tasks. Figures 7 and A.4 show that HDM-SPDE can successfully perform inpainting. super-resolution, depixelation, and deblurring tasks without any additional training. HDM-SPDE capable of generating images that resemble the background color to some extent for pixelated images and also generates samples leveraging the contours of blurred images. This possible since low frequencies are filled with the frequencies of low-resolution images, whereas the range of high frequencies is filled with generated frequencies through the learned score model. Diffusion models trained by [52] fail to generate samples that capture the color or contours of pixelated blurred images (see Appendix D.5).

---

5
Related Works
Diffusion models in anfinite-dimensionall space have been actively investigated [18, 23, 29, 38, 48, 49). [29] introduces a discrete-time denoising diffusion model in unfnite-dimennional space, which provided one of the key inspirations for studying a continuous-time approach connecting with score- based methods through the SDE framework [52]. [18] is another discrete-ting time approach proposed to generalize neural processes [20], but it utilizes non-trace class white noise that causes an ill-posedness in the infinitedimmensionall setting. See [1 [14] for reference of Hilbert space valued Wiener processes.
A challenging part of the continuous-time approach in Hilbert space defining the score function without using the density function that no longer exists in the proposes using a discretized process circumvent this problem and [48] suggests a continuous-time model based on spectral decomposition stemming from the Karhunen-Loëwe theorem; however, they do not include the general SDE framework [52] nor time-dependent operators. Based on [44], [19] only considers constant-ting diffusion coefficients, which do not fully recover the desired framework requiring variance scheduling [46]. Section 2.3.1 shows the intersection between [19, 23] and our approach and how the proposed framework covers SDEs in infinite dimensions.
setting. [23]
Concurrently, [38, 49] propose continuous-time models; however, their theoretical results also confined to SDEs with constant-time operators since their approach is primarily grounded in semi- group theory [14], which has an intrinsic limitation when dealing with variable coefficients [39]. We note that the multiple noise scale case is revealed in [38, Section 4.4]. yet it is constrained to discrete-time models. On the contrary, our approach grounded in the study of variational approach to stochastic evolution equations [34, 35] and forward and backward Kolmogorov equations with functional derivatives [6, 15], which serves as more established approach to identifying relations (Theorem 2.1) 1) between time- dependent evolution operators and invariant measures of stochastic equa- tions in Hilbert spaces [5]. It is noteworthy that proof of Theorem 2.1 (Appendix B.2) can be viewed as generalization of the classical techniques employed by [1] and [24] in finteedimeensionl spaces to general Hilbert spaces. Consequently it able to fully recover the SDE framework proposed from [52], thereby enabling the variance scheduling [46], and building a bridge between the discrete-time model presented in IHDM [50] and the continuous-
score-based approach.
Similar to [11, 23, 29, 38], which combine diffusion models with neural operators [32, 37], we propose a modified cine-conditioned FNO for the HDM-SDE in the functional and motion generation task, which has slightly different modules from the original neural operator. For the image generation task with the HDM-SPDE, however, we use the U-Net [51] to compare with the setting in variations of diffusion models [4, 27, 50], which utilize image transformations instead of drift coefficients.
6 Limitation and Conclusion
We extend the SDE framework proposed by [52] and propose a class of continuous time score-basee generative models based on stochastic evolution equations in Hilbert spaces. Our derivation of the time-reversal formula considering dimeddepnnennt coefficients enlarges the practical application of score-based generative models in function spaces and advances the performance of recent variations of diffusion models which use blurring [50] or pixelation [4]. Although performance gaps remain in image generation compared to the state-of-the-art results, conclude that the proposed framework opens the potential for exploring various evolution operators and sample spaces in this area.
Acknowledgments and Disclosure of F Funding
This work was supported by LG AI Research. This work was also supported by Institute of Infor- mation & communications Technology Planning & Evaluation(IITPP) grant funded by the Korea governmen(MMSITXNN. 2022-0-00613 Geometric and Physical Commonsense Reasoning based Behavior Intelligence for Embodied AI; No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University), ). and National Research Foundation of Korea(NRF) funded by the Korea governmen((MSST)) IR1 A4A30331440 This work was supported by Artificial intelligence in- dustrial convergence cluster development project funded by the Ministry of Science and ICT(MIIT, Korea) & Gwangju Metropolitan City.
10

---

References
[1] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):31--326, 1982. ISSN 0304-4149. doi: https://doi.org/10.11111
0304-4149682)90051-55. URL https: http:::/wwwws.cienneeeeee www.sciinceeiiecct..ccccc..cccce cow wwm./cciieee//rrriccle... p11/0304411488290655..
[2] VV Baklan, On the existence of solutions of stochastic equations in hilbert space. Depov. Akad. Nauk. Ukr. USr, 10: 1299-1333, 1963.
[3] VV Baklan, Equations in variational derivatives and markov processes in hilbert spaces, In Dokl. Akad. Nauk. SSSR, volume 159, pages 707-710, 1964.
[4] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie $ Li, Hamid Kazemi, Furong Huang. Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.0900 09392, 2022.
[5] Yana Belopolskaya. Invariant measures of diffusion processes in hilbert spaces and hilbert manifolds. Probability Theory and Mathematical Statisties: Proceedings of the Seventh Vilnius Conference, page 67. Walter de Gruyter GmbH & Co KG, 2020.
[6] Yana Belopolskaya and Yu L. Dalecky. Stochastic equations and differential geometry, vol- ume 30. Springer Science & Business Media, 2012.
[7] Yana Isaevna Belopol' skaya and Yurii L'vovich Daletskii. Diffusion processes smooth banach spaces and manifolds. i. Trudy Moskovskogo Matematicheskogo Obshchestva, 37:107-141, 1978.
[8] Vladimir Bogachev and Michael Rockner. Regularity of invariant measures on finite and infinite dimensional spaces and applications. Journal of Functional Analysis, 133(1):168-22) 1995.
[9] Vladimir I Bogachev, Nicolai V Krylov, and Michael Röckner. Regularity of invariant measures: the case of non-constant diffusion part. Journal of Functional Analysis, 138(1):223-2) 222-242, 1996.
[10] Vladimir I Bogachev, Nicolai V Krylov, Michael Röckner, and Stanislav V Shaposhnikov. Fokker-Planck Kolnnooroo Equations, volume 207. American Mathematical Society, 2022.
[11] S Tond-Taylor and CG Willcocks. infinite-dift diff: Infinite resolution diffusion with subsampled mollified states. arXiv preprint arXiv:2303. 18242, page 2023.
[12) Patrick Cattiaux, Giovanni Conforti, Ivan Gentil, and Christian Léonard. Time reversal of diffusion processes under finite entropy condition. In Annales de I' I'Institut Henri Poincaré (B) Probabilités et Statistiques, 2022.
[13] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[14] Giuseppe Da Prato and Jerzy Zabczyk. Stochastic equations in infinite dimensions. Cambridge university press, 2014.
[15] Yu L. Dalecky and Sergei Vasil evich Fomin. Measures and differential equations in infinite- dimensional space. Springer, 1991.
[16] Yu L Daletskii. Infinite-dimensiona elliptic operators and parabolic equations connected with them. Russian Mathematical Surveys, 22(4):1, 1967.
[17] Yurii L'vovich Daletskii. Multiplicative operators diffuuson processes and differential equa- tions in sections of vector bundles. Uspekhi Matematicheskiihh Nauk. 30(2):209-210, 1975.
[18] Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes, arXiv preprint arXiv:2206.033: 2022.

---

[19] Giulio Franzese, Simone Rossi, Dario Rossi, Markus Heinonen, Maurizio Filippone, and Pietro Michiardi. Continuous-time functional diffusion processes. arXiv preprint WrXiv:2303.00800 2023.
[20] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:/807.0162. 2018.
[21] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng. Minglun Gong. and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proc. of the ACM International Conference on Multimedia, pages 2021-2023 2020.
[22] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152- 516521166,11 2022.
[23] Paul Hagemann, Lars Ruthotto, Gabriele Steidl, and Nicole Tianjiao Yang. Multilevel diffu- sion: Infinite dimensional score-based diffusion models for image generation. arXiv preprint arXiv:2303.04777 2023.
[24] Uirich G Haussmann and Etienne Pardoux. Time reversal of diffusions. The Annals of Probability, pages 188-1205, 1986.
[25] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proc. of the Conference on Neural Information Processing Systems (NeurIPS),
2020.
[26] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.0m 2.44452 2022.
[27] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. In International Con- ference on Learning Representations, 2023. URL 0jDkc57x5sz.
met/forummide
(28) Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396-4405, 2019. doi: 10.1009/VVRR2211904453
[29] Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models infinite dimensions. arXiv preprint (.Xii:2222.008880 2022.
[30] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2426-2435 2022.
[31] Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. In Proc. of the Association for the Advancement of Artificial Intelligence (AAAI), 2023.
[32] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research (JMLR), 24(89):1-97, 2023. URL http: / /mwrr./../rr///////
/aaerss/22221111 1524..2242 html.
[33] Nicolai V. Krylov. A relatively short proof of Itö's formula for SPDEs and its applications. arXiv preprint arXiv: /208.3709 2012.
[34] Nicolai V Krylov and Boris L Rozovskii. Stochastic evolution equations. Journal of Soviet Mathematics, 1111223312777 1981.
[35] Nicolai Krylov and Boris L Rozovskii. Stochastic evolution equations. In Stochastic Differential Equations: Theory And Applications: A Volume in Honor of Professor Boris L Rozovskii, pages 1-69. World Scientific, 2007.
[36] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2324 1998. doi: 10. 10011199777779
12

---

[37] Zongyi LI, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arXiv:2010.08895, 2020.
[38] Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Az- izzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, al. score-based diffusion models in function space. arXiv preprint arXiv:2302.07400, 2023.
[39] Wei Liu and Michael Röckner. Spde in hilbert space with locally monotone coefficients. Journal of Functional Analysis, 259(11):2902-299) 2010.
[40] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
[41] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons Mool, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision, pages 5442-5451, October 2019.
[42] Charles A Micchelli and Massimiliano Pontil. On learning vector-valued functions. Neural computation, 17(1):177-204, 2005.
[43] Annic Millet, David Nualart, and Marta Sanz. Integration by parts and time reversal for diffusion processes. The Annals of Probability, pages 208-238, 1989.
[44] Annie Millet, David Nualart, and Marta Sanz. Time reversal for infinite-dimensional diffusions. Probubility theory and related fields, 82(3)311--377, 1989.
[45] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In Proc. of the International Conference on Machine Learning (ICML), 2020.
[46] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabílistic models. In Proc. of the International Conference on Machine Learning (ICML), pages 8162- 8171. PMLR, 2021.
[47] Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[48] Angus Phillips, Thomas Seror, Michael John Hutchinson, Valentin De Bortoli, Arnaud Doucet, and Emile Mathieu. Spectral diffusion processes. In NeurlPS Workshop on Score-Based Methods, 2022. URL https::/ wwtp:/eeeoooiidddndmmmoooooooeee
[49] Jakiw Pidstrigach, Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinite-dimensional diffusion models for function spaces. arXiv preprint arXiv:2302. 10130, 2023.
[50] Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. In Proc. of the International Conference on Learning Representations (ICLR), 2023.
URL. https: /tpp//evie/eeeeeeee/e/ee wwtp//euum/niii//e -dPPTUBB92200011
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. of the Medical Image Computing and Computer- Assisted Intervention (MICCAI), pages 234-241. Springer, 2015.
[52] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. based generative modeling through stochastic differential equations. In Proc. of the International Conference on Learning Representations (ICLR), 2020.
[53] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proc. of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2818-2826, 2016. doi: 10.11099CVR.22066.300
[54] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural 2011.
computation,
13

---

[55] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. volume 2. MIT press Cambridge, MA, 2006.
[56] George Wynne and Andrew Duncan. A kerne two-sample test for functional data. Journal of Machine Learning Research (JMLR), 23(73): 2022.
[57] Eunbi Yoon, Keehun Park, Jinhyeok Kim, and Sungbin Lim. Score-based generative models with lévy processes. In NeurlPS Workshop on Score-Based Methods, 2022. URL https:
/ http:///www.aeeeeoootuodrr/aaaaa
[58] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN; construction of large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365,, 2015. URL http:///rzian org/ abs/ 11/./.500.000 03365.
[59] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li, On the continuity of rotation representations in neural networks. In Proc. of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages $745-575, 2019.
14


Reviews:
[{"review_id": "YKL4fu4TNT", "reviewer": "Reviewer_sBaK", "summary": "The paper proposes a time-dependent stochastic generative model that can be implemented in Hilbert spaces. The presented method unifies diffusion processes (commonly defined in diffusion models) with a more general stochastic evolution equation. A stochastic evolution equation is defined with satisfied conditions which mimics the framework of the forward process of the SDE equation derived for diffusion models. A time-reversal formula is derived from forward stochastic evolution equation based on the Kolmogorov equation and closely resembles the SDE equation of the score-based model.\n\nSimilar to score-based models, the score function is learned by a deep learning model with Fourier layers in the network. The model shows promising qualitative and quantitative results for unique tasks (e.g., motion generation, infinite dimension),", "questions": "- Do you think this proposed method would naturally perform better than the DDPM baseline model for audio datasets due to the Fourier transform layers?\n- What are some real world applications for infinite dimension tasks? Could these relate to Gigapixel images and multimodal datasets?", "limitations": "- The limitations were properly addressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is well written/organized and easy to follow. The author does a good job introducing related theoretical information.\n- The figures are mostly helpful for visualization.\n- Equations are well structured, but it may be helpful if there is a brief background or insight on some claims (e.g., line 124 the reason behind choosing the class of smooth cylinder functions, Line 109 basic description of Fomin differentiable probability measures).\n- In general, a very theoretically sound paper.\n", "weaknesses": "- It would be helpful if some of the datasets are briefly explained.\n- Many of the figure descriptions are not very clear (e.g., Figure 1 (a) and I didn't quite understand (b), add title for Figure 3 images)\n- It would be nice if some intuition or insight could be given on why the baseline DDPM model cannot perform well with some of these tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. In the camera-ready version, we will supplement additional explanations regarding the dataset, Figures 1 and 3, and the theoretical background such as smooth cylinder functions and Fomin differentiable measures, which were briefly discussed due to the space limit.\n\n> ***It would be nice if some intuition or insight could be given on why the baseline DDPM model cannot perform well with some of these tasks.***\n> \n\nOne of the main advantages of the proposed method is its ability to determine the solution space of stochastic equations driven by a Hilbert space-valued Wiener process formulated by a specific kernel function, which can contain the characteristics of target functionals such as smoothness or curvature information by optimizing the kernel function from the training data, e.g., a model selection method for a Gaussian process, to benefit from sampling efficiency. On the contrary, DDPM is based on white noise which ignores correlated information, which may deteriorate the sample efficiency when modeling smooth functions. In particular, human motion is represented as a sequence of skeletons where a skeleton consists of a sequence of 3D rotation matrices (or, equivalently, a product of SO(3) groups) whose movement should be continuous and smooth, and this motivates us to propose a diffusion model on function spaces to synthesize human motions. Consequently, our motion generation results well examined our hypothesis in that our proposed method (HDM-SDE) successfully models the human motion distribution, whereas the baseline model fails. We would like to note that coefficient scheduling is also a critical component in our implementation, distinct from other diffusion models on function spaces.\n\n> ***Do you think this proposed method would naturally perform better than the DDPM baseline model for audio datasets due to the Fourier transform layers?***\n> \n\nEven though we did not try to experiment with our methodology in the audio dataset, we believe it could be applied for future research. As mentioned earlier, DDPM destroys correlated information in the signal space, requiring more data for training the score model. With the proposed method, HDM-SDE, correlated information can be preserved through Hilbert space-valued Wiener process and Fourier transform layers in Fourier Neural Operator (FNO). However, there have been some claims that FNO is limited to restoring high-frequency information, so appropriate architecture design will be necessary.\n\n> ***What are some real world applications for infinite dimension tasks? Could these relate to Gigapixel images and multimodal datasets?***\n> \n\nIn the context of Infinite-dimensional tasks, real applications can be proposed in two main scopes. Firstly, as suggested in Section 4.1, it demonstrates the possibility of sampling from functional data spaces with multi-modality properties even though the proposed experiment is limited to 1D. Additionally, resolution-independent sampling is also possible since HDM-SDE utilizes the time-conditioned FNO. Secondly, for motion generation tasks requiring smooth trajectory sampling in higher dimensional settings, the proposed HDM-SDE model shows better performance than DDPM regarding sample efficiency, as shown in Section 4.2.\n\nAlthough HDM-SPDE experiments with image generation with the U-Net architecture for comparison with IHDM in a finite-dimensional setting, it can be converted to an infinite-dimensional setting using FNO architectures, similar to HDM-SDE. This direction is planned to be proposed as future research in subsequent studies."}, {"review_id": "ciyGu43qtX", "reviewer": "Reviewer_yicF", "summary": "This work studies continuous-time diffusion processes in Hilbert spaces, and their applications to generative modeling. In particular, a time-reversal formula is derived for Hilbert-space SDEs, allowing the continuous-time diffusion model framework to be generalized to infinite-dimensional Hilbert spaces. The proposed methodology is evaluated on several tasks, including generation of images and time-series.\n", "questions": "- Time reversals of infinite-dimensional diffusions have been studied previously [1-2] -- how does the main result (Theorem 2.1) relate to these existing works?\n- Could you elaborate on why you believe the DDPM model performed so poorly in Table 1, given that previous works [3-4] have successfully developed function-space generalizations of this model? E.g. it was unclear if a white-noise DDPM model was used or a trace-class DDPM model was used (as discussed  in [3-4]).\n\n[1] Time reversal for infinite-dimensional diffusions. Annie Millet, David Nualart, Marta Sanz, 1989.\n\n[2] Time reversal of infinite-dimensional diffusions. H. Follmer, A. Wakolbinger, 1986. \n\n[3] Diffusion generative models in infinite dimensions. Gavin Kerrigan, Justin Ley, Padhraic Smyth, 2023.\n\n[4] Score based diffusion models in function space. Jae Hyun Lim et al., 2023. ", "limitations": "The limitations of the work are appropriately addressed within the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The exposition is clear and the paper is generally well-written throughout.\n- This work adds to the growing literature on function-space diffusion models, and lays a solid theoretical foundation for future work in this area. The results are likely to be of significant interest to the community. \n- Section 2.3 proposes a novel continuous-time generalization of the IHDM model (Riassanen et al., 2023) which shows strong empirical performance (Section 4) compared to the discrete-time model. \n- The proposed method shows favorable performance on a functional generation task when compared to several existing functional baselines. \n- To the best of my knowledge the proofs are correct, and the empirical claims are well-supported by the provided evidence.", "weaknesses": "- The paper could be better contextualized within the existing literature on this topic, i.e. there is no related work section. E.g. the authors briefly mention several closely related works on generating functional data (Line 30) but do not detail the differences between this submission and those existing works. See also the question below regarding time-reversals of infinite dimensional diffusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s constructive feedback.\n\n> ***The paper could be better contextualized within the existing literature on this topic, i.e. there is no related work section.***\n\nIn fact, we compare our approach to the existing literature in Appendix C.1, but we do not have a corresponding section in the main text due to lack of space. We promise to elaborate on it in more detail in the revised manuscript.\n\n> ***Time reversals of infinite-dimensional diffusions have been studied previously [1-2] -- how does the main result (Theorem 2.1) relate to these existing works?*** \n\nThe work of [Follmer & Wakolbinger, 1986] considers a sequence of independent standard Wiener processes, utilizing an identity operator as the diffusion coefficient. This study represents a specific instance of the more general framework outlined in [Millet et al., 1989]. In Section 2.3.1 of our paper, we reference Theorem 5.3 from [Millet et al., 1989], which leverages the theory of stochastic calculus of variations to identify the drift operator (eq. 19) for SDEs in infinite dimensions. Additionally, in Appendix C.2, we demonstrate that this result can be viewed as a special case of our main result (eq. 16). \n\nIt is also worth noting that although [Millet et al., 1989] offer a satisfactory time-reversal theory for SDEs in infinite dimensions, it does not readily lend itself to the creation of a unified framework capable of encompassing diffusion models based on stochastic PDEs (eq. 21) described in Section 2.3.2. In contrast, our approach is grounded in the study of variational approach to stochastic evolution equations [Krylov & Rozovskii, 1981, 2007] and forward and backward Kolmogorov equations with functional derivatives [Dalecky & Fomin, 1991; Belopolskaya & Dalecky, 2012], which serves as a more established approach to identifying relations between time-dependent evolution operators and invariant measures of stochastic equations in Hilbert spaces [Y. Belopolskaya, 2020]. It is noteworthy that proof of Theorem 2.1 (Appendix B.2) can be viewed as a generalization of the classical techniques employed by [B.D.O Anderson, 1982] and [Haussmann & Pardoux, 1987] in finite-dimensional spaces to general Hilbert spaces; consequently it is able to fully recover the SDE framework proposed from [Song et al., 2021].\n\n> ***Could you elaborate on why you believe the DDPM model performed so poorly in Table 1, given that previous works [3-4] have successfully developed function-space generalizations of this model? E.g. it was unclear if a white-noise DDPM model was used or a trace-class DDPM model was used (as discussed in [3-4]).***\n\nIn our 1D function generation experiment, we utilized a white-noise DDPM [Ho et al., 2020], as shown in Table 1. White-noise DDPM models show limited adaptability to variable discretizations of the data as discussed in [Lim et al., 2023; Kerrigan et al., 2023]. By potentially ignoring correlated information, white-noise DDPM could lead to a deterioration in sample efficiency, particularly when modeling smooth functions. In this point of view, trace-class DDPM models [Lim et al., 2023; Kerrigan et al., 2023] probably show comparable results with HDM-SDE.\n\nIn our analysis, we examined the performance of the trace-class Wiener process, specifically considering $\\mathbf{B}_t = I, \\mathbf{G}_t = \\sqrt{2}I$ case, as discussed in [Section 4.1, Lim et al., 2023], aligns closely with the results of SP-SGM [Phillips et al., 2022]. To further elucidate the comparison, we conducted additional 1D function generation experiments, utilizing the infinite-dimensional Langevin dynamics method as proposed by [Lim et al., 2023]. The power(%) of a kernel two-sample test on the Quadratic and Melbourne datasets are reported as follows:\n\n- Quadratic: $6.9 \\pm 0.5$, Melbourne: $4.0 \\pm 0.3$\n\nThe above result is comparable to SP-SGM (Quadratic: $5.4 \\pm 0.7$, Melbourne: $5.3 \\pm 0.7$) but are generally less favorable than HDM-SDE (Quadratic: $4.2 \\pm 0.3$, Melbourne: $0.2 \\pm 0.1$). We claim that this discrepancy in performance is attributable to the scheduling strategy; hence our results underscore the significance of the dual drift formula presented in Theorem 2.1. \n\nHowever, it should be noted that the above experiment does not undervalue previous works since [Lim et al., 2023; Kerrigan et al., 2023] also present **discrete-time models** considering time scheduling, which recovers the DDPM framework [Ho et al., 2020]. A key difference between previous works and ours is that we consider a **continuous-time model**, which fully recover SDEs (VP-SDE and sub-VP-SDE) from [Song et al., 2021] and can utilize time-dependent coefficients, thereby enabling cosine-beta scheduling.\n\n---\n[B. D. O Anderson, 1982] Reverse time diffusion equation models. *Stochastic Processes and their Applications,* 1982.\n\n[Follmer & Wakolbinger, 1986] Time reversal of infinite-dimensional diffusions. *Stochastic Processes and their Applications*, 1986.\n\n[Haussmann & Pardoux, 1987] Time reversal of diffusions. *Annals of Probability*, 1987.\n\n[Ho et al., 2020] Denoising diffusion probabilistic models. *NeurIPS*, 2020.\n\n[Kerrigan et al., 2023] Diffusion generative models in infinite dimensions. *AISTATS*, 2023.\n\n[Lim et al., 2023] Score based diffusion models in function space. *arXiv*, 2023.\n\n[Millet et al., 1989] Time reversal for infinite-dimensional diffusions. *Probability theory and related fields,* 1989.\n\n[Phillips et al., 2022] Spectral diffusion processes. NeurIPS Workshop on SBM, 2022.\n\n[Song et al., 2021] Score-based generative modeling through stochastic differential equations. *ICLR*, 2021.\n\n[Dalecky & Fomin, 1991] Measures and differential equations in infinite-dimensional space, 1991.\n\n[Belopolskaya & Dalecky, 2012] Stochastic equations and differential geometry, 2012.\n\n[Krylov & Rozovskii, 1981, 2007] Stochastic evolution equations, 1981 (Russian) 2007 (English)."}, {"review_id": "6Ic2PdQVx1", "reviewer": "Reviewer_gdSN", "summary": "The authors propose a framework called the Hilbert Diffusion Model (HDM), which generalizes and unifies the time-reversal diffusion generative models with stochastic differential equations (SDEs) in Hilbert spaces. This model enables SDEs to be used in infinite-dimensional settings and to include stochastic partial differential equations (SPDEs) in Euclidean settings. The HDM's performance is verified against several datasets, outperforming existing diffusion models formulated in function spaces. It's particularly successful in human motion synthesis, where it applies the Wiener process in the Hilbert space.", "questions": "See weaknesses.", "limitations": "The authors already discussed the current limitations of their model (e.g., limited performance on image generation).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper effectively aims to expand the time-reversal diffusion generative model into Hilbert space. The authors also provide a comprehensive background and overview of Stochastic Differential Equations (SDEs) in Hilbert space, as illustrated in sections 2.1 and 2.2 before the time-reversal theorem.\n\nThe paper is very well-structured, featuring clear and logical derivations. The authors deserve commendation for their efforts in:\n\n- Establishing the time-reversal diffusion property in Hilbert space (as shown in Theorem 2.1 and Appendix B.2) through the use of infinitesimal generators;\n- Introducing and unifying the time-reversal diffusion in Hilbert space (as seen in Appendix C.2);\n- Employing parabolic SPDEs as functional examples and presenting the connection between parabolic SPDEs and the stochastic process used in IHDM (refer to Section 2.3.2).\n\nThe mathematical rigor demonstrated throughout the paper is thorough and cohesive.", "weaknesses": "Major Points:\n\n- This work is not the first to extend time-reversal diffusion models to infinite-dimensional space, and the concept of time-reversal diffusion on Hilbert space is already addressed in [1][2][3]. As these are mentioned in your citations, it would be beneficial to delve into the relationship between HDM and these models. For instance, the authors might discuss whether they are particular cases of HDM. This further elaboration could help clarify how HDM contributes to the field beyond what has been covered in these previous works.\n\n- It would be advantageous to incorporate an evaluation comparison with [1][2][3], given their direct relevance. I would be interested in seeing, for example, how your model compares to [2] using the AEMET dataset[2] or how their model performs on the datasets you're presenting.\n\nMinor Points:\n\n- For the convenience of readers, it would be more effective if the authors could present the experimental results of image generation in tables instead of embedding them in the text.\n\n- In terms of image generation, the performance of SPDE does not match up to DDPM or VE-SDE. However, since this paper primarily concentrates on generation in Hilbert space - an issue already noted in the limitations - it may not be necessary to highlight this discrepancy in this context.\n\n\n[1] Pidstrigach, J., Marzouk, Y., Reich, S., & Wang, S. (2023). Infinite-dimensional diffusion models for function spaces. arXiv preprint arXiv:2302.10130.\n\n[2] Kerrigan, G., Ley, J., & Smyth, P. (2022). Diffusion generative models in infinite dimensions. arXiv preprint arXiv:2212.00886.\n\n[3] Lim, J. H., Kovachki, N. B., Baptista, R., Beckham, C., Azizzadenesheli, K., Kossaifi, J., ... & Anandkumar, A. (2023). Score-based diffusion models in function space. arXiv preprint arXiv:2302.07400.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s valuable feedback. \n\n> ***This work is not the first to extend time-reversal diffusion models to infinite-dimensional space, and the concept of time-reversal diffusion on Hilbert space is already addressed in [1][2][3]. As these are mentioned in your citations, it would be beneficial to delve into the relationship between HDM and these models. For instance, the authors might discuss whether they are particular cases of HDM. This further elaboration could help clarify how HDM contributes to the field beyond what has been covered in these previous works.***\n\nWe appreciate the reviewer's suggestion. There is a theoretical difference between the proposed method and the existing methodologies [Kerrigan et al., 2023, Lim et al., 2023, Pidstrigach et al., 2023]. We promise to elaborate on it in more detail. Please see the **Author Rebuttal by Authors** for more details. We emphasize that the proof of Theorem 2.1 (Appendix B.2) can be viewed as a generalization of the classical techniques employed by [B.D.O Anderson, 1982] and [Haussmann & Pardoux, 1987] in finite-dimensional spaces to general Hilbert spaces; consequently it is able to fully recover the SDE framework proposed from [Song et al., 2021].\n\n> ***It would be advantageous to incorporate an evaluation comparison with [1][2][3], given their direct relevance. I would be interested in seeing, for example, how your model compares to [2] using the AEMET dataset[2] or how their model performs on the datasets you're presenting.***\n\nThanks for your suggestion. We experiment with our method on the AEMET dataset presented in [Kerrigan et al., 2023] for the further comparison. For HDM-SDE, we use the SE kernel as presented in the main paper instead of the Mátern kernel used in [Kerrigan et al., 2023], while the rest of the hyperparameter settings are the same. The MSE performance of HDM-SDE is reported as follows:\n\n- [Kerrigan et al., 2023] (Best result): Mean: 0.0152, Variance: 1.0748, Autocorr: 2.551e-06\n- HDM-SDE: **Mean: 0.0001**, **Variance: 0.0293**, **Autocorr: 1.1272e-06**\n\nAlso, the power(%) of two-sample test for HDM-SDE records a value of $0.48\\pm0.95$, implying that the statistical difference between the generated and target datasets is negligible and hardly discernable. We believe these experimental results may compare an empirical difference between the proposed and existing methods [Kerrigan et al., 2023].\n\nIn our analysis, we examined the performance of the trace-class Wiener process, specifically considering **constant-time coefficients** (i.e., $\\mathbf{B}_t = I$ and $\\mathbf{G}_t = \\sqrt{2}I$, as discussed in [Section 4.1, Lim et al., 2023]), aligns closely with the results of SP-SGM [Phillips et al., 2022]. To further elucidate the comparison, we conducted additional 1D function generation experiments, utilizing the infinite-dimensional Langevin dynamics method as proposed by [Lim et al., 2023]. The power(%) of a kernel two-sample test on the Quadratic and Melbourne datasets are reported as follows:\n\n- Quadratic: $6.9 \\pm 0.5$, Melbourne: $4.0 \\pm 0.3$\n\nThe above result is comparable to SP-SGM (Quadratic: $5.4 \\pm 0.7$, Melbourne: $5.3 \\pm 0.7$) but are generally less favorable than HDM-SDE (Quadratic: $4.2 \\pm 0.3$, Melbourne: $0.2 \\pm 0.1$). We claim that this discrepancy in performance is attributable to the scheduling strategy; hence our results underscore the significance of the dual drift formula presented in Theorem 2.1.\n\nHowever, it should be noted that the above experiment does not undervalue previous works since [Lim et al., 2023; Kerrigan et al., 2023] also present discrete-time models considering time scheduling, which recovers the DDPM framework [Ho et al., 2020]. A key difference between previous works and ours is that we consider a continuous-time model, which fully recover SDEs (VP-SDE and sub-VP-SDE) from [Song et al., 2021] and can utilize time-dependent coefficients, thereby enabling cosine-beta scheduling.\n\n> ***For the convenience of readers, it would be more effective if the authors could present the experimental results of image generation in tables instead of embedding them in the text.***\n> \n\nThanks for the suggestion. We will try to make the changes in the revised manuscript.\n\n> ***In terms of image generation, the performance of SPDE does not match up to DDPM or VE-SDE. However, since this paper primarily concentrates on generation in Hilbert space - an issue already noted in the limitations - it may not be necessary to highlight this discrepancy in this context.***\n> \n\nAs mentioned by the reviewer, there is room for improvement in the 2D image generation task, so we also noted in the limitation section. The proposed HDM-SPDE focuses on building a bridge between the discrete-time model presented in IHDM [Rissanen et al., 2023] and the continuous-time models, so it is different from the generative models used in DDPM [Ho et al., 2020], and VP-SDE [Song et al., 2023]. We will address these issues in future research.\n\n---\n\n[B.D.O Anderson, 1982] Reverse time diffusion equation models. *Stochastic Processes and their Applications,* 1982.\n\n[Haussmann & Pardoux, 1987] Time reversal of diffusions. *Annals of Probability*, 1987.\n\n[Ho et al., 2020] Denoising diffusion probabilistic models. *NeurIPS*, 2020.\n\n[Kerrigan et al., 2023] Diffusion generative models in infinite dimensions. *AISTATS*, 2023.\n\n[Lim et al., 2023] Score based diffusion models in function space. *arXiv*, 2023.\n\n[Pidstrigach et al., 2023] Infinite-dimensional diffusion models for function spaces. *arXiv*, 2023.\n\n[Rissanen et al., 2023] Generative modelling with inverse heat dissipation. *ICLR*, 2023.\n\n[Song et al., 2021] Score-based generative modeling through stochastic differential equations. *ICLR*, 2021."}, {"review_id": "AOvMonS6BL", "reviewer": "Reviewer_8erp", "summary": "In this paper, the authors introduce Hilbert Diffusion Model (HDM). This generative model is a generalization of diffusion models to infinite dimensional state space (in this case Hilbert spaces) through the use of generalized SDEs. The authors extend the notion of score and time-reversal to this setting. Their framework can be applied to infinite dimensional diffusion models and to Stochastic Partial Differential Equations (SPDEs). They propose to use Fourier Neural Operator (FNO) as a basis of the architecture of the neural network to learn the score, similar to other infinite dimensional diffusion models. They showcase the efficiency of their model on function and image generation tasks.", "questions": "* In Section 2.1 can the authors define what is a Gaussian measure in a Hilbert space\n\n* We define Cameron-Martin space --> We define \"the\" Cameron-Martin space (?)\n\n* In Section 2.1 it seems that there is no need for the operator $\\Lambda$ to be positive. However, the authors take its inverse. Am I missing something here? In Example 2.1 what happens if $\\Sigma$ is not invertible?\n\n* The authors state \"the time-reversal process is well-studied for diffusion processes in abstract settings\" and cite [1]. Then, could the authors simply use the results from [1] to establish Theorem 2.1? [1] uses only the carre du champs operator and so, since the authors also describe things in terms of infinitesimal generator, it seems to me that their main result could be applied. Again, I might be missing something.\n\n* For parabolic SPDE what is the motivation behind Equation (21)? It is not clear to me what is the role of the power of $k$ on the terms in front of the white noise Brownian motion.  Also in Equation (22) it seems that $1+k/2$ has turned into $k/2$. In Equation (23) where is the score? I think the authors should give more explanations in that section. \n\n* The losses in Equation (25) and Equation (26) could be improved by using weighting terms or using $v$-prediction [2]. Have the authors considered this?\n\n* It is not clear to me why the authors do not choose a FNO for the SPDE application, considering that the Laplacian is diagonalized in a Fourier basis.\n\n* I am bit confused by the experiments on HumanML3D. DDPM seems to be performing extremely poorly in that setting. I am quite surprised by that and especially the discrepancy with the model proposed by the authors. Is there any good reason for that?\n\n* I don't understand the \"Controllable Coarse-to-FIne Generation\" paragraph. How do authors solve inverse problems with a model trained for generative modeling? Is there a guidance function somewhere? Are the degraded images passed as input to the diffusion model where the forward has been chosen appropriately? More details are needed.\n\n[1] Cattiaux et al. (2021) -- Time reversal of difusion processes under finite entropy condition\n\n[2] Salimans et al. (2022) -- Progressive Distillation for fast sampling of diffusion models", "limitations": "Adressed in the conclusion.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper presents a strong theoretical framework for the theory of Hilbert diffusion models. The ideas are very well-presented (even though I think the authors could give more details about their SPDE application, see my Questions).\n\n* The experiments are convincing. In particular, the model seems to perform better than other models which are not based on a noising perturbation but on other degradations such as [1].\n\n[1] Rissanen et al. (2023) -- Generative modelling with inverse heat dissipation", "weaknesses": "* One of my main remark is that there is little discussion on the relation between the current work and the existing literature on diffusion models. For instance, what is the novelty in the derived theorem 2.1? It seems to me that [1, Section 3.4] already studied the time-reversal of infinite diffusion models. Similarly, the framework of Cameron-Martin theory to derive infinite dimensional Brownian motion and Gaussian measures was already considered in [1,2]. What is the difference between the current work and the works [1,2,4,6] from a theoretical point of view? From a practical point of view, the authors should also specify how different their method is from existing works. As they point out the Fourier Neural Operator (FNO) was already used in several works [2,4,5]. I think the authors omit to discuss [3] which not only use the FNO but found that Galerkin operators worked well. At the end of the day what are the practical differences with [2,4,5]? Finally, could the authors discuss the relationship between their model and other infinite dimensional approaches such as [7,8]?\n\n* My second remark is more general and has to do with the motivation of the work. It is not clear to me why, from a practical perspective, would anyone want to use infinite dimensional diffusion models? When it comes to implementation the space is going to be discretised and advantages that are usually cited to be given by the infinite dimensional setting (infinite resolution) can also be incoporated in existing finite dimensional diffusion models (by interpolating on a sub-pixel grid for instance). For example the fact that we consider continuous-time diffusion models could also be criticized since everything is then discretized at implementation time but it turns out that this formulation yields some advantages (gives ideas for better samplers, allows for simple integration of the Ornstein-Uhlenbeck to sample without Markov chains, allows for log-likelihood computation using ideas from continuous normalizing flows). Could the authors present such motivation in the case of infinite dimensional diffusion model? I think coming up with an example where infinite dimensional diffusion models are needed would be key to motivate the study.\n\n* My third remark concerns the motivation behind other choices of perturbation than a Gaussian noising. Could the authors explain why not using a Gaussian noising might be beneficial? While it is possible to do so I have seen little practical justification of this so far.\n\n[1] Pidstrigach et al. (2023) -- Infinite-dimensional diffusion models for function space\n\n[2] Kerrigan et al. (2022) -- Diffusion generative models in infinite dimensions\n\n[3] Bond-Taylor and Willcocks (2023) -- Infinite Diff: Infinite Resolution Diffusion with Subsampled Mollified States\n\n[4] Lim et al. (2023) -- Score-based diffusion models in function space\n\n[5] Hagemann et al. (2023) -- Multilevel diffusion: Infinite dimensional score-based diffusion models for image generation\n\n[6] Franzese et al. (2023) -- Continuous-time functional diffusion processes\n\n[7] Dutordoir et al. (2022) -- Neural Diffusion Processes\n\n[8] Philips et al. (2022) -- Spectral Diffusion Processes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, [Bond-Taylor & Willcocks, 2023] is a paper we had not cited at the time of submission, and we are very grateful to the reviewer for pointing it out. For the first main remark, Please see the **Author Rebuttal by Authors** for more details. \n\nFor the second remark, real applications can be proposed in two main scopes in the context of infinite-dimensional tasks. Firstly, motivated from a Functional Data Analysis (FDA), proposing a generative model in function space is valid from a sample efficiency perspective. Suppose the functional sample should be smooth or periodic. In that case, we can imagine a probability measure defined on the corresponding function spaces with desired properties. As suggested in Section 4.1, the proposed method demonstrates better sampling efficiency from functional data spaces with multi-modality. The reviewer also can see this in the motion generation task requiring smooth trajectory sampling in higher dimensional settings, as shown in Section 4.2.\n\nAlso, we promise to fix some errata below. \n\n> ***In Section 2.1 it seems that there is no need for the operator to be positive. However, the authors take its inverse.***\n\nThanks for pointing that out. I think it was accidentally omitted due to space limitations. The theory presented in this paper assumes non-negativity, but Example 2.1 requires to be positive-definite. We will fix it in the revised manuscript.\n\n> ***The authors state the time-reversal process is well-studied for diffusion processes in abstract settings and cite [1]. Then, could the authors simply use the results from [1] to establish Theorem 2.1?***\n\nThanks for the wonderful question. In fact, it may be possible to partially derive the result of Theorem 2.1 by applying Theorem 3.17 in [Cattiaux et al., 2021], since their approach is also based on the Integration by Part (IbP) formula. However, to specifically complete the dual-drift formula for the time-dependent operator, we need to compute the exact score operator using the calculations in Appendix B.1. For time-independent operators, this derivation is relatively trivial, but for time-dependent operators, we need the Itō’s formula in Hilbert space. Therefore, we must discuss the functional derivative, which naturally leads to the proposed Kolmogorov operators. In this sense, we claim that Theorem 2.1 can be viewed as a generalization of [B.D.O Anderson, 1982] and [Haussmann & Pardoux, 1987] in finite-dimensional spaces to general Hilbert spaces.\n\n> ***For parabolic SPDE what is the motivation behind Equation (21)? It is not clear to me what is the role of the power of on the terms in front of the white noise Brownian motion.***\n\nThanks for pointing out this crucial point. It seems that an errata occurred while fixing the formula due to a lack of space. First, the role of $k$ is the order of the Bessel potential operator that transforms the Wiener process into smooth function space (Sobolev space). For $d \\geq 2$, parabolic SPDEs become an ill-posed problem without such an operator and suffer regularity problems [N. V. Krylov, 1999]. Please also see [Section 4.2, Lim et al. 2023].\n\nWe need to modify eq. (22) as follows:\n\n$$ \\text{d}\\hat{u}_t = -b(t)\\left(\\sigma\\mathbf{I}-\\mathbf{D}\\right)\\hat{u}_t \\text{d}t+\\sqrt{2b(t)}(\\sigma\\mathbf{I}-\\mathbf{D})^{-\\frac{k}{2}}\\text{d}\\widehat{\\mathcal{W}}_t $$\n\nTherefore, eq. (23) also needs to be modified by adding the vector logarithmic derivative. The derivation of the exact formula is derived in Appendix C.3 (some minor errata therein, and we will fix them in the camera ready-version). Again, we deeply appreciate for pointing errata.\n\n> ***The losses in Equation (25) and Equation (26) could be improved by using weighting terms or using $v$-prediction [2]. Have the authors considered this?***\n> \n\nThanks for the suggestion. We did not use $v$-prediction nor weighting terms in the experiments to compare with other approaches in functional generation. It is expected to bring performance gain in the image generation task compared with IHDM [Rissanen et al., 2023].\n\n> ***It is not clear to me why the authors do not choose a FNO for the SPDE application, considering that the Laplacian is diagonalized in a Fourier basis.***\n> \n\nAlthough HDM-SPDE experiments with image generation with the U-Net architecture for comparison with IHDM in a finite-dimensional setting, as reviewer mentioned, it can be converted to an infinite-dimensional setting using FNO architectures, similar to HDM-SDE. This direction is planned to be proposed as future research.\n\n> ***I am bit confused by the experiments on HumanML3D. Is there any good reason for that?***\n> \n\nOne of the main advantages of the proposed method is its ability to determine the solution space of stochastic equations driven by a Hilbert space-valued Wiener process formulated by a specific kernel function, which can contain the characteristics of target functionals such as smoothness or curvature information by optimizing the kernel function from the training data, e.g., a model selection method for a Gaussian process, to benefit from sampling efficiency. On the contrary, white-noise DDPM is based on white noise which ignores correlated information, which may deteriorate the sample efficiency when modeling smooth functions. \n\n> ***I don't understand the \"Controllable Coarse-to-FIne Generation\" paragraph. More details are needed.***\n> \n\nIndeed, we have explained the parts of the coarse-to-fine generation in Appendix D.4. To solve the inverse problems, we apply the inpainting technique described in Appendix I.2 of (Song et al., ICLR 2021] to restore masked parts of the given image using the pre-trained score model without a guidance function. We will supplement more details in the revised manuscript.\n\n---\n\n[Cattiaux et al., 2021], Time reversal of diffusion processes under finite entropy condition, 2021.\n\n[N. V. Krylov, 1999], An analytic approach to SPDEs, 1999.\n"}]

Author rebuttal:
We appreciate the reviewers' valuable comments. Since some reviewers question the theoretical difference between the proposed method and the existing methodologies, e.g., [Kerrigan et al., 2023, Lim et al., 2023, Pidstrigach et al., 2023], etc., we would like to address those issues. We promise to elaborate on it in more detail in the camera-ready version.

First, [Kerrigan et al., 2023] introduced a **discrete-time model** in infinite-dimensional space, effectively generalizing the DDPM [Ho et al., 2020]. This work serves as one of the key inspirations for our study. Within that paper, the authors acknowledged a remaining challenge for functional diffusion models, stating: *'A remaining challenge for functional diffusion models is to consider the continuous-time limit and elucidating connections with score-based methods [Song et al., 2021].'* It is worth noting that the our main result, Theorem 2.1, provides a comprehensive response to this challenge. 

Next, the works of [Lim et al., 2023; Pidstrigach et al., 2023] focus on continuous-time models, contrary to [Kerrigan et al., 2023]. Both methods and our approach start on a standard theoretical setting, utilizing Cameron-Martin space and Gaussian measure theory on Hilbert spaces. It is crucial to recognize, however, that the theoretical results in [Lim et al., 2023; Pidstrigach et al., 2023] are confined to SDEs with **constant-time coefficients** (i.e., $B_{t}=I$ and $G_{t}=\sqrt{2}I$, as detailed in [Section 3, Pidstrigach et al., 2023] and [Section 4.1, Lim et al., 2023]). In [Section 4.4, Lim et al., 2023], the multiple noise scale case is revealed, yet it is constrained to discrete-time models only. It is because their approach is largely grounded at theory of linear semigroups [Da Prato & Zabczyk, 2014], which is applicable to a wide range of differential equations but has a limitation when dealing with variable coefficients. [Hagemann et al., 2023] proposes using a discretized reverse process with time-dependent coefficients and [Franzese et al.,2023] considers constant-time diffusion coefficient based on [Millet et al., 1989], which do not fully recover the SDE framework in [Song et al., 2021]. [Phillips et al., 2022; Dutordoir et al., 2022] are also infinite-dimensional approaches, but they differ significantly from our methodology in theoretical aspects. [Dutordoir et al., 2022] is a discrete-time model proposed to generalize neural processes, but it utilizes white noise, which causes an ill-posed problem in infinite-dimensional spaces. [Phillips et al., 2022] proposed a continuous-time model based on spectral decomposition and demonstrated its effectiveness on multi-modal datasets, but it does not include a general SDE framework and SPDE approach.

On the contrary, our approach is grounded in the study of variational approach to stochastic evolution equations [Krylov & Rozovskii, 1981, 2007] and forward and backward Kolmogorov equations with functional derivatives [Dalecky & Fomin, 1991; Belopolskaya & Dalecky, 2012], which serves as a more established approach to identifying relations between time-dependent evolution operators and invariant measures of stochastic equations in Hilbert spaces [Y. Belopolskaya, 2020]. It is noteworthy that proof of Theorem 2.1 (Appendix B.2) can be viewed as a generalization of the classical techniques employed by [B.D.O Anderson, 1982] and [Haussmann & Pardoux, 1987] in finite-dimensional spaces to general Hilbert spaces; consequently it is able to fully recover the SDE framework proposed from [Song et al., 2021], and can utilize time-dependent coefficients, thereby enabling cosine-beta scheduling. 

We also point out that we newly design the time-conditioned FNO (Figure 2 in Section 3) for HDM-SDE by adding a positional embedding and a linear layer to embed the time variable, which has a different structure from the original FNO architecture [Zongyi Li et al., 2020]. This modification is beneficial, especially for functional and motion-generation tasks, as shown in Tables 1 and A.1.

---

[B.D.O Anderson, 1982] Reverse time diffusion equation models. *Stochastic Processes and their Applications,* 1982.

[Y. Belopolskaya, 2020]. Invariant measures of diffusion processes in Hilbert spaces and Hilbert manifolds, 2020.

[Belopolskaya & Dalecky, 2012] Stochastic equations and differential geometry, 2012.

[Dalecky & Fomin, 1991] Measures and differential equations in infinite-dimensional space, 1991.

[Da Prato & Zabczyk, 2014] Stochastic equations in infinite dimensions, 2014.

[Dutordoir et al., 2022] Neural Diffusion Processes, *arXiv*, 2022.

[Franzese et al.,2023] Continuous-time functional diffusion processes. *arXiv*, 2023.

[Hagemann et al., 2023] Multilevel Diffusion: Infinite Dimensional Score-Based Diffusion Models for Image Generation. *arXiv*, 2023.

[Haussmann & Pardoux, 1987] Time reversal of diffusions. *Annals of Probability*, 1987.

[Ho et al., 2020] Denoising diffusion probabilistic models. *NeurIPS*, 2020.

[Kerrigan et al., 2023] Diffusion generative models in infinite dimensions. *AISTATS*, 2023.

[Krylov & Rozovskii, 1981, 2007] Stochastic evolution equations, 1981 (Russian) 2007 (English).

[Lim et al., 2023] Score based diffusion models in function space. *arXiv*, 2023.

[Millet et al., 1989] Time reversal for infinite-dimensional diffusions. *Probability theory and related fields,* 1989.

[Phillips et al., 2022] Spectral diffusion processes. *NeurIPS Workshop on SBM*, 2022.

[Pidstrigach et al., 2023] Infinite-dimensional diffusion models for function spaces. *arXiv*, 2023.

[Song et al., 2021] Score-based generative modeling through stochastic differential equations. *ICLR*, 2021.

[Zongyi Li et al., 2020] Fourier neural operator for parametric partial differential equations, *ICLR*, 2021.

Meta-review: This paper presents a novel theoretical framework called Hilbert Diffusion Models (HDM) that unifies time-reversal diffusion models and stochastic differential equations in infinite-dimensional Hilbert spaces.

The reviewers unanimously gave scores of 7 (Accept) in their initial reviews, with reasonable confidence levels. The authors provided an extensive rebuttal comprehensively addressing the reviewers' questions and concerns. In particular, they clarified the differences from related prior work, especially the ability to handle time-dependent coefficients in a continuous-time setting. They also provided additional motivation and use cases for infinite-dimensional models. The reviewers seemed satisfied with the clarifications and two reviewers increased their score accordingly.

The paper is technically solid, with reviewers praising the strong theoretical foundations. The exposition and experiments are also of good quality. The feedback from reviewers was largely constructive, aimed at improving the contextualization and motivation. I believe the authors have adequately addressed these concerns in their rebuttal.

In conclusion, I recommend accepting this paper given its solid technical contributions, promising empirical results on functional data and human motion modeling, and the authors' responsiveness to feedback about framing and comparisons. The reviews validate it being an impactful advancement worthy of inclusion and presentation at the conference. I expect the camera-ready version to be improved based on the reviewer discussion.
Decision: Accept (spotlight)