Title: Score-based Generative Modeling through Stochastic Evolution Equations in Hilbert Spaces
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative Model

Score-based Method

Diffusion Model
Abstract: Continuous-time score-based generative models consist of a pair of stochastic differential equations (SDEs)—a forward SDE that smoothly transitions data into a noise space and a reverse SDE that incrementally eliminates noise from a Gaussian prior distribution to generate data distribution samples—are intrinsically connected by the time-reversal theory on diffusion processes. In this paper, we investigate the use of stochastic evolution equations in Hilbert spaces, which expand the applicability of SDEs in two aspects: sample space and evolution operator, so they enable encompassing recent variations of diffusion models, such as generating functional data or replacing drift coefficients with image transformation. To this end, we derive a generalized time-reversal formula to build a bridge between probabilistic diffusion models and stochastic evolution equations and propose a score-based generative model called Hilbert Diffusion Model (HDM). Combining with Fourier neural operator, we verify the superiority of HDM for sampling functions from functional datasets with a power of kernel two-sample test of 4.2 on Quadratic, 0.2 on Melbourne, and 3.6 on Gridwatch, which outperforms existing diffusion models formulated in function spaces. Furthermore, the proposed method shows its strength in motion synthesis tasks by utilizing the Wiener process with values in Hilbert space. Finally, our empirical results on image datasets also validate a connection between HDM and diffusion models using heat dissipation, revealing the potential for exploring evolution operators and sample spaces.

Score-based Generative Modeling through Stochastic Evolution Equations in Hilbert Spaces

Abstract

Continuous-time score-based generative models consist of a pair of stochastic differential equations (SDEs)-a forward SDE that smoothly transitions data into a noise space and a reverse SDE that incrementally eliminates noise from a Gaussian prior distribution to generate data distribution samples-are intrinsically connected by the time-reversal theory on diffusion processes. In this paper, we investigate the use of stochastic evolution equations in Hilbert spaces, which expand the applicability of SDEs in two aspects: sample space and evolution operator, so they enable encompassing recent variations of diffusion models, such as generating functional data or replacing drift coefficients with image transformation. To this end, we derive a generalized time-reversal formula to build a bridge between probabilistic diffusion models and stochastic evolution equations and propose a score-based generative model called H ilbert D iffusion M odel (HDM). Combining with Fourier neural operator, we verify the superiority of HDM for sampling functions from functional datasets with a power of kernel two-sample test of 4.2 on Quadratic, 0.2 on Melbourne, and 3.6 on Gridwatch, which outperforms existing diffusion models formulated in function spaces. Furthermore, the proposed method shows its strength in motion synthesis tasks by utilizing the Wiener process with values in Hilbert space. Finally, our empirical results on image datasets also validate a connection between HDM and diffusion models using heat dissipation, revealing the potential for exploring evolution operators and sample spaces.

1 Introduction

Score-based generative models [52] have shown success in various domains, including the generation of images [25], texts [30], videos [26], motions [31]. [52] proposes a framework for continuous-time score-based generative models that use a pair of stochastic differential equations (SDEs); a forward SDE smoothly transitions data into noise space, and a reverse SDE incrementally eliminates noise from a Gaussian prior distribution to generate samples from the data distribution. Both SDEs have the same marginal distribution when their coefficients satisfy the time-reversal formula [1, 24], which is an ingenious application of the Kolmogorov-Fokker-Planck equation for diffusion processes.

A Unified Framework with Stochastic Evolution Equations in Hilbert Spaces

+

/u1D6FF

(a) SDE in Infinite-dimensions (HDM-SDE) :

(b) Parabolic Stochastic PDE (HDM-SPDE) :

B

/u1D461

=

-

/u1D44F

(

/u1D461

)

I

,

G

/u1D461

=

√

2

/u1D44F

(

/u1D461

)

I

B

/u1D461

=

/u1D44F

(

/u1D461

)(

Δ

-

/u1D70E

I

)

,

d

X

/u1D461

=

B

/u1D461

(

X

/u1D461

)

d

/u1D461

+

G

/u1D461

d

W

/u1D461

d

̂

X

/u1D461

=

[

-

B

/u1D447

-

/u1D461

(

̂

X

/u1D461

) +

S

(

/u1D447

-

/u1D461

,

̂

X

/u1D461

)]

d

/u1D461

+

G

/u1D447

-

/u1D461

d

W

/u1D461

Forward Evolution

P

data

P

prior

P

data

G

t

=

√

2

b

(

t

)(

σ

I

-

∆

)

-

k

2

(

t

=

T

)

(

t

=

0

)

(

t

=

0

)

Score Operator

(B) Utilizing a prior distribution generated from heat dissipation-type forward evolution, we can generate data distribution samples via Gaussian noise ( ) addition, akin to IHDM, and execute reverse evolution through a stochastic PDE. δ Recently, there has been an active proposal of variations of diffusion models, such as generating functional data [18, 19, 23, 29, 38, 48, 49] or replacing drift coefficients with image transformation [4, 27, 50], which cannot be covered by the original SDE framework [52] formulated in the Euclidean space. Motivated by this problem, we propose a unified framework for continuous-time score-based generative models by using stochastic evolution equations in Hilbert spaces [6, 14, 34, 35],

d X t = B t ( X t ) d t + G t d W t , d ̂ X t = [ -B T -t ( ̂ X t ) + S ( T -t, ̂ X t )] d t + G T -t d W t , (1) where ( X t , ̂ X t ) is a pair of forward and reverse stochastic processes with evolution operators ( B t , G t ) , W t is a Wiener process with values in some Hilbert space H , and S ( t, · ) : [0 , T ] × H → H is a score operator , which generalizes the notion of score functions in the Euclidean space. Since we can choose H and ( B t , G t ) in an abstract setting, stochastic equations (1) naturally extend the use of original SDEs to the infinite-dimensional setting and include stochastic partial differential equations (SPDEs), such as a parabolic equation driven by white noise (see Figure 1). Consequently, we can expand the SDE framework [52] in two aspects: sample space and evolution operator, enabling it to encompass recent variations of diffusion models in a continuous-time score-based generative model.

Contributions An essential key to identifying the relationship between the forward process X t and the reverse process ̂ X t in (1) is the derivation of time reversal formula with time-dependent operators B t and G t , which has been a challenging problem [23, 29, 38, 49] when we utilize coefficient schedulings [46, 52]. Contrary to previous approaches using semigroup theory [14], our approach is grounded in Kolmogorov equations with functional derivatives, well-studied in [2, 3, 7, 15-17], so that we can consider a wide scope of stochastic evolution equations with time-dependent operators.

We derive a generalized time-reversal formula and unify the SDE framework [52] with stochastic equations in Hilbert spaces. Consequently, we propose a class of continuous-time score-based generative models, H ilbert D iffusion M odel (HDM), which enables using SDEs in infinite-dimensional settings (HDM-SDE) and parabolic SPDEs (HDM-SPDE).

Combining with neural operators [37], HDM-SDE shows the superiority for sampling functions from functional datasets with multi-modalities compared to GP [55], NP [20], DDPM [25], and other diffusion models formulated in function spaces, NDP [18] and SP-SGM [48]. We also validate that HDM-SDE plays a crucial role in successful human motion synthesis compared to DDPM, by utilizing the Wiener process in the Hilbert space.

We build a bridge between parabolic SPDE and diffusion models using heat dissipation [50]. HDM-SPDE improves the performance in image generation as a continuous-time version of IHDM. Also, the proposed method shows the inductive bias in controllable coarse-to-fine image generation, e.g., inpainting, deblurring, and depixelation, without additional training.

<latexi sh

1_b

64="ZLP8FK

/gD

dzB

M

0Gm

9

yA

>

EJH

cjV

N

2p

TwS

Y

Q

I

O

W

o7k3

U

C

X

r

nR

+

5

u

v

f

q

<latexi sh

1_b

64="qN

jwGQ

T

fU

VWc8

g

/

p

>A

C

X

DL

I

FO3

J

zHR

Zk

v

Ym

5

u

SK

0

d

r

y

n

9

2

o

+

B

E

P

M

7

<latexi sh

1_b

64="

WkqU

YrRK9

w+f

Vz

O8

>A

Ec3

j

PL

N

F

pDZT

S

MG

CJ

5

B

m0o

d

v2

H

y

/I

uQ

X

n

g

7

<latexi sh

1_b

64="

fnzBp98

QZ

drw

S

3

DI

M

>A

ELX

cjVP

N

F

2

g

ou

R

CJ

5

K

G

W

m

O

Y

7

v

/

q0

+

y

k

H

T

U

<latexi sh

1_b

64="o

0G

U

V

u

9qw

>A

EPX

c

L

N

F

2

ZT

SmHJ

Q

B

C

y

n

d

/

7O

+I

K

D

zM

W

R

3

v

r

5

8

<latexi sh

1_b

64="

nLy57Cgw

EdF

8

c

>A

JX

jVP

N

2p

K

B

oI

R

Y

U

3

9

O

k

z

M+

q

Hf

r

/

u

v

W

D

<latexi sh

1_b

64="fuC

+

DMLGVU

980Y

rz

B

J

>A

TX

c

gI

FO3

Q

N

HR

Zk vj

m

5

W

d

q

o

/

K

7wE

S2

n

y

p

P

<latexi sh

1_b

64="AE

mDZ

vrCU

OX

2M

S

>

cjVPL

N

F

p

T

W

IQ

Y

w

q

G

7

9

d8g f0

y

B

R

+

3

/

u

H

5

J

<latexi sh

1_b

64="zqDf

R

KS

/oO

cHWIJ

E

T8

>A

3

jVP

N

B0X

y

n

kZU

F

Q

g

C

p

m

9

r2

d

u

M

w

7

5

Y

G

+

v

L

<latexi sh

1_b

64="T

co

Wm

B7D

yMJuVkKv gU

>A

E

3

j

PL

N

F

2p

Z

wS

H

C

q

0

Yn

R8d r5

/

+

G

9

QX

f

z

I

O

<latexi sh

1_b

64="K+U/83ZA

kv

B

q

dQ

5GNjC0

>

c

VF

M

EHW

JfTI

7

u

o

2

S

yO

p

R

X9

wz

P

r

g

D

n

Y

L

m

<latexi sh

1_b

64="ZLP8FK

/gD

dzB

M

0Gm

9

yA

>

EJH

cjV

N

2p

TwS

Y

Q

I

O

W

o7k3

U

C

X

r

nR

+

5

u

v

f

q

<latexi sh

1_b

64="qN

jwGQ

T

fU

VWc8

g

/

p

>A

C

X

DL

I

FO3

J

zHR

Zk

v

Ym

5

u

SK

0

d

r

y

n

9

2

o

+

B

E

P

M

7

2 A Foundation on Time-Reversal of Stochastic Evolution Equations

We first present a preliminary theory on the stochastic evolution equations in Hilbert space and introduce the logarithmic derivatives of probability measures to propose a Hilbert space valued score operator (see Section 2.2) which generalizes the notion of score functions in the Euclidean space. Then we derive the time-reversal formula (Theorem 2.1), which is a key to obtain the time-reversal of the diffusion processes by using the Kolmogorov equations in Hilbert space.

2.1 Preliminaries: Stochastic Evolution Equations in Hilbert Spaces

In preliminaries, we introduce elements of stochastic evolution equations in Hilbert spaces. See B.1 or we refer [5, 6, 14, 34, 35] for basic setting.

Gelfand Triple and Cameron-Martin Space Let H be a separable Hilbert space with the H -norm defined by the inner product ‖ · ‖ H = √ 〈 · , · 〉 H . Let L ( H ) denotes the set of all bounded linear operators on H . We assume there exists a Radon centered Gaussian measure λ on the Borel σ -field B ( H ) with the nonnegative, symmetric, self-adjoint, and the finite trace covariance operator Λ ∈ L ( H ) , hence there exists eigensystem { ( λ ( /lscript ) , φ ( /lscript ) ) ∈ R + × H : /lscript ∈ N } such that Λ ( φ ( /lscript ) ) = λ ( /lscript ) φ ( /lscript ) holds and Tr ( Λ ) = ∑ ∞ /lscript =1 λ ( /lscript ) < ∞ . We define Cameron-Martin space by H λ := Λ 1 / 2 ( H ) with the inner product 〈 h 1 , h 2 〉 H λ := 〈 Λ -1 / 2 ( h 1 ) , Λ -1 / 2 ( h 2 ) 〉 H . Let H ∗ ⊂ H ∗ λ be the dual space of H and H λ , respectively. By identifying H λ with H ∗ λ via the Riesz isomorphism, we can consider a continuous inclusion map i H λ : H ∗ → H λ such that

Thus, we can also identify H ∗ with Λ 1 / 2 ( H λ ) (see [8, Example 1.1]). For notational convenience, we write i H λ ( ϕ ) as Λ ( ϕ ) by regarding ϕ as an element in H . In summary,

We refer to ( H ∗ , H λ , H ) as Gelfand triple which provides groundwork for the SDE theory in H .

Example 2.2 (Reproducing kernel Hilbert space) . Let H denote the class of square-integrable functions on finite measure space ( X , µ X ) with the usual L 2 -norm, ‖ f ‖ H := (∫ X | f ( x ) | 2 µ X ( d x ) ) 1 / 2 . Due to the assumption on Λ , there exists symmetric and positive definite kernel function K λ such that

Example 2.1 (Euclidean space) . Let H := R d and λ = N (0 , Σ ) be a Gaussian measure with a positive-definite covariance matrix Σ ∈ R d × d . Then ∣ ∣ Σ -1 / 2 x ∣ ∣ < ∞ for x ∈ R d , hence H λ = H . /square

Hence Cameron-Martin space H λ is reproducing kernel Hilbert space (RKHS) with K λ . /square

Wiener Processes and Stochastic Evolution Equations in Hilbert Spaces For T > 0 , let ( W t ) t ∈ [0 ,T ] be H -valued F t -adapted Λ -Wiener process (see Definition B.1 for details). By the Kosambi-Karhunen-Loève theorem, W t has a series representation:

where W t = { W ( /lscript ) t : /lscript ∈ N } is white noise of which the components W ( /lscript ) t are independent realvalued Wiener processes. Then Cov ( W t ) = t Λ holds and E 〈 W t , W s 〉 H = min { t, s } · Tr ( Λ ) < ∞ since Λ has the finite trace. If H λ is RKHS with K λ (Example 2.2), W t preserves spatial correlation,

Let L 2 ( H ) denote the class of Hilbert-Schmidt operator on H . Following the usual construction of the stochastic integral with respect to W t (e.g., see [14, Section 4] and [35, Section 2]), we can define the following stochastic evolution equation in H ,

where B t : [0 , T ] × H → H and G t : [0 , T ] → L 2 ( H ) are progressively measurable satisfying regular conditions [35, Section 3.2] which guarantee solvability in H . Under these assumptions, there exists a unique H -valued solution X t to (7) for every t ∈ [0 , T ] such that X 0 ∈ H and W t ∈ H are independent. We assume G t is constant on H , which still recovers the SDE framework of [52].

2.2 Score Operator and Time-Reversal Formula

Logarithmic Derivative of Probability Measure and Score Operator A score function in the Euclidean space is defined by a gradient of logarithmic probability density ∇ x log p t ( x ) . For Hilbert spaces, we employ logarithmic derivatives of Fomin differentiable probability measures [8, 9]. Let us define the class of smooth cylinder functions [10], which is dense in the Sobolev space on H , by

A condition for the existence of the time-reversal process (8) is well-studied for diffusion processes in abstract settings (e.g., see [12]), hence we focus on identifying evolutionary operators ̂ B and ̂ G .

For f ϕ 1: m ∈ FC ∞ b , we compute the H λ -gradient at u ∈ H in the direction of ξ ∈ H λ by where Df ϕ 1: m denotes a Gâteaux derivative. A differentiable measure µ on H along a vector ξ ∈ H λ has the partial logarithmic derivative ρ µ ξ ∈ L 1 ( H , µ ) if the following integration by parts holds,

If there exists a Borel map ρ µ H λ ( · ) : H → H such that 〈 ρ µ H λ ( u ) , ξ 〉 H λ = ρ µ ξ ( u ) for ξ ∈ H λ , then ρ µ H λ is called the vector logarithmic derivative of µ associated with H λ . For instance, if µ is a Gaussian measure with mean m µ ∈ H and covariance operator Λ µ ∈ L ( H ) satisfying Λ ( H ) ⊂ Λ µ ( H ) , then ρ µ H λ is computed by (see [8, Example 2.5] and [14, Proposition 2.26]),

which generalizes the score function ∇ x log p ( x ) of a Gaussian density p ( x ) ∼ N ( m µ , Σ µ ) . For a sequence of probability measures ( µ t ) t ∈ [0 ,T ] of the solution to (7) with µ 0 = P data, we define a score operator S λ ( t, · ) : [0 , T ] × H → H by multiplying G t G ∗ t to the vector logarithmic derivative ρ µ t H λ ,

Hence the score operator S λ ( t, · ) is analogous to g 2 ( t ) ∇ x log p t ( x ) in the reverse SDE of [52].

Kolmogorov Equations and Time-Reversal Formula Score-based generative model through SDE is grounded on the Fokker-Planck-Kolmogorov equations [1, 24]. Following [5, 6, 8-10], we define Kolmogorov operator L t on the class of smooth cylinder functions FC ∞ b by

where A t := G t G ∗ t and D 2 f ϕ 1: m denotes the second-order Gâteaux derivative. The Kolmogorov operators ( L t ) t ∈ [0 ,T ] describes evolution of ( µ t ) t ∈ [0 ,T ] and becomes a generator of Markov diffusion process [5, 6, 24]. Now we present our main result on the time-reversal formula.

where ̂ A t and ̂ B t satisfy the following time-reversal formula

See Appendix B.1 for the proof of Theorem 2.1. The proof is based on the Kolmogorov forward equation and the Kolmogorov backward equation with functional derivatives to show that (15) is the generator of the time-reversal diffusion process (8), which is analogous to the proof in the Euclidean space [24]. Due to (16), we can rewrite (8) by

2.3 Applications of Theorem 2.1

Hence we can generate a sample from ̂ X T ∼ P data by running (17) if we obtain S λ ( t, · ) for t ∈ [0 , T ] .

This section presents two important applications of Theorem 2.1; SDEs in infinite-dimensional space and parabolic SPDEs in finite-dimensional space. We utilize these stochastic evolution equations to propose a unified framework of continuous-time score-based generative models in Section 3.

2.3.1 SDEs in Infinite-Dimensions

We first generalize the SDE framework [52] in the infinite-dimensions based on Theorem 2.1. Let { λ ( /lscript ) , φ ( /lscript ) } /lscript ∈ N be an eigensystem of H and consider the following SDE system with values in H , where b ( /lscript ) t : [0 , T ] → R and σ ( /lscript ) t : [0 , T ] → R + are real-valued functions for each /lscript ∈ N . Set β ( /lscript ) t := exp ( ∫ t 0 b ( /lscript ) s d s ) and Σ ( /lscript ) β ( t ) := ( σ ( /lscript ) t / β ( /lscript ) t ) 2 . Conditioning at X ( /lscript ) 0 , the law of X ( /lscript ) t follows Gaussian distribution, hence we can apply [44, Theorem 5.3] to (18), which induces B t ( x ) by

Indeed, (19) is a special case of (16) (see Appendix C.1).

Example 2.3 (VP-SDE and sub-VP-SDE in Infinite-Dimensions) . Set b ( /lscript ) t = -1 2 b ( t ) and σ ( /lscript ) t = √ b ( t ) for every /lscript ∈ N . Then we have β t = e -1 2 ∫ t 0 b ( s ) d s and Σ β ( t ) = b ( t ) e ∫ t 0 b ( s ) d s . Thus,

Hence we can compute the conditional score function S λ ( t, X t | X 0 ) by using (19). As a result, (20) generalizes VP-SDE [52] in infinite-dimensions. Similarly, we can recover the sub-VP-SDE in infinite-dimensions, hence (19) recovers SDEs proposed in [52] when Λ = Id and H = R d .

2.3.2 Parabolic SPDEs

We introduce parabolic SPDEs [34, 35] based on Theorem 2.1. For k > -1 , we define on a bounded domain O with a zero-Neumann boundary condition. By following [50], we use the discrete cosine transformation (DCT) from a coordinate vector u t expressed in the standard basis to a coordinate vector ˆ u t , with the eigendecomposition ∆ /defines VDV /latticetop . Hence we can rewrite (21) as

where ˆ u t = V /latticetop u t and ̂ W t = V /latticetop W t . Since V is orthogonal, we have ̂ W t d = W t for each t , so we can regard ̂ W t as standard space-time white noise. Thus, we can apply Theorem 2.1 to (22) with B t = b ( t )( D -σ I ) and G t = √ 2 b ( t )( σ I -D ) -k 2 in R d (see Appendix C.2). Therefore,

If we set b ( t ) = 1 , σ = 0 , and k = 0 , then (21) becomes stochastic heat equation (SHE) and has a connection between the generative process of IHDM [50]. Indeed, the mild solution of SHE is

where e ∆ t is a C 0 -semigroup, which induces the mean vector of u t as e ∆ t u 0 = V exp( D t ) V /latticetop u 0 , and the remaining stochastic integration part becomes Gaussian process [14, Theorem 5.2]. If we consider a continuous-time version of IHDM with a variance scheduling, then (23) induces the appropriate reverse SPDE for sampling. See Section 4.3 for empirical results.

3 Hilbert Diffusion Models

This section is devoted to describing Hilbert Diffusion Models (HDMs), a class of score-based generative models based on Theorem 2.1 and stochastic evolution equations presented in Section 2.3. We introduce two versions of HDM, HDM-SDE and HDM-SPDE, which utilize SDEs in infinite-dimensions (18) and parabolic SPDEs (21), respectively.

3.1 Estimating Score Operators

To estimate the score operator S λ ( t, · ) in Hilbert space, we propose a time-dependent score model s θ ( t, · ) : [0 , T ] × H → H parameterized with θ ∈ R p , which is trained by the following objective,

where U denotes the uniform distribution on [0 , T ] . Due to [38, 54], note that minimizing (25) is equivalent to minimizing the following objective,

where µ t | X 0 denotes the conditional distribution of X t conditioned at X 0 ∼ P data, and S λ ( t, X t | X 0 ) is the corresponding score operator, which is estimated by a time-dependent neural network.

Positional

Embedding (   )

Lifting

Layer

Fourier

Layer 1

s

θ

(

t

,

x

)

[B, C, D]

[B, E, D]

[B, E, D]

[B, E, D]

[B, E]

+

Element-wise addition operation

B

: Batch

E

: Lifting Channels

C

: Channels

D

: Dimension of Data

N

: Number of Fourier Layers

[B, C, D]

Fourier

Layer

N

[B, 1]

[B, E, D]

Net

(

P

(

t

))

Linear Layer

(      )

Net

P

[B, E, D]

MLP

Skip-connection block

Skip-connection block

+

+

Activation

(GeLU)

Activation

(GeLU)

Activation

(GeLU)

t

x

…

…

[B, E, D]

Activation

(GeLU)

Fourier

Layer

(n)

Projection

Layer

+

FFT

Factorized

SpectralConv

iFFT

Architectures of HDM We use different architectures for modeling HDM-SDE and HDM-SPDE since the corresponding stochastic evolution equations (18) and (21) have distinct values in infinitedimensions and Euclidean space, respectively. For HDM-SDE, same as other diffusion models in infinite-dimensions [23, 29, 38], we modify Fourier Neural Operator (FNO) [37], called timeconditioned FNO, by adding a positional embedding and a linear layer to embed the time variable t (see Figure 2). The spatial variable x first passes through the lifting layer, increasing its dimensions to match the lifting channels. Then, the embedded time variable undergoes the linear layer to adjust its dimensions to match the lifted spatial variable, and is pointwisely added. The Fourier layer takes the input and applies Fast Fourier Transformation (FFT) to transform it to the frequency space, followed by spectral convolution. See Appendix C.3 for more details.

For HDM-SPDE, we use U-Net [51] similar to other diffusion models [25, 52], mostly varying the number of multipliers of resolutions and the number of residual blocks. Detailed architecture parameters according to datasets are described in Table A.3.

<latexi sh

1_b

64="VoOWk9

+

7ASCdq

Z8fNM

F

>

Un

c

L

TgI

3

E

m

5

B

w2

z

J

v

KDr

Q

/

0pH

X

y

u

R

Y

<latexi sh

1_b

64="PDk

F7R

C

K+Sn o2B

5H

8

>A

X

c

V

LTgI

O

Q

NJ

z

W

G

j

0

p

v

d

9

/

3

m

u

f

w

Z

U

E

q

Y

<latexi sh

1_b

64="

P

0S7

+rT

XQw

R

FBzg

>A

C

c

VDL

I

O2

8

NJ

W

M

y

G5

j

p

k

v

o

d

9n

/

K

3

m

u

f

Z

U

E

H

q

Y

<latexi sh

1_b

64="0

R7w

Qu

Kg3

D

ENH/

q

U

>A

C

n

c

VL

T

M

5

BY

ok

S8

pW

IJF

f

rv

XP

j

9

zO

G

d

+

y

2

m

Z

<latexi sh

1_b

64="NJQqLMDF

2UfG5

yAWv3nw

H

0

>

C

c

V

TgI

j

8E

mS

+

u

B

d

9o

K

/

z

Z

O

r

X

kY

P

7

p

R

3.2 Training and Sampling

Loss Functions Based on (26), both HDM-SDE and HDM-SPDE use the mean-square loss as the training loss function. Fix x 0 ∼ P data and let x t be a generated sample at time t from µ t | x 0 so that x t ∼ N ( m µ t | x 0 , Λ µ t | x 0 ) . Then the loss functions are represented by

Here we use the cosine beta schedule [46] with β ( t ) = 2 1+ s π 2 tan ( t + s 1+ s π 2 ) for both HDM-SDE and HDM-SPDE for a small number s > 0 .

Solving Reverse Stochastic Evolution Equations After training a score model s θ ( t, · ) for all t ∈ [0 , T ] , we can generate samples from P data by running the reverse process (17) with the trained score model instead of S λ ( t, · ) . Numerically, we utilize the Euler-Maruyama method to approximate a solution of (17) starting at X t 0 ∼ GP (0 , Λ ) (see Algorithms 1 and 2), where N t m i.i.d. ∼ GP (0 , Λ ) , and ∆ t := t m +1 -t m for m ∈ { 0 , . . . , M } with t 0 = 0 and t M = T .

4 Experiments

This section presents the experimental settings and results that apply HDM-SDE to generating function and motion and HDM-SPDE to image generation. For more details, see Appendix D.

4.1 Function Generation through HDM-SDE

Datasets and Implementation We validate HDM-SDE for generating functions from 1D functional datasets; Quadratic, Melbourne, and Gridwatch, to follow the setting in [48]. We refer to Section D.1 for more details about datasets and settings. We tune the kernel hyperparameter of len parameter for 0.8 in Quadratic, 2.0 in Melbourne, and 1.8 in Gridwatch. We commonly use gain parameter 1.0 for all datasets. Grid parameter adjusted to 100 in Quadratic, 24 in Melbourne, and 288 in Gridwatch. In the Quadratic and Melbourne datasets, we train our model during 1K iterations (1.5K in Gridwatch). For sampling, we set the number of sampling steps as 1,000. For a quantitative evaluation, we calculate the power of a kernel two-sample test [56] based on Maximum Mean Discrepancy (MMD).

Power:

4.2 ± 0.3

Power:

99.0

Power:

99.0

Dataset

(a) DDPM

(c) HDM-SDE

(b) NDP

100

100

100

100

75

75

75

75

50

50

50

50

25

25

25

25

0

0

0

0

-25

25

25

25

50

-50

50

50

-75

75

75

75

100

-100

100

100

10.0

-7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

10.0

7.5

5.0

-2.5

0.0

2.5

5.0

7.5

10.0

10.0

7.5

-5.0

-2.5

0.0

2.5

5.0

7.5

10.0

10.0

7.5

-5.0

-2.5

0.0

2.5

5.0

7.5

10.0

Results Table 1 shows the results of the test power analysis, and HDM-SDE demonstrates excellence compared to Gaussian process [55], Neural process [20], DDPM (VP-SDE) [25], and other diffusion models (NDP [18] and SP-SGM [48]) formulated on function spaces. Our method records a power(%) of 4.2 ± 0 . 3 on Quadratic, 0.2 ± 0 . 1 on Melbourne, and 3.6 ± 0 . 0 on Gridwatch. Figures 3 and A.1 show that HDM-SDE method has outperformed results compared to the diffusion models utilizing the

white-noise, due to their limited adaptability to variable discretizations as discussed in [38]. A trace-class diffusion model using Langevin dynamics [38] records power of 6.9 ± 0 . 5 on Quadratic and of 4.0 ± 0 . 3 on Melbourne, which is comparable to SP-SGM but are less favorable than HDM-SDE.

4.2 Motion Generation through HDM-SDE

DDPM

HDM-SDE

(Ours)

Frame

DDPM HDM-SDE (Ours) Frame Datasets and Implementation We further investigate the performance of HDM-SDE for motion generation tasks using a HumanML3D dataset [22]. The HumanML3D dataset consists of 14,616 motions, where each motion consists of 144-dimensional trajectories, derived from 24 joints and a 6D representation and we randomly sample 100 motions for the experiment. The root poses of the skeletons are fixed to the origin, and the motion length is trimmed to 128. We compare our method with the baseline (DDPM) [25] utilizing the FNO architecture except the input and output channel sizes of FNO are changed from 1 to 144 to handle 144-dimensional trajectories. Hence, our proposed method and the baseline have identical network architectures and loss functions, and the only difference is the noise sampling procedures in forward and reverse processes. We utilize the squared-exponential kernel for generating the Wiener process in RKHS where the length parameters of each 144-dimensional trajectory are selected using a likelihood-based model selection of a Gaussian process [55]. See Appendix D.3 for details.

Results The synthesized motions of the proposed method are shown in Figure 4, where we overlay right and left hands trajectories with red and blue, respectively. While the proposed method successfully generates human motions, the baseline fails to generate meaningful motions. Our proposed method significantly outperforms the baseline in terms of motion synthesis as the baseline fails to generate human-like motions. This clearly highlights the advantage of utilizing H -valued Wiener process in modeling multi-dimensional complex trajectories. The quantitative results demonstrate better performance in both hand and foot trajectories within the task space are shown in Table A.2.

4.3 Image Generation through HDM-SPDE

Datasets 2D image experiment includes MNIST [36], CIFAR10 [42], LSUN-church [58], AFHQ [13], and FFHQ [28] datasets. We adjust the image resolutions as IHDM [50], LSUN-church images are resized into 128 × 128; higher resolutions, including AFHQ and FFHQ, have resized into 256 × 256. See Appendix D.4 for implementation details.

Evaluation To evaluate the sampling quality, we use the clean FID [47] using InceptionV3 [53]. Clean-FID scales generated and reference images into 299 × 299 resolution via bicubic interpolation and compute statistics. Following IHDM [50], we sample 50,000 images with a resolution smaller than 256 × 256 and calculate the reference samples as the training set. Otherwise, we sample 10,000 images and use the entire dataset as reference data for evaluating metrics.

FID: 17.84

FID: 17.94

FID: 15.21

FID: 29.09

MNIST

CIFAR10

LSUN

AFHQ

FFHQ

32

9

5

S

3

3

2

9

7

1

9

53

S

9

83

3

3

3

D

Q

3

1

O

3

72

8

（

L

b

2

232459

9

7

3269482

LSUN

AFHQ

FFHQ

HDM Prior

IHDM Prior

Origin

NFE: 30

NFE: 70

Degraded

Super-resolution

Depixelate

NFE: 100

NFE: 1000

Deblurring

Results Figure 5 summarizes uncurated image generation results of HDM-SPDE on various image datasets. Our model records FID scores of 17.84 on CIFAR10 and 29.09 on LSUN-church, which is better than that of IHDM [50] (CIFAR: 18.96, LSUN: 45.06). Especially, our model records PRDC [45] of (P: 0.696, R: 0.633, D: 0.733, C: 0.727) on CIFAR10, which is higher than that of IHDM (P: 0.596, R: 0.767, D: 0.580, C: 0.723) except for recall. For AFHQ and FFHQ, HDM-SPDE achieves at least twice the better FID score of 17.94 on AFHQ and 15.21 on FFHQ than those of IHDM (AFHQ: 41.39, FFHQ: 64.91). Although there remain gaps compared to the state-of-the-art results using diffusion models [25, 52], the improvement in sample quality as a continuous-time version of the IHDM is significant. Thus, we conclude that the proposed framework opens the potential for exploring various evolution operators having an inductive bias in image generation.

Connection between HDM and IHDM As we addressed in Section 2.3.2, SPDE and IHDM have a connection such that the mean trajectory of the SPDE is equivalent to the trajectory of the forward process of IHDM. Hence we hypothesize that the HDM can generate images by running the reverse SPDE from the prior distribution of IHDM. Figure 6 illustrates the generated images by trained HDM-SPDE provided with HDM, IHDM, and Gaussian priors. Notably, the reverse SPDE generates similar images from IHDM prior to HDM one, contrary to Gaussian prior. On AFHQ and FFHQ, HDM prior results in FID of 17.94 and 19.40, better than 19.40 and 20.89 of IHDM prior, still outperform the results reported by [50], 43.39 and 64.91, respectively.

Controllable Coarse-to-Fine Generation Motivated from [4], we examine whether HDM-SPDE can leverage the learned inductive bias in controllable coarse-to-fine image generation tasks. Figures 7 and A.4 show that HDM-SPDE can successfully perform inpainting, super-resolution, depixelation, and deblurring tasks without any additional training. HDM-SPDE is capable of generating images that resemble the background color to some extent for pixelated images and also generates samples leveraging the contours of blurred images. This is possible since low frequencies are filled with the frequencies of low-resolution images, whereas the range of high frequencies is filled with generated frequencies through the learned score model. Diffusion models trained by [52] fail to generate samples that capture the color or contours of pixelated or blurred images (see Appendix D.5).

5 Related Works

Diffusion models in infinite-dimensional space have been actively investigated [18, 23, 29, 38, 48, 49]. [29] introduces a discrete-time denoising diffusion model in infinite-dimensional space, which provided one of the key inspirations for studying a continuous-time approach connecting with scorebased methods through the SDE framework [52]. [18] is another discrete-time approach proposed to generalize neural processes [20], but it utilizes non-trace class white noise that causes an ill-posedness in the infinite-dimensional setting. See [14] for a reference of Hilbert space valued Wiener processes.

A challenging part of the continuous-time approach in Hilbert space is defining the score function without using the density function that no longer exists in the infinite-dimensional setting. [23] proposes using a discretized process to circumvent this problem and [48] suggests a continuous-time model based on spectral decomposition stemming from the Karhunen-Loève theorem; however, they do not include the general SDE framework [52] nor time-dependent operators. Based on [44], [19] only considers constant-time diffusion coefficients, which do not fully recover the desired framework requiring variance scheduling [46]. Section 2.3.1 shows the intersection between [19, 23] and our approach and how the proposed framework covers SDEs in infinite dimensions.

Concurrently, [38, 49] propose continuous-time models; however, their theoretical results are also confined to SDEs with constant-time operators since their approach is primarily grounded in semigroup theory [14], which has an intrinsic limitation when dealing with variable coefficients [39]. We note that the multiple noise scale case is revealed in [38, Section 4.4], yet it is constrained to discrete-time models. On the contrary, our approach is grounded in the study of variational approach to stochastic evolution equations [34, 35] and forward and backward Kolmogorov equations with functional derivatives [6, 15], which serves as a more established approach to identifying relations (Theorem 2.1) between time-dependent evolution operators and invariant measures of stochastic equations in Hilbert spaces [5]. It is noteworthy that proof of Theorem 2.1 (Appendix B.2) can be viewed as a generalization of the classical techniques employed by [1] and [24] in finite-dimensional spaces to general Hilbert spaces. Consequently it is able to fully recover the SDE framework proposed from [52], thereby enabling the variance scheduling [46], and building a bridge between the discrete-time model presented in IHDM [50] and the continuous-time score-based approach.

Similar to [11, 23, 29, 38], which combine diffusion models with neural operators [32, 37], we propose a modified time-conditioned FNO for the HDM-SDE in the functional and motion generation task, which has slightly different modules from the original neural operator. For the image generation task with the HDM-SPDE, however, we use the U-Net [51] to compare with the setting in variations of diffusion models [4, 27, 50], which utilize image transformations instead of drift coefficients.

6 Limitation and Conclusion

We extend the SDE framework proposed by [52] and propose a class of continuous-time score-based generative models based on stochastic evolution equations in Hilbert spaces. Our derivation of the time-reversal formula considering time-dependent coefficients enlarges the practical application of score-based generative models in function spaces and advances the performance of recent variations of diffusion models which use blurring [50] or pixelation [4]. Although performance gaps remain in image generation compared to the state-of-the-art results, we conclude that the proposed framework opens the potential for exploring various evolution operators and sample spaces in this area.

Acknowledgments and Disclosure of Funding

This work was supported by LG AI Research. This work was also supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT)(No. 2022-0-00612, Geometric and Physical Commonsense Reasoning based Behavior Intelligence for Embodied AI; No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University), and National Research Foundation of Korea(NRF) funded by the Korea government(MSIT)(2021R1A4A3033149). This work was supported by Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) & Gwangju Metropolitan City.

References

Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982. ISSN 0304-4149. doi: https://doi.org/10.1016/ 0304-4149(82)90051-5. URL https://www.sciencedirect.com/science/article/ pii/0304414982900515 .

VV Baklan. On the existence of solutions of stochastic equations in hilbert space. Depov. Akad. Nauk. Ukr. USr , 10:1299-1303, 1963.

VV Baklan. Equations in variational derivatives and markov processes in hilbert spaces. In Dokl. Akad. Nauk. SSSR , volume 159, pages 707-710, 1964.

Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392 , 2022.

Yana Belopolskaya. Invariant measures of diffusion processes in hilbert spaces and hilbert manifolds. In Probability Theory and Mathematical Statistics: Proceedings of the Seventh Vilnius Conference , page 67. Walter de Gruyter GmbH & Co KG, 2020.

Yana Belopolskaya and Yu L Dalecky. Stochastic equations and differential geometry , volume 30. Springer Science & Business Media, 2012.

Yana Isaevna Belopol'skaya and Yurii L'vovich Daletskii. Diffusion processes in smooth banach spaces and manifolds. i. Trudy Moskovskogo Matematicheskogo Obshchestva , 37:107-141, 1978.

Vladimir I Bogachev and Michael Rockner. Regularity of invariant measures on finite and infinite dimensional spaces and applications. Journal of Functional Analysis , 133(1):168-223, 1995.

Vladimir I Bogachev, Nicolai V Krylov, and Michael Röckner. Regularity of invariant measures: the case of non-constant diffusion part. Journal of Functional Analysis , 138(1):223-242, 1996.

Vladimir I Bogachev, Nicolai V Krylov, Michael Röckner, and Stanislav V Shaposhnikov. Fokker-Planck-Kolmogorov Equations , volume 207. American Mathematical Society, 2022.

S Bond-Taylor and CG Willcocks. infinite-diff: Infinite resolution diffusion with subsampled mollified states. arXiv preprint arXiv:2303.18242 , page 1, 2023.

Patrick Cattiaux, Giovanni Conforti, Ivan Gentil, and Christian Léonard. Time reversal of diffusion processes under a finite entropy condition. In Annales de l'Institut Henri Poincaré (B) Probabilités et Statistiques , 2022.

Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Giuseppe Da Prato and Jerzy Zabczyk. Stochastic equations in infinite dimensions . Cambridge university press, 2014.

Yu L Dalecky and Sergei Vasil'evich Fomin. Measures and differential equations in infinitedimensional space . Springer, 1991.

Yu L Daletskii. Infinite-dimensional elliptic operators and parabolic equations connected with them. Russian Mathematical Surveys , 22(4):1, 1967.

Yurii L'vovich Daletskii. Multiplicative operators of diffusion processes and differential equations in sections of vector bundles. Uspekhi Matematicheskikh Nauk , 30(2):209-210, 1975.

Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes. arXiv preprint arXiv:2206.03992 , 2022.

Giulio Franzese, Simone Rossi, Dario Rossi, Markus Heinonen, Maurizio Filippone, and Pietro Michiardi. Continuous-time functional diffusion processes. arXiv preprint arXiv:2303.00800 , 2023.

Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622 , 2018.

Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proc. of the ACM International Conference on Multimedia , pages 2021-2029, 2020.

Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5152-5161, 2022.

Paul Hagemann, Lars Ruthotto, Gabriele Steidl, and Nicole Tianjiao Yang. Multilevel diffusion: Infinite dimensional score-based diffusion models for image generation. arXiv preprint arXiv:2303.04772 , 2023.

Ulrich G Haussmann and Etienne Pardoux. Time reversal of diffusions. The Annals of Probability , pages 1188-1205, 1986.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proc. of the Conference on Neural Information Processing Systems (NeurIPS) , 33:6840-6851, 2020.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv:2204.03458 , 2022.

Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= OjDkC57x5sz .

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4396-4405, 2019. doi: 10.1109/CVPR.2019.00453.

Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models in infinite dimensions. arXiv preprint arXiv:2212.00886 , 2022.

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2426-2435, 2022.

Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. In Proc. of the Association for the Advancement of Artificial Intelligence (AAAI) , 2023.

Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research (JMLR) , 24(89):1-97, 2023. URL http://jmlr.org/papers/v24/21-1524.html .

Nicolai V. Krylov. A relatively short proof of It¯ o's formula for SPDEs and its applications. arXiv preprint arXiv:1208.3709 , 2012.

Nicolai V Krylov and Boris L Rozovskii. Stochastic evolution equations. Journal of Soviet Mathematics , 16:1233-1277, 1981.

Nicolai V Krylov and Boris L Rozovskii. Stochastic evolution equations. In Stochastic Differential Equations: Theory And Applications: A Volume in Honor of Professor Boris L Rozovskii , pages 1-69. World Scientific, 2007.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. doi: 10.1109/5.726791.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 , 2020.

Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, et al. Score-based diffusion models in function space. arXiv preprint arXiv:2302.07400 , 2023.

Wei Liu and Michael Röckner. Spde in hilbert space with locally monotone coefficients. Journal of Functional Analysis , 259(11):2902-2922, 2010.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG) , 34(6):1-16, 2015.

Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision , pages 5442-5451, October 2019.

Charles A Micchelli and Massimiliano Pontil. On learning vector-valued functions. Neural computation , 17(1):177-204, 2005.

Annie Millet, David Nualart, and Marta Sanz. Integration by parts and time reversal for diffusion processes. The Annals of Probability , pages 208-238, 1989.

Annie Millet, David Nualart, and Marta Sanz. Time reversal for infinite-dimensional diffusions. Probability theory and related fields , 82(3):315-347, 1989.

Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In Proc. of the International Conference on Machine Learning (ICML) , 2020.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proc. of the International Conference on Machine Learning (ICML) , pages 81628171. PMLR, 2021.

Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Angus Phillips, Thomas Seror, Michael John Hutchinson, Valentin De Bortoli, Arnaud Doucet, and Emile Mathieu. Spectral diffusion processes. In NeurIPS Workshop on Score-Based Methods , 2022. URL https://openreview.net/forum?id=bOmLb2i0W_h .

Jakiw Pidstrigach, Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinite-dimensional diffusion models for function spaces. arXiv preprint arXiv:2302.10130 , 2023.

Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. In Proc. of the International Conference on Learning Representations (ICLR) , 2023. URL https://openreview.net/forum?id=4PJUBT9f2Ol .

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. of the Medical Image Computing and ComputerAssisted Intervention (MICCAI) , pages 234-241. Springer, 2015.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. of the International Conference on Learning Representations (ICLR) , 2020.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2818-2826, 2016. doi: 10.1109/CVPR.2016.308.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA, 2006.

George Wynne and Andrew B Duncan. A kernel two-sample test for functional data. Journal of Machine Learning Research (JMLR) , 23(73):1-51, 2022.

Eunbi Yoon, Keehun Park, Jinhyeok Kim, and Sungbin Lim. Score-based generative models with lévy processes. In NeurIPS Workshop on Score-Based Methods , 2022. URL https: //openreview.net/forum?id=ErzyBArv6Ue .

Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. CoRR , abs/1506.03365, 2015. URL http://arxiv.org/abs/1506.03365 .

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5745-5753, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce Hilbert Diffusion Model (HDM). This generative model is a generalization of diffusion models to infinite dimensional state space (in this case Hilbert spaces) through the use of generalized SDEs. The authors extend the notion of score and time-reversal to this setting. Their framework can be applied to infinite dimensional diffusion models and to Stochastic Partial Differential Equations (SPDEs). They propose to use Fourier Neural Operator (FNO) as a basis of the architecture of the neural network to learn the score, similar to other infinite dimensional diffusion models. They showcase the efficiency of their model on function and image generation tasks.\n\nSTRENGTHS:\n* The paper presents a strong theoretical framework for the theory of Hilbert diffusion models. The ideas are very well-presented (even though I think the authors could give more details about their SPDE application, see my Questions).\n\n* The experiments are convincing. In particular, the model seems to perform better than other models which are not based on a noising perturbation but on other degradations such as [1].\n\n[1] Rissanen et al. (2023) -- Generative modelling with inverse heat dissipation\n\nWEAKNESSES:\n* One of my main remark is that there is little discussion on the relation between the current work and the existing literature on diffusion models. For instance, what is the novelty in the derived theorem 2.1? It seems to me that [1, Section 3.4] already studied the time-reversal of infinite diffusion models. Similarly, the framework of Cameron-Martin theory to derive infinite dimensional Brownian motion and Gaussian measures was already considered in [1,2]. What is the difference between the current work and the works [1,2,4,6] from a theoretical point of view? From a practical point of view, the authors should also specify how different their method is from existing works. As they point out the Fourier Neural Operator (FNO) was already used in several works [2,4,5]. I think the authors omit to discuss [3] which not only use the FNO but found that Galerkin operators worked well. At the end of the day what are the practical differences with [2,4,5]? Finally, could the authors discuss the relationship between their model and other infinite dimensional approaches such as [7,8]?\n\n* My second remark is more general and has to do with the motivation of the work. It is not clear to me why, from a practical perspective, would anyone want to use infinite dimensional diffusion models? When it comes to implementation the space is going to be discretised and advantages that are usually cited to be given by the infinite dimensional setting (infinite resolution) can also be incoporated in existing finite dimensional diffusion models (by interpolating on a sub-pixel grid for instance). For example the fact that we consider continuous-time diffusion models could also be criticized since everything is then discretized at implementation time but it turns out that this formulation yields some advantages (gives ideas for better samplers, allows for simple integration of the Ornstein-Uhlenbeck to sample without Markov chains, allows for log-likelihood computation using ideas from continuous normalizing flows). Could the authors present such motivation in the case of infinite dimensional diffusion model? I think coming up with an example where infinite dimensional diffusion models are needed would be key to motivate the study.\n\n* My third remark concerns the motivation behind other choices of perturbation than a Gaussian noising. Could the authors explain why not using a Gaussian noising might be beneficial? While it is possible to do so I have seen little practical justification of this so far.\n\n[1] Pidstrigach et al. (2023) -- Infinite-dimensional diffusion models for function space\n\n[2] Kerrigan et al. (2022) -- Diffusion generative models in infinite dimensions\n\n[3] Bond-Taylor and Willcocks (2023) -- Infinite Diff: Infinite Resolution Diffusion with Subsampled Mollified States\n\n[4] Lim et al. (2023) -- Score-based diffusion models in function space\n\n[5] Hagemann et al. (2023) -- Multilevel diffusion: Infinite dimensional score-based diffusion models for image generation\n\n[6] Franzese et al. (2023) -- Continuous-time functional diffusion processes\n\n[7] Dutordoir et al. (2022) -- Neural Diffusion Processes\n\n[8] Philips et al. (2022) -- Spectral Diffusion Processes\n\nQUESTIONS:\n* In Section 2.1 can the authors define what is a Gaussian measure in a Hilbert space\n\n* We define Cameron-Martin space --> We define \"the\" Cameron-Martin space (?)\n\n* In Section 2.1 it seems that there is no need for the operator $\\Lambda$ to be positive. However, the authors take its inverse. Am I missing something here? In Example 2.1 what happens if $\\Sigma$ is not invertible?\n\n* The authors state \"the time-reversal process is well-studied for diffusion processes in abstract settings\" and cite [1]. Then, could the authors simply use the results from [1] to establish Theorem 2.1? [1] uses only the carre du champs operator and so, since the authors also describe things in terms of infinitesimal generator, it seems to me that their main result could be applied. Again, I might be missing something.\n\n* For parabolic SPDE what is the motivation behind Equation (21)? It is not clear to me what is the role of the power of $k$ on the terms in front of the white noise Brownian motion.  Also in Equation (22) it seems that $1+k/2$ has turned into $k/2$. In Equation (23) where is the score? I think the authors should give more explanations in that section. \n\n* The losses in Equation (25) and Equation (26) could be improved by using weighting terms or using $v$-prediction [2]. Have the authors considered this?\n\n* It is not clear to me why the authors do not choose a FNO for the SPDE application, considering that the Laplacian is diagonalized in a Fourier basis.\n\n* I am bit confused by the experiments on HumanML3D. DDPM seems to be performing extremely poorly in that setting. I am quite surprised by that and especially the discrepancy with the model proposed by the authors. Is there any good reason for that?\n\n* I don't understand the \"Controllable Coarse-to-FIne Generation\" paragraph. How do authors solve inverse problems with a model trained for generative modeling? Is there a guidance function somewhere? Are the degraded images passed as input to the diffusion model where the forward has been chosen appropriately? More details are needed.\n\n[1] Cattiaux et al. (2021) -- Time reversal of difusion processes under finite entropy condition\n\n[2] Salimans et al. (2022) -- Progressive Distillation for fast sampling of diffusion models\n\nLIMITATIONS:\nAdressed in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a framework called the Hilbert Diffusion Model (HDM), which generalizes and unifies the time-reversal diffusion generative models with stochastic differential equations (SDEs) in Hilbert spaces. This model enables SDEs to be used in infinite-dimensional settings and to include stochastic partial differential equations (SPDEs) in Euclidean settings. The HDM's performance is verified against several datasets, outperforming existing diffusion models formulated in function spaces. It's particularly successful in human motion synthesis, where it applies the Wiener process in the Hilbert space.\n\nSTRENGTHS:\nThis paper effectively aims to expand the time-reversal diffusion generative model into Hilbert space. The authors also provide a comprehensive background and overview of Stochastic Differential Equations (SDEs) in Hilbert space, as illustrated in sections 2.1 and 2.2 before the time-reversal theorem.\n\nThe paper is very well-structured, featuring clear and logical derivations. The authors deserve commendation for their efforts in:\n\n- Establishing the time-reversal diffusion property in Hilbert space (as shown in Theorem 2.1 and Appendix B.2) through the use of infinitesimal generators;\n- Introducing and unifying the time-reversal diffusion in Hilbert space (as seen in Appendix C.2);\n- Employing parabolic SPDEs as functional examples and presenting the connection between parabolic SPDEs and the stochastic process used in IHDM (refer to Section 2.3.2).\n\nThe mathematical rigor demonstrated throughout the paper is thorough and cohesive.\n\nWEAKNESSES:\nMajor Points:\n\n- This work is not the first to extend time-reversal diffusion models to infinite-dimensional space, and the concept of time-reversal diffusion on Hilbert space is already addressed in [1][2][3]. As these are mentioned in your citations, it would be beneficial to delve into the relationship between HDM and these models. For instance, the authors might discuss whether they are particular cases of HDM. This further elaboration could help clarify how HDM contributes to the field beyond what has been covered in these previous works.\n\n- It would be advantageous to incorporate an evaluation comparison with [1][2][3], given their direct relevance. I would be interested in seeing, for example, how your model compares to [2] using the AEMET dataset[2] or how their model performs on the datasets you're presenting.\n\nMinor Points:\n\n- For the convenience of readers, it would be more effective if the authors could present the experimental results of image generation in tables instead of embedding them in the text.\n\n- In terms of image generation, the performance of SPDE does not match up to DDPM or VE-SDE. However, since this paper primarily concentrates on generation in Hilbert space - an issue already noted in the limitations - it may not be necessary to highlight this discrepancy in this context.\n\n\n[1] Pidstrigach, J., Marzouk, Y., Reich, S., & Wang, S. (2023). Infinite-dimensional diffusion models for function spaces. arXiv preprint arXiv:2302.10130.\n\n[2] Kerrigan, G., Ley, J., & Smyth, P. (2022). Diffusion generative models in infinite dimensions. arXiv preprint arXiv:2212.00886.\n\n[3] Lim, J. H., Kovachki, N. B., Baptista, R., Beckham, C., Azizzadenesheli, K., Kossaifi, J., ... & Anandkumar, A. (2023). Score-based diffusion models in function space. arXiv preprint arXiv:2302.07400.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors already discussed the current limitations of their model (e.g., limited performance on image generation).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies continuous-time diffusion processes in Hilbert spaces, and their applications to generative modeling. In particular, a time-reversal formula is derived for Hilbert-space SDEs, allowing the continuous-time diffusion model framework to be generalized to infinite-dimensional Hilbert spaces. The proposed methodology is evaluated on several tasks, including generation of images and time-series.\n\nSTRENGTHS:\n- The exposition is clear and the paper is generally well-written throughout.\n- This work adds to the growing literature on function-space diffusion models, and lays a solid theoretical foundation for future work in this area. The results are likely to be of significant interest to the community. \n- Section 2.3 proposes a novel continuous-time generalization of the IHDM model (Riassanen et al., 2023) which shows strong empirical performance (Section 4) compared to the discrete-time model. \n- The proposed method shows favorable performance on a functional generation task when compared to several existing functional baselines. \n- To the best of my knowledge the proofs are correct, and the empirical claims are well-supported by the provided evidence.\n\nWEAKNESSES:\n- The paper could be better contextualized within the existing literature on this topic, i.e. there is no related work section. E.g. the authors briefly mention several closely related works on generating functional data (Line 30) but do not detail the differences between this submission and those existing works. See also the question below regarding time-reversals of infinite dimensional diffusions.\n\nQUESTIONS:\n- Time reversals of infinite-dimensional diffusions have been studied previously [1-2] -- how does the main result (Theorem 2.1) relate to these existing works?\n- Could you elaborate on why you believe the DDPM model performed so poorly in Table 1, given that previous works [3-4] have successfully developed function-space generalizations of this model? E.g. it was unclear if a white-noise DDPM model was used or a trace-class DDPM model was used (as discussed  in [3-4]).\n\n[1] Time reversal for infinite-dimensional diffusions. Annie Millet, David Nualart, Marta Sanz, 1989.\n\n[2] Time reversal of infinite-dimensional diffusions. H. Follmer, A. Wakolbinger, 1986. \n\n[3] Diffusion generative models in infinite dimensions. Gavin Kerrigan, Justin Ley, Padhraic Smyth, 2023.\n\n[4] Score based diffusion models in function space. Jae Hyun Lim et al., 2023.\n\nLIMITATIONS:\nThe limitations of the work are appropriately addressed within the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a time-dependent stochastic generative model that can be implemented in Hilbert spaces. The presented method unifies diffusion processes (commonly defined in diffusion models) with a more general stochastic evolution equation. A stochastic evolution equation is defined with satisfied conditions which mimics the framework of the forward process of the SDE equation derived for diffusion models. A time-reversal formula is derived from forward stochastic evolution equation based on the Kolmogorov equation and closely resembles the SDE equation of the score-based model.\n\nSimilar to score-based models, the score function is learned by a deep learning model with Fourier layers in the network. The model shows promising qualitative and quantitative results for unique tasks (e.g., motion generation, infinite dimension),\n\nSTRENGTHS:\n- The paper is well written/organized and easy to follow. The author does a good job introducing related theoretical information.\n- The figures are mostly helpful for visualization.\n- Equations are well structured, but it may be helpful if there is a brief background or insight on some claims (e.g., line 124 the reason behind choosing the class of smooth cylinder functions, Line 109 basic description of Fomin differentiable probability measures).\n- In general, a very theoretically sound paper.\n\nWEAKNESSES:\n- It would be helpful if some of the datasets are briefly explained.\n- Many of the figure descriptions are not very clear (e.g., Figure 1 (a) and I didn't quite understand (b), add title for Figure 3 images)\n- It would be nice if some intuition or insight could be given on why the baseline DDPM model cannot perform well with some of these tasks.\n\nQUESTIONS:\n- Do you think this proposed method would naturally perform better than the DDPM baseline model for audio datasets due to the Fourier transform layers?\n- What are some real world applications for infinite dimension tasks? Could these relate to Gigapixel images and multimodal datasets?\n\nLIMITATIONS:\n- The limitations were properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}