SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathrm {d} \hat {u} _ {t} = b (t) \left(\mathbf {D} \hat {u} _ {t} - \sigma \hat {u} _ {t}\right) \mathrm {d} t + \sqrt {2 b (t)} \left(\sigma \mathbf {I} - \mathbf {D}\right) ^ {- \frac {k}{2}} \mathrm {d} \widehat {\mathcal {W}} _ {t}, \quad t \in [ 0, T ], \tag {22}
$$
where  $\hat{u}_t = \mathbf{V}^\top u_t$  and  $\widehat{\mathcal{W}}_t = \mathbf{V}^\top \mathcal{W}_t$ . Since  $\mathbf{V}$  is orthogonal, we have  $\widehat{\mathcal{W}}_t \stackrel{d}{=} \mathcal{W}_t$  for each  $t$ , so we can regard  $\widehat{\mathcal{W}}_t$  as standard space-time white noise. Thus, we can apply Theorem 2.1 to (22) with  $\mathbf{B}_t = b(t)(\mathbf{D} - \sigma \mathbf{I})$  and  $\mathbf{G}_t = \sqrt{2b(t)} (\sigma \mathbf{I} - \mathbf{D})^{-\frac{k}{2}}$  in  $\mathbb{R}^d$  (see Appendix C.2). Therefore,
$$
\widehat {\mathbf {B}} _ {t} = b (T - t) (\sigma \mathbf {I} - \mathbf {D}) + 2 b (T - t) (\sigma \mathbf {I} - \mathbf {D}) ^ {- k} \rho_ {\mathcal {H} _ {\lambda}} ^ {\hat {\mu} _ {T - t}}. \tag {23}
$$
If we set  $b(t) = 1$ ,  $\sigma = 0$ , and  $k = 0$ , then (21) becomes stochastic heat equation (SHE) and has a connection between the generative process of IHDM [50]. Indeed, the mild solution of SHE is
$$
u _ {t} = e ^ {\Delta t} u _ {0} + \int_ {0} ^ {t} e ^ {\Delta (t - s)} \mathrm {d} \mathcal {W} _ {s}, \quad t \in [ 0, T ], \tag {24}
$$
where  $e^{\Delta t}$  is a  $C_0$ -semigroup, which induces the mean vector of  $u_{t}$  as  $e^{\Delta t}u_{0} = \mathbf{V}\exp (\mathbf{D}t)\mathbf{V}^{\top}u_{0}$ , and the remaining stochastic integration part becomes Gaussian process [14, Theorem 5.2]. If we consider a continuous-time version of IHDM with a variance scheduling, then (23) induces the appropriate reverse SPDE for sampling. See Section 4.3 for empirical results.
3 Hilbert Diffusion Models
This section is devoted to describing Hilbert Diffusion Models (HDMs), a class of score-based generative models based on Theorem 2.1 and stochastic evolution equations presented in Section 2.3. We introduce two versions of HDM, HDM-SDE and HDM-SPDE, which utilize SDEs in infinite-dimensions (18) and parabolic SPDEs (21), respectively.
3.1 Estimating Score Operators
To estimate the score operator  $\mathbf{S}_{\lambda}(t,\cdot)$  in Hilbert space, we propose a time-dependent score model  $\mathbf{s}_{\theta}(t,\cdot):[0,T]\times \mathcal{H}\to \mathcal{H}$  parameterized with  $\theta \in \mathbb{R}^p$ , which is trained by the following objective,
$$
\theta^ {*} = \underset {\theta \in \mathbb {R} ^ {p}} {\arg \min } \mathbb {E} _ {t \sim \mathcal {U} [ 0, T ]} \mathbb {E} _ {\mathbf {X} _ {t} \sim \mu_ {t}} \| \mathbf {s} _ {\theta} (t, \mathbf {X} _ {t}) - \mathbf {S} _ {\lambda} (t, \mathbf {X} _ {t}) \| _ {\mathcal {H}} ^ {2}, \tag {25}
$$
where  $\mathcal{U}$  denotes the uniform distribution on  $[0, T]$ . Due to [38, 54], note that minimizing (25) is equivalent to minimizing the following objective,
$$
\theta^ {*} = \underset {\theta \in \mathbb {R} ^ {p}} {\arg \min } \mathbb {E} _ {t \sim \mathcal {U} [ 0, T ]} \mathbb {E} _ {\mathbf {X} _ {0} \sim \mathbb {P} _ {\mathrm {d a t a}}} \mathbb {E} _ {\mathbf {X} _ {t} \sim \mu_ {t} | \mathbf {X} _ {0}} \| \mathbf {s} _ {\theta} (t, \mathbf {X} _ {t}) - \mathbf {S} _ {\lambda} (t, \mathbf {X} _ {t} | \mathbf {X} _ {0}) \| _ {\mathcal {H}} ^ {2}, \tag {26}
$$
where  $\mu_t|_{\mathbf{X}_0}$  denotes the conditional distribution of  $\mathbf{X}_t$  conditioned at  $\mathbf{X}_0 \sim \mathbb{P}_{\mathrm{data}}$ , and  $\mathbf{S}_{\lambda}(t, \mathbf{X}_t | \mathbf{X}_0)$  is the corresponding score operator, which is estimated by a time-dependent neural network.
Figure 2: The architecture of time-conditioned FNO.
Architectures of HDM We use different architectures for modeling HDM-SDE and HDM-SPDE since the corresponding stochastic evolution equations (18) and (21) have distinct values in infinite-dimensions and Euclidean space, respectively. For HDM-SDE, same as other diffusion models in infinite-dimensions [23, 29, 38], we modify Fourier Neural Operator (FNO) [37], called time-conditioned FNO, by adding a positional embedding and a linear layer to embed the time variable  $t$  (see Figure 2). The spatial variable  $x$  first passes through the lifting layer, increasing its dimensions to match the lifting channels. Then, the embedded time variable undergoes the linear layer to adjust its dimensions to match the lifted spatial variable, and is pointwisely added. The Fourier layer takes the input and applies Fast Fourier Transformation (FFT) to transform it to the frequency space, followed by spectral convolution. See Appendix C.3 for more details.
For HDM-SPDE, we use U-Net [51] similar to other diffusion models [25, 52], mostly varying the number of multipliers of resolutions and the number of residual blocks. Detailed architecture parameters according to datasets are described in Table A.3.
3.2 Training and Sampling
Loss Functions Based on (26), both HDM-SDE and HDM-SPDE use the mean-square loss as the training loss function. Fix  $\mathbf{x}_0\sim \mathbb{P}_{\mathrm{data}}$  and let  $\mathbf{x}_t$  be a generated sample at time  $t$  from  $\mu_t|\mathbf{x}_0$  so that  $\mathbf{x}_t\sim \mathcal{N}(m_{\mu_t|\mathbf{x}_0},\Lambda_{\mu_t|\mathbf{x}_0})$ . Then the loss functions are represented by
$$
L _ {\mathrm {S D E}} (\theta) = \mathbb {E} _ {t \sim \mathcal {U} (0, T ]} \mathbb {E} _ {\mathbf {x} _ {0} \sim \mathbb {P} _ {\mathrm {d a t a}}} \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, \mathbf {I})} \left\| \mathbf {s} _ {\theta} (t, \mathbf {x} _ {t}) + \beta (t) \Lambda_ {\mu_ {t} | \mathbf {x} _ {0}} ^ {1 / 2} (\epsilon) \right\| _ {2} ^ {2}, \tag {27}
$$
$$
L _ {\mathrm {S P D E}} (\theta) = \mathbb {E} _ {t \sim \mathcal {U} [ 0, T ]} \mathbb {E} _ {\mathbf {x} _ {0} \sim \mathbb {P} _ {\mathrm {d a t a}}} \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, \mathbf {I})} \left\| \mathbf {s} _ {\theta} (t, \mathbf {x} _ {t}) + \mathbf {V} \left(\beta (t) \Lambda_ {\mu_ {t} | \mathbf {x} _ {0}} ^ {1 / 2} (\epsilon)\right) \right\| _ {2} ^ {2}. \tag {28}
$$
Here we use the cosine beta schedule [46] with  $\beta(t) = \frac{2}{1 + s} \frac{\pi}{2} \tan \left( \frac{t + s}{1 + s} \frac{\pi}{2} \right)$  for both HDM-SDE and HDM-SPDE for a small number  $s > 0$ .
Solving Reverse Stochastic Evolution Equations After training a score model  $\mathbf{s}_{\theta}(t,\cdot)$  for all  $t\in [0,T]$ , we can generate samples from  $\mathbb{P}_{\mathrm{data}}$  by running the reverse process (17) with the trained score model instead of  $\mathbf{S}_{\lambda}(t,\cdot)$ . Numerically, we utilize the Euler-Maruyama method to approximate a solution of (17) starting at  $\widehat{\mathbf{X}}_{t_0}\sim \mathcal{GP}(0,\Lambda)$  (see Algorithms 1 and 2),
$$
\widehat {\mathbf {X}} _ {t _ {m + 1}} = \widehat {\mathbf {X}} _ {t _ {m}} + \left(- \mathbf {B} _ {T - t _ {m}} \left(\widehat {\mathbf {X}} _ {t _ {m}}\right) + \mathbf {s} _ {\theta} \left(T - t _ {m}, \widehat {\mathbf {X}} _ {t _ {m}}\right)\right) \Delta t + \sqrt {\Delta t} \mathbf {G} _ {T - t _ {m}} \mathcal {N} _ {t _ {m}}, \tag {29}
$$
where  $\mathcal{N}_{t_m} \stackrel{\mathrm{i.i.d.}}{\sim} \mathcal{GP}(0, \Lambda)$ , and  $\Delta t \coloneqq t_{m+1} - t_m$  for  $m \in \{0, \dots, M\}$  with  $t_0 = 0$  and  $t_M = T$ .
This section presents the experimental settings and results that apply HDM-SDE to generating function and motion and HDM-SPDE to image generation. For more details, see Appendix D.
4.1 Function Generation through HDM-SDE
Datasets and Implementation We validate HDM-SDE for generating functions from 1D functional datasets; Quadratic, Melbourne, and Gridwatch, to follow the setting in [48]. We refer to Section D.1 for more details about datasets and settings. We tune the kernel hyperparameter of 1en parameter for 0.8 in Quadratic, 2.0 in Melbourne, and 1.8 in Gridwatch. We commonly use gain parameter 1.0 for all datasets. Grid parameter adjusted to 100 in Quadratic, 24 in Melbourne, and 288 in Gridwatch. In the Quadratic and Melbourne datasets, we train our model during 1K iterations (1.5K in Gridwatch). For sampling, we set the number of sampling steps as 1,000. For a quantitative evaluation, we calculate the power of a kernel two-sample test [56] based on Maximum Mean Discrepancy (MMD).
(a) DDPM
Figure 3: Comparison of functional samples generated by DDPM (VP-SDE) [25], NDP [18], and HDM-SDE (ours) on Quadratic dataset.
(b) NDP
(c) HDM-SDE
Results Table 1 shows the results of the test power analysis, and HDM-SDE demonstrates excellence compared to Gaussian process [55], Neural process [20], DDPM (VP-SDE) [25], and other diffusion models (NDP [18] and SP-SGM [48]) formulated on function spaces. Our method records a power  $(\%)$  of  $4.2_{\pm 0.3}$  on Quadratic,  $0.2_{\pm 0.1}$  on Melbourne, and  $3.6_{\pm 0.0}$  on Gridwatch. Figures 3 and A.1 show that HDM-SDE method has outperformed results compared to the diffusion models utilizing the
GP [55]	NP [20]	DDPM [25]	NDP [18]	SP-SGM [48]	HDM-SDE (Ours)
Quadratic	100.0±0.0	8.6±1.5	≥99.0	≥99.0	5.4±0.7	4.2±0.3
Melbourne	20.1±4.0	10.1±1.9	3.3±0.2	12.8±0.4	5.3±0.7	0.2±0.1
Gridwatch	29.2±5.5	51.8±15.1	16.6±1.9	16.3±1.8	4.7±0.5	3.6±0.0
Table 1: Power(%) of a kernel two-sample test on 1D datasets. Lower is better.
white-noise, due to their limited adaptability to variable discretizations as discussed in [38]. A trace-class diffusion model using Langevin dynamics [38] records power of  $6.9_{\pm 0.5}$  on Quadratic and of  $4.0_{\pm 0.3}$  on Melbourne, which is comparable to SP-SGM but are less favorable than HDM-SDE.
4.2 Motion Generation through HDM-SDE
Figure 4: Generated motion by the baseline (DDPM) [25] and HDM-SDE (ours).
Datasets and Implementation We further investigate the performance of HDM-SDE for motion generation tasks using a HumanML3D dataset [22]. The HumanML3D dataset consists of 14,616 motions, where each motion consists of 144-dimensional trajectories, derived from 24 joints and a 6D representation and we randomly sample 100 motions for the experiment. The root poses of the skeletons are fixed to the origin, and the motion length is trimmed to 128. We compare our method with the baseline (DDPM) [25] utilizing the FNO architecture except the input and output channel sizes of FNO are changed from 1 to 144 to handle 144-dimensional trajectories. Hence, our proposed method and the baseline have identical network architectures and loss functions, and the only difference is the noise sampling procedures in forward and reverse processes. We utilize the squared-exponential kernel for generating the Wiener process in RKHS where the length parameters of each 144-dimensional trajectory are selected using a likelihood-based model selection of a Gaussian process [55]. See Appendix D.3 for details.
Results The synthesized motions of the proposed method are shown in Figure 4, where we overlay right and left hands trajectories with red and blue, respectively. While the proposed method successfully generates human motions, the baseline fails to generate meaningful motions. Our proposed method significantly outperforms the baseline in terms of motion synthesis as the baseline fails to generate human-like motions. This clearly highlights the advantage of utilizing  $\mathcal{H}$ -valued Wiener process in modeling multi-dimensional complex trajectories. The quantitative results demonstrate better performance in both hand and foot trajectories within the task space are shown in Table A.2.
4.3 Image Generation through HDM-SPDE
Datasets 2D image experiment includes MNIST [36], CIFAR10 [42], LSUN-church [58], AFHQ [13], and FFHQ [28] datasets. We adjust the image resolutions as IHDM [50], LSUN-church images are resized into  $128 \times 128$ ; higher resolutions, including AFHQ and FFHQ, have resized into  $256 \times 256$ . See Appendix D.4 for implementation details.
Evaluation To evaluate the sampling quality, we use the clean FID [47] using InceptionV3 [53]. Clean-FID scales generated and reference images into  $299 \times 299$  resolution via bicubic interpolation and compute statistics. Following IHDM [50], we sample 50,000 images with a resolution smaller than  $256 \times 256$  and calculate the reference samples as the training set. Otherwise, we sample 10,000 images and use the entire dataset as reference data for evaluating metrics.
Figure 5: Uncurated generated samples for different image datasets via HDM-SPDE.
Figure 6: Sampling from HDM-SPDE prior, IHDM prior, and Gaussian prior  $\delta = 0.01$  using trained HDM-SPDE.
Figure 7: Sample trajectory of HDM-SPDE on super-resolution, depixelate, and deblurring tasks.
Results Figure 5 summarizes uncurated image generation results of HDM-SPDE on various image datasets. Our model records FID scores of 17.84 on CIFAR10 and 29.09 on LSUN-church, which is better than that of IHDM [50] (CIFAR: 18.96, LSUN: 45.06). Especially, our model records PRDC [45] of (P: 0.696, R: 0.633, D: 0.733, C: 0.727) on CIFAR10, which is higher than that of IHDM (P: 0.596, R: 0.767, D: 0.580, C: 0.723) except for recall. For AFHQ and FFHQ, HDM-SPDE achieves at least twice the better FID score of 17.94 on AFHQ and 15.21 on FFHQ than those of IHDM (AFHQ: 41.39, FFHQ: 64.91). Although there remain gaps compared to the state-of-the-art results using diffusion models [25, 52], the improvement in sample quality as a continuous-time version of the IHDM is significant. Thus, we conclude that the proposed framework opens the potential for exploring various evolution operators having an inductive bias in image generation.
Connection between HDM and IHDM As we addressed in Section 2.3.2, SPDE and IHDM have a connection such that the mean trajectory of the SPDE is equivalent to the trajectory of the forward process of IHDM. Hence we hypothesize that the HDM can generate images by running the reverse SPDE from the prior distribution of IHDM. Figure 6 illustrates the generated images by trained HDM-SPDE provided with HDM, IHDM, and Gaussian priors. Notably, the reverse SPDE generates similar images from IHDM prior to HDM one, contrary to Gaussian prior. On AFHQ and FFHQ, HDM prior results in FID of 17.94 and 19.40, better than 19.40 and 20.89 of IHDM prior, still outperform the results reported by [50], 43.39 and 64.91, respectively.
Controllable Coarse-to-Fine Generation Motivated from [4], we examine whether HDM-SPDE can leverage the learned inductive bias in controllable coarse-to-fine image generation tasks. Figures 7 and A.4 show that HDM-SPDE can successfully perform inpainting, super-resolution, depixelation, and deblurring tasks without any additional training. HDM-SPDE is capable of generating images that resemble the background color to some extent for pixelated images and also generates samples leveraging the contours of blurred images. This is possible since low frequencies are filled with the frequencies of low-resolution images, whereas the range of high frequencies is filled with generated frequencies through the learned score model. Diffusion models trained by [52] fail to generate samples that capture the color or contours of pixelated or blurred images (see Appendix D.5).
Diffusion models in infinite-dimensional space have been actively investigated [18, 23, 29, 38, 48, 49]. [29] introduces a discrete-time denoising diffusion model in infinite-dimensional space, which provided one of the key inspirations for studying a continuous-time approach connecting with score-based methods through the SDE framework [52]. [18] is another discrete-time approach proposed to generalize neural processes [20], but it utilizes non-trace class white noise that causes an ill-posedness in the infinite-dimensional setting. See [14] for a reference of Hilbert space valued Wiener processes.
A challenging part of the continuous-time approach in Hilbert space is defining the score function without using the density function that no longer exists in the infinite-dimensional setting. [23] proposes using a discretized process to circumvent this problem and [48] suggests a continuous-time model based on spectral decomposition stemming from the Karhunen-Löève theorem; however, they do not include the general SDE framework [52] nor time-dependent operators. Based on [44], [19] only considers constant-time diffusion coefficients, which do not fully recover the desired framework requiring variance scheduling [46]. Section 2.3.1 shows the intersection between [19, 23] and our approach and how the proposed framework covers SDEs in infinite dimensions.
Concurrently, [38, 49] propose continuous-time models; however, their theoretical results are also confined to SDEs with constant-time operators since their approach is primarily grounded in semigroup theory [14], which has an intrinsic limitation when dealing with variable coefficients [39]. We note that the multiple noise scale case is revealed in [38, Section 4.4], yet it is constrained to discrete-time models. On the contrary, our approach is grounded in the study of variational approach to stochastic evolution equations [34, 35] and forward and backward Kolmogorov equations with functional derivatives [6, 15], which serves as a more established approach to identifying relations (Theorem 2.1) between time-dependent evolution operators and invariant measures of stochastic equations in Hilbert spaces [5]. It is noteworthy that proof of Theorem 2.1 (Appendix B.2) can be viewed as a generalization of the classical techniques employed by [1] and [24] in finite-dimensional spaces to general Hilbert spaces. Consequently it is able to fully recover the SDE framework proposed from [52], thereby enabling the variance scheduling [46], and building a bridge between the discrete-time model presented in IHDM [50] and the continuous-time score-based approach.
Similar to [11, 23, 29, 38], which combine diffusion models with neural operators [32, 37], we propose a modified time-conditioned FNO for the HDM-SDE in the functional and motion generation task, which has slightly different modules from the original neural operator. For the image generation task with the HDM-SPDE, however, we use the U-Net [51] to compare with the setting in variations of diffusion models [4, 27, 50], which utilize image transformations instead of drift coefficients.
6 Limitation and Conclusion
We extend the SDE framework proposed by [52] and propose a class of continuous-time score-based generative models based on stochastic evolution equations in Hilbert spaces. Our derivation of the time-reversal formula considering time-dependent coefficients enlarges the practical application of score-based generative models in function spaces and advances the performance of recent variations of diffusion models which use blurring [50] or pixelation [4]. Although performance gaps remain in image generation compared to the state-of-the-art results, we conclude that the proposed framework opens the potential for exploring various evolution operators and sample spaces in this area.
This work was supported by LG AI Research. This work was also supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT)(No. 2022-0-00612, Geometric and Physical Commonsense Reasoning based Behavior Intelligence for Embodied AI; No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University), and National Research Foundation of Korea(NRF) funded by the Korea government(MSIT)(2021R1A4A3033149). This work was supported by Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) & Gwangju Metropolitan City.
[55] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
[56] George Wynne and Andrew B Duncan. A kernel two-sample test for functional data. Journal of Machine Learning Research (JMLR), 23(73):1-51, 2022.
[57] Eunbi Yoon, Keehun Park, Jinhyeok Kim, and Sungbin Lim. Score-based generative models with lévy processes. In NeurIPS Workshop on Score-Based Methods, 2022. URL https://openreview.net/forum?id=ErzyBArv6Ue.
[58] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365, 2015. URL http://arxiv.org/abs/1506.03365.
[59] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5745-5753, 2019.