SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The theory of FDPs developed so far is valid for real, separable Hilbert spaces. Our goal now is to specify for which subclass of functions it is possible to perfectly reconstruct the original function given only its evaluation in a countable set of points. We present a generalization of the sampling theorem (Shannon, 1949), which allows us to move from generic Hilbert spaces to a domain which is amenable to a practical implementation of FDPs, and their application to common functional representation of data such as images, data on manifolds, and more. We model these functions as objects belonging to the set of square integrable functions over  $C^\infty$  homogeneous manifolds  $M$  (such as  $\mathbb{R}^N$ ,  $\mathbb{S}^N$ , etc...), i.e., the Hilbert space  $H = L_2(M)$ . Then, exact reconstruction implies that all the relevant information about the considered functions is contained in the set of sampled points.
First, we define functions that are band-limited:
Definition 1. A function  $x$  in  $H = L_2(M)$  is a spectral entire function of exponential type  $\nu$  (SE-  $\nu$ ) if  $|\Delta^{\frac{k}{2}}x|\leq \nu^{k}|x|, k\in \mathbb{N}$ . Informally, the "Fourier Transform" of  $x$  is contained in the interval  $[0,\nu ]$  (Pesenson, 2000).
Second, we define grids that cover the manifold with balls, without too much overlap. Those grids will be used to collect the function samples. Their formal definition is as follows:
Definition 2.  $Y(r, \lambda)$  denotes the set of all sets of points  $Z = \{p_i\}$  such that: i)  $\inf_{j \neq i} \text{dist}(p_j, p_i) > 0$  and ii) balls  $B(p_i, \lambda)$  form a cover of  $M$  with multiplicity  $< r$ .
Combining the two definitions, we can state the key result of this Section. As long as the sampled function is band-limited, if the samples grid is sufficiently fine, exact reconstruction is possible:
Theorem 2. For any set  $Z \in Y(r, \lambda)$ , any SE-  $\nu$  function  $x$  is uniquely determined by its set of values in  $Z$  (i.e.  $\{x[p_i]\}$ ) as long as  $\lambda < d$ , that is
$$
x = \sum_ {p _ {i} \in Z} x \left[ p _ {i} \right] m _ {p _ {i}}, \tag {14}
$$
where  $m_{p_i} : M \to H$  are known polynomials², and the notation  $x[p]$  indicates that the function  $x$  is evaluated at point  $p$ .
A precise definition of the value of the constant  $d$  and its interpretation is outside the goal of this work, and we refer the interested reader to Pesenson (2000) for additional details. For our purposes, it is sufficient to interpret the condition in Theorem 2 as a generalization of the classical Shannon-Nyquist sampling theorem (Shannon, 1949). Under this light, Theorem 2 has practical relevance, because it gives the conditions for which the sampled version of functions contains all the information of the original functions. Indeed, given the set of points  $p_i$  on which function  $x$  is evaluated, it is possible to reconstruct exactly  $x[p]$  for arbitrary  $p$ .
The uncertainty principle. It is not always possible to define Hilbert spaces of square integrable functions that are simultaneously homogeneous and separable, for all the manifolds  $M$  of interest. In other words, it is difficult in practice to satisfy both the requirements for FDPs to exist, and for the sampling theorem to be valid (see an example in Appendix C). Nevertheless, it is possible to quantify the reconstruction error, and realize that practical applications of FDPs are viable. Indeed, given a compactly supported function  $x$ , and a set of points  $Z$  with finite cardinality, we can upper-bound the reconstruction error  $\left\| \sum_{p_i \in Z} x[p_i] m_{p_i} - x \right\|_H$  with:
$$
\underbrace {\left\| \sum_ {p _ {i} \in Z} \left(x \left[ p _ {i} \right] - x ^ {\nu} \left[ p _ {i} \right]\right) m _ {p _ {i}} \right\| _ {H}} _ {\epsilon_ {1}} + \underbrace {\left\| \sum_ {p _ {i} \in Z} x ^ {\nu} \left[ p _ {i} \right] m _ {p _ {i}} - x ^ {\nu} \right\| _ {H}} _ {\epsilon_ {2}} + \underbrace {\left\| x ^ {\nu} - x \right\| _ {H}} _ {\epsilon_ {3}} = \epsilon , \tag {15}
$$
where  $x^{\nu}$  is the SE-  $\nu$  bandlimited version of  $x$ , obtained by filtering out – in the frequency domain – any component larger than  $\nu$ . The error  $\epsilon_{1}$  is due to  $x \neq x^{\nu}$ . The term  $\epsilon_{2}$  is the reconstruction error
Precisely, they are the limits of spline polynomials that form a Riesz basis for the Hilbert space of polyharmonic functions with singularities in  $Z$  (Pesenson, 2000).
due to finiteness of  $|Z|$ : the sampling theorem applies to  $x^{\nu}$ , but the corresponding sampling grid has infinite cardinality. Finally, the term  $\epsilon_3$  quantifies the energy omitted by filtering out the frequency components of  $x^{\nu}$  larger than  $\nu$ . This (loose) upper bound allows us to understand quantitatively the degree to which the sampling theorem does not apply for the cases of interest. Although deriving tighter bounds is possible, this is outside the scope of this work. What suffices is that in many practical cases, when functions are obtained from natural sources, it has been observed that functions are nearly time and bandwidth limited (Slepian, 1983). Consequently, as long as the sampling grid is sufficiently fine, the reconstruction error  $\epsilon$  is negligible.
We now hold all the ingredients to formulate generative functional diffusion models using the Hilbert space formalism and implement them using a finite grid of points, which is what we do next.
4 Score Network Architectural Implementations
We are now equipped with the ELBO (Equation (8)) and a score function  $s_{\theta}$  that implements the mapping  $H \times [0, T] \times \mathbb{R}^m \to H$ . We could then train the score by optimizing the ELBO and produce samples arbitrary close to the true data measure  $\rho_0$ . However, since the domain of the score function is the infinite-dimensional Hilbert space, such a mapping cannot be implemented in practice. Indeed, having access to samples of functions on finite grid of points is, in general, not sufficient. However, when the conditions for Theorem 2 hold, we can substitute – with no information loss –  $x \in H$  with its collection of samples  $\{x[p_i], p_i\}$ . This allows considering score network architectures that receive as input a collection of points, and not abstract functions. Such architectures should be flexible enough to work with an arbitrary number of input samples at arbitrary grid points, and produce as outputs functions in  $H$ .
4.1 Implicit Neural Representation
The first approach we consider in this work is based on the idea of Implicit Neural Representations (INRs) (Sitzmann et al., 2020). These architectures can receive as inputs functions sampled at arbitrary, possibly irregular points, and produce output functions evaluated at any desired point. Unfortunately, the encoding of the inputs is not as straightforward as in the Neural Fourier Operator (NFO) case, and some form of autoencoding is necessary. Note, however, that in traditional score-based diffusion models (Song et al., 2021), the parametric score function can be thought of as a denoising autoencoder. This is a valid interpretation also in our case, as it is evident by observing the term  $\mathbb{E}[X_0\mid X_t = x]$  of the true score function in Equation (11). Since INRs are powerful denoisers (Kim et al., 2022a), combined with their simple design and small number of parameters, in this Section we discuss how to implement the score network of FDPs using INRs.
We define a valid INR as a parametric family  $(\psi, t, \theta)$  of functions in  $H$ , i.e., mappings  $\mathbb{R}^m \times [0, T] \times \mathbb{R}^m \to H$ . A valid INR is the central building block for the implementation of the parametric score function, and it relies on two sets of parameters:  $\theta$ , which are the parameters of the score function that we optimize according to Equation (8), and  $\psi$ , which serve the purpose of building a mapping from  $H$  into a finite dimensional space. More formally:
Definition 3. Given a manifold  $M$ , a valid Implicit Neural Representation (INR) is an element of  $H$  defined by a family of parametric mappings  $n(\psi, t, \pmb{\theta})$ , with  $t \in [0, T]$ ,  $\pmb{\theta}$ ,  $\psi \in \mathbb{R}^m$ . That is, for  $p \in M$ , we have  $n(\psi, t, \pmb{\theta})[p] \in \mathbb{R}$ . Moreover, we require  $n(\psi, t, \pmb{\theta}) \in L_2(M)$ .
A valid INR as defined in Definition 3 is not sufficiently flexible to implement the parametric score function  $s_{\theta}$ , as it cannot accept input elements from the infinite-dimensional Hilbert space  $H$ : indeed, the score function is defined as a mapping over  $H \times [0,T] \times \mathbb{R}^m \to H$ , whereas the valid INR is a mapping defined over  $\mathbb{R}^m \times [0,T] \times \mathbb{R}^m \to H$ . Then, we use the second set of parameters  $\psi$  to condensate all the information of a generic  $x \in H$  into a finite-dimensional vector. When the conditions for Theorem 2 hold, we can substitute — with no information loss —  $x \in H$  with its collection of samples  $\{x[p_i], p_i\}$ . Then, we can construct an implicitly defined mapping  $g: H \times [0,T] \times \mathbb{R}^m \to \mathbb{R}^m$  as:
$$
g \left(\left\{x \left[ p _ {i} \right], p _ {i} \right\}, t, \boldsymbol {\theta}\right) = \arg \min  _ {\boldsymbol {\psi}} \sum_ {p _ {i}} \left(n (\boldsymbol {\psi}, t, \boldsymbol {\theta}) \left[ p _ {i} \right] - x \left[ p _ {i} \right]\right) ^ {2}. \tag {16}
$$
In this work, we consider the modulation approach to INRs. The set of parameters  $\psi$  are obtained by minimizing Equation (16) using few steps of gradient descent on the objective
$\sum_{p_i} (n(\psi, t, \theta)[p_i] - x[p_i])^2$ , starting from the zero initialization of  $\psi$ . This approach, also explored by Dupont et al. (2022b), is based on the concept of meta-learning (Finn et al., 2017). In summary, our method constructs mappings  $H \times [0, T] \times \mathbb{R}^m \to H$ , where the same INR is used first to encode  $x$  into  $\psi$ , and subsequently to output the value functions for any desired input point  $p$ , thus implementing the following score network:
$$
s _ {\boldsymbol {\theta}} (x, t) = - \left(\mathcal {S} (t)\right) ^ {- 1} (x - \exp (t \mathcal {A}) n \left(g \left\{\left\{x \left[ p _ {i} \right], p _ {i} \right\}, t, \boldsymbol {\theta}\right), t, \boldsymbol {\theta}\right)). \tag {17}
$$
4.2 Transformers
As an alternative approach, we consider implementing the score function  $s_{\theta}$  using transformer architectures Vaswani et al. (2017), by interpreting them as mappings between Hilbert spaces (Cao, 2021). We briefly summarize here such a perspective, focusing on a single attention layer for simplicity, and adapt the notation used throughout the paper accordingly.
Consider the space  $L_{2}(M)$ , with the usual collection of samples  $\{x[p_i], p_i\}$ . As a first step, both the "features"  $\{x[p_i]\}$  and positions  $\{p_i\}$  are embedded into some higher dimensional space and summed together, to obtain a sequence of vectors  $\{y_i\}$ . Then, three different (learnable) matrices  $\theta^{(Q)}, \theta^{(K)}, \theta^{(V)}$  are used to construct the linear transformations of the vector sequence  $\{y_i\}$  as  $\hat{Y}^{(Q)} = \{\hat{y}_i^{(Q)} = \theta^{(Q)} y_i\}$ ,  $\hat{Y}^{(K)} = \{\hat{y}_i^{(K)} = \theta^{(K)} y_i\}$ ,  $\hat{Y}^{(V)} = \{\hat{y}_i^{(V)} = \theta^{(V)} y_i\}$ . Finally, the three matrices  $\hat{Y}^{(Q,K,V)}$  are multiplied together, according to any variant of the attention mechanism. Indeed, different choices for the order of multiplication and normalization schemes in the products and in the matrices correspond to different attention layers Vaswani et al. (2017). In practical implementations, these operations can be repeated multiple times (multiple attention layers) and can be done in parallel according to multiple projection matrices (multiple heads).
The perspective explored in (Cao, 2021) is that it is possible to interpret the sequences  $\hat{y}_i^{(Q,K,V)}$  as learnable basis functions in some underlying latent Hilbert space, evaluated at the set of coordinates  $\{p_i\}$ . Furthermore, depending on the type of attention mechanism selected, the operation can be interpreted as a different mapping between Hilbert spaces, such as Fredholm equations of the second-kind or Petrov-Galerkin-type projections (Cao (2021) Eqs. 9 and 14).
While a complete treatment of such an interpretation is outside the scope of this work, what suffices is that it is possible to claim that transformer architectures are a viable candidate for the implementation of the desired mapping  $H \times [0,T] \times \mathbb{R}^m \to H$ , a possibility that we explore experimentally in this work. It is worth noticing that, compared to the approach based on INRs, resolution invariance is only learned, and not guaranteed, and that the number of parameters is generally higher compared to an INR. Nevertheless, learning the parameters of transformer architectures does not require meta-learning, which is a practical pain-point of INRs used in our context. Additional details for the transformer-based implementation of the score network are available in Appendix E.
Finally, for completeness, it is worth mentioning that a related class of architectures, the Neural Operators and NFOs (Kovachki et al., 2021; Li et al., 2020), are also valid alternatives. However, such architectures require the input grid to be regularly spaced (Li et al., 2020), and their output function is available only at the same points  $p_i$  of the input, which would reduce the flexibility of FDPs.
5 Training and sampling of FDPs
Given the parametric score function  $s_{\theta}$  from Equation (17), by simulating the reverse FDP, we generate samples whose statistical measure  $\chi_0$  is close in KL sense to  $\rho_0$ . Next, we explain how to numerically compute of the quantities in Equation (13), which is part of the ELBO in Equation (8), and how to generate new samples from the trained FDP (simulation of Equation (10)).
ELBO Computation. Equation (8) involves Equation (13), which requires the computation of the Hilbert space norm. The grid of points  $x[p_i]$  is interpolated in  $H$  as  $\sum_{i} x[p_i]\xi^i$ . Then, the norm of interest can be computed as:
$$
\left\| \sum_ {i} x [ p _ {i} ] \xi^ {i} \right\| _ {R ^ {\frac {1}{2}} H} ^ {2} = \langle R ^ {- \frac {1}{2}} \sum_ {i} x [ p _ {i} ] \xi^ {i}, R ^ {- \frac {1}{2}} \sum_ {i} x [ p _ {i} ] \xi^ {i} \rangle_ {H} = \sum_ {k = 1} ^ {\infty} (r ^ {k}) ^ {- 1} \left(\left\langle \sum_ {i = 1} ^ {N} x [ p _ {i} ] \xi^ {i}, e ^ {k} \right\rangle\right) ^ {2}. \tag {18}
$$
Depending on the choice of  $\xi^i, e^i$ , the sum w.r.t the index  $k$  is either naturally truncated or it needs to be further approximated by selecting a cutoff index value. Finally, training can then be performed by minimizing:
$$
\mathbb {E} _ {\mathbb {Q}} \left[ \int_ {0} ^ {T} \| \tilde {\gamma} _ {\boldsymbol {\theta}} (X _ {t}, t) \| _ {R ^ {\frac {1}{2}} H} ^ {2} \mathrm {d} t \right] \simeq \mathbb {E} _ {\sim (2 0)} \left[ \int_ {0} ^ {T} \sum_ {k = 1} ^ {\infty} \left(r ^ {k}\right) ^ {- 1} \left(\left\langle \sum_ {i = 1} ^ {N} \left(\tilde {\gamma} _ {\boldsymbol {\theta}} \left(\sum_ {i} X _ {t} [ p _ {i} ] \xi^ {i}, t\right) [ p _ {i} ]\right) \xi^ {i}, e ^ {k} \right\rangle\right) ^ {2} \mathrm {d} t \right]. \tag {19}
$$
Numerical integration. Simulation of infinite dimensional SDEs is a well studied domain (Debussche, 2011), including finite difference schemes (Gyöngy, 1998, 1999; Yoo, 2000), finite element methods and/or Galerkin schemes (Hausenblas, 2003a,b; Shardlow, 1999). In this work, we adopt a finite element approximate scheme, and introduce the interpolation operator, from  $\mathbb{R}^{|Z|}$  to  $H$ , i.e.  $\sum_{i}x[p_{i}]\xi^{i}$  (Hausenblas, 2003b). Notice that, in general, the functions  $\xi^i$  differ from the basis  $e^i$ . In addition, the projection operator maps functions from  $H$  into  $\mathbb{R}^L$ , as  $\langle x,\zeta^j\rangle ,\zeta^j\in H$ . Usually,  $L = |Z|$ . When  $\zeta^i = \xi^i$  the scheme is referred to as the Galerkin scheme. We consider instead a point matching scheme (Hausenblas, 2003b), in which  $\zeta^i = \delta [p - p_i]$  with  $\delta$  in Dirac sense, and consequently  $\langle x,\zeta^i\rangle = x[p_i]$ . Then, the infinite dimensional SDE of the forward process from Equation (9) is approximated by the finite  $(|Z|)$  dimensional SDE:
$$
\mathrm {d} X _ {t} [ p _ {k} ] = \left(\left\langle \mathcal {A} \sum_ {i} X _ {t} [ p _ {i} ] \xi^ {i}, \zeta^ {k} \right\rangle\right) \mathrm {d} t + \mathrm {d} W _ {t} [ p _ {k} ], \quad k = 1, \dots , | Z |. \tag {20}
$$
Similarly, the reverse process described by Equation (10) corresponds to the following SDE:
$$
\mathrm {d} \hat {X} _ {t} [ p _ {k} ] = \left(- \left\langle \mathcal {A} \sum_ {i} \hat {X} _ {t} [ p _ {i} ] \xi^ {i}, \zeta^ {k} \right\rangle + \left\langle R s _ {\boldsymbol {\theta}} \left(\sum_ {i} \hat {X} _ {t} [ p _ {i} ] \xi^ {i}, T - t\right), \zeta^ {k} \right\rangle\right) \mathrm {d} t + \mathrm {d} \hat {W} _ {t} [ p _ {k} ], \tag {21}
$$
$$
k = 1, \dots , | Z |.
$$
Equation (21) is a finite dimensional SDE, and consequently we can use any known numerical integrator to simulate its paths. In Appendix D we provide a complete toy example to illustrate our approach in a simple scenario, where we emphasize practical choices.
Despite a rather involved theoretical treatment, the implementation of FDPs is simple. We implemented our approach in JAX (Bradbury et al., 2018), and use WANDB (Biewald, 2020) for our experimental protocol. Additional details on implementation, and experimental setup, as well as more experiments are available in Appendix E.
We evaluate our approach on image data, using the CELEBA  $64 \times 64$  (Liu et al., 2015) dataset. Our comparative analysis with the state-of-the-art includes generative quality, using the FID score (Heusel et al., 2017), and parameter count for the score network. We also discuss (informally) the complexity of the network architecture, as a measure of the engineering effort in exploring the design space of the score network. We compare against vanilla Score Based Diffusion (SBD) (Song et al., 2021), From Data To Functa (FD2F) (Dupont et al., 2022a) which diffuses latent variables obtained from an INR, Infinite Diffusion ( $\infty$ -DIFF) (Bond-Taylor & Willcocks, 2023), which is a recent approach that is only partially suited for the functional domain, as it relies on the combination of Fourier Neural Operators and a classical convolutional U-NET backbone. Our FDP method is implemented using either MLP or Transformers. In the first case, we consider a score network implemented as a simple MLP with 15 layers and 256 neurons in each layer. The activation function is a Gabor wavelet activation function (Saragadam et al., 2023). In the latter case, our approach is built upon the UViT backbone as detailed by Bao et al. (2022). The architecture comprises 7 layers, with each layer composed of a self-attention mechanism with 8 attention heads and a feedforward layer.
We present quantitative results in Table 1, showing that our method FDP(MLP) achieves an impressively low FID score, given the extremely low parameter count, and the simplicity of the architecture. FD2F obtains a worse (larger) FID score, while having many more parameters, due to the complex
parametrization of their score network. As a reference we report the results of SBD, where the price to be pay to achieve an extremely low FID is to have many more parameters and a much more intricate architecture. Finally, the very recent  $\infty$ -DIFF method, has low FID-CLIP score (Kynkänniemi et al., 2022), but requires a very complex architecture and more than 2 orders of magnitude more parameters than our approach. Showcasing the flexibility of the proposed methodology, we consider a more complex architecture based on Vision Transformers (FDP(UViT)). These corresponding results indicate improvements in terms of image quality (FID score=11) and do not require meta-learning steps, but require more parameters (O(20M)) than the INR variant. To the best of our knowledge, none of related work in the purely functional domain (Lim et al., 2023; Hagemann et al., 2023; Dutordoir et al., 2022; Kerrigan et al., 2022) provides results going beyond simple data-sets. Finally, we present some qualitative results in Figures 1 and 2 clearly showing that the proposed methodology is capable of producing diverse and detailed images.
Table 1: Quantitative results, CELEBA data-set. (FID-CLIP (Kynkänniemi et al., 2022))
7 Conclusion, Limitations and Broader Impact
We presented a theoretical framework to define functional diffusion processes for generative modeling. FDPs generalize traditional score-based diffusion models to infinite-dimensional function spaces, and in this context we were the first to provide a full characterization of forward and backward dynamics, together with a formal derivation of an ELBO that allowed the estimation of the parametric score function driving the reverse dynamics.
To use FDPs in practice, we carefully studied for which subset of functions it was possible to operate on a countable set of samples without losing information. We then proceeded to introduce a series of methods to jointly model – using only a simple INR or a Transformer – an approximate functional representation of data on discrete grids, and an approximate score function. Additionally, we detailed practical training procedures of FDPs, and integration schemes to generate new samples.
The implementation of FDPs for generative modeling was simple. We validated the viability of FDPs through a series of experiments on real images, where we show, while only using a simple MLP for learning the score network, extremely promising results in terms of generation quality.
Like other works in the literature, the proposed method can have both positive (e.g., synthesizing new data automatically) and negative (e.g., deep fakes) impacts on society depending on the application.
GF gratefully acknowledges support from Huawei Paris and the European Commission (ADROIT6G Grant agreement ID: 101095363). MF gratefully acknowledges support from the AXA Research Fund and the Agence Nationale de la Recherche (grant ANR-18-CE46-0002 and ANR-19-P3IA-0002).
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations (ICLR), 2022.
Hyek Yoo. Semi-discretization of stochastic partial differential equations on  $r$  by a finite-difference method. Mathematics of computation, 69(230):653-666, 2000.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adbelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806, 2020.
Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M. Susskind, and Miguel Ángel Bautista. Diffusion probabilistic fields. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=ik91mY-2GN.
Supplementary Material: Continuous-Time Functional Diffusion Processes
A Reverse Functional Diffusion Processes
In this Section, we review the mathematical details to obtain the backward FDP discussed in Theorem 1. Depending on the considered class of noise, different approaches are needed. First, we present in Appendix A.1 the conditions to ensure existence of the backward process, which we use if the  $C$  operator is an identity matrix,  $C = I$ . Then we move to a different approach in Appendix A.2 for the case  $C \neq I$ .
A.1 Follmer Formulation
The work in Föllmer (1986) is based on a finite entropy condition, which we report here as Condition 1. One simple way to ensure that the condition is satisfied is to assume: