Diffusion Probabilistic Fields
Authors: Peiye Zhuang
Samira Abnar
Jiatao Gu
Alex Schwing
Joshua M. Susskind
Miguel Ángel Bautista
Venue: ICLR 2023 poster (2023)
Keywords: Generative Models
Field Representation
Diffusion Models

Abstract: Diffusion probabilistic models have quickly become a major approach for generative modeling of images, 3D geometry, video and other domains. However, to adapt diffusion generative modeling to these domains the denoising network needs to be carefully designed for each domain independently, oftentimes under the assumption that data lives in a Euclidean grid. In this paper we introduce Diffusion Probabilistic Fields (DPF), a diffusion model that can learn distributions over continuous functions defined over metric spaces, commonly known as fields. We extend the formulation of diffusion probabilistic models to deal with this field parametrization in an explicit way, enabling us to define an end-to-end learning algorithm that side-steps the requirement of representing fields with latent vectors as in previous approaches (Dupont et al., 2022a; Du et al., 2021). We empirically show that, while using the same denoising network, DPF effectively deals with different modalities like 2D images and 3D geometry, in addition to modeling distributions over fields defined on non-Euclidean metric spaces.

Published as a conference paper at ICLR 2023
DIFFUSION PROBABILISTIC FIELDS
Peiye Zhuang" Samira Abnar .Jiatao Gu² Alexander G. Schwing",
Joshua M. Susskind Miguel Ángel Bautista² Stannfor University AAple University of Illinois at Urbana- Urbana-Champaa Champaign "peiye "peiye@stanford. edu (abbar, jgu32, jsusskind, edd
mbaut cnntiistaaat..o @apple. com
ABSTRACT
Diffusion probabilistic models have quickly become a major approach for gen- erative modeling of images, 3D geometry. video and other domains. However, to adapt diffusion generative modeling to these domains the denoising network needs be carefully designed for each domain independently, oftentimes under the assumption that data lives in Euclidean grid. In this paper we introduce Diffusion Probabilistic Fields (DPF), a diffusion model that can learn distributions over continuous functions defined over metric spaces, commonly known fields. We extend the formulation of diffusion probabilistic models to deal with this field parametrization in explicit way, enabling to define end-to-end learning algorithm that side-steps the requirement of representing fields with latent vectors in previous approaches (Dupont et al. 2022a Du et al.) 202T). We empirically show that, while using the same denoising network, DPF effectively deals with different modalities like 2D images and 3D geometry, in addition to modeling distributions over fields defined on non-Euclidean metric spaces.
INTRODUCTION
Diffusion probabilistic modeling has quickly become central approach for learning data distributions,
obtaining impressive empirical results across multiple domains like images (Nichol & Dhariwal 2021), videos (Ho al., 2022) or even 3D geometry (Luo Hu, 202T). In particular, Denoising Diffusion Probabilistic Models (often referred to as DDPMs or diffusion generative models) (Ho extension (Song et al.) 202 lb) both present a training objective that is more stable than precursors like generative adversarial nets
ct al. 2020 Nichol Dhariwal 2021) and their continuous-ting
(GANs) (Goodfellow et al. 2014) or energy-based based models (EBMs) (Du et al. 2020). In addition, diffusion generative models have shown to empirically outperform GANs in the image domain
(Dhariwal & Nichol 2021 and to suffer less from modc-seeking pathologies during training (Kodali
al. 2017).
diffusion generative model consists of three main components: the forward (or diffusioon) process. the backward (or inference) process, and the denoising network (also referred to as the score networq"] due to its equivalence with denoising score-matching approaches Dickstein et al. (2015)). A substan- tial body of work has addressed different definitions of the forward and backward processes (Rissanen et al. 2022 Bansal et al. 2022) Song et al., 2021a),, focusing on the image domain. However, there are two caveats with current diffusion models that remain open. The first one that data is typically as- sumed to live on discrete Euclidean grid (exceptions include work on molecules (Hoogeboom et al. 2022] and point clouds (Luo & Hu (02T)). ) The second one is that the denoising network is heavily tuned for each specific data domain, with different network architectures used for images (Nichol &
Dhariwal 2021). video (Ho et al. 2022). or geometry (Luo & Hu 2021).
In order to extend the success of diffusion generative models to the large number of diverse areas in science and engineering, unification of the score formulation is required. Importantly, a unification enables use of the same score network across different data domains exhibiting different geometric structure without requiring data to live in or to be projected into a discrete Euclidean grid.
Work was completed while P.Z. was an intern with Apple We use the terms score network/function and denoising network/function exchangeahly in the paper.

---

Published as a conference paper at ICLR 2023
To achieve this, in this paper, introduce the Diffusion Probabilistic Field (DPF). DPFs make progress towards the ambitious goal of unifying diffusion generative modeling across domains by learning distributions over continuous functions. For this, take functional view of the data, interpreting a data point æ: E Rd as a function f: M - y (Dupont et al. 2022a; Du et al. (021). The function f maps elements from metric space M to signal space J This functional view of the data is commonly referred to as a field representation (Xie et al. 2022), which we use to refer to functions of this type. An illustration of this field interpretation is provided in Fig. [ Using the image domain as an illustrative example we can see that one can either interpret images as multidimensional array 2; E f xxw x R or as field f R + Rs that maps 2D pixel coordinates to RGB values. This field view enables a unification of seemingly different data domains under the same parametrization. For instance, 3D geometry data is represented via f: IR$ - IR, and spherical signals become fields : IRd.
In an effort to unify generative modeling across different data domains, field data representations have shown promise three recent approaches: From data to functa (Functa) (Dupont al. 2022a),
GEnerative Manifold learning (GEM) (Du et al. 202T) and Generative Adversarial Stochastic Process (GASP) (Dupont al. 2022b). The first two approaches adopt a latent field parametrization (Park jet al. 2019), where a field network is parametrized via a Hypernetwork (Ha et al. (017)) that takes as input trainable latent vector. During training, latent vector for each field is optimized in an
initial reconstruction stage (Park et al. 2019). In Functa (Dupont et al.] (022a) the authors then
propose learn the distribution of optimized latents in an independent second training stage, similar to the approach by Rombach et al. (2022); ); Vahdat et al. (2021). Du et al. (2021) define additional latent neighborhood regularizers during the reconstruction stage. Sampling
then performed in non-parametry way: one chooses random latent vector from the set of optimized latents and projects it into its local neighborhood before adding Gaussian noise. See Fig. [ F
a visual summary of the differences between Functa (Dupont et al., 2022a), GEM (Du et al. 202 and our DPF, GASP (Dupont et al., 2022b) employs GAN paradigm: the generator produces a field while the discriminator operates on discrete points from the field, and distinguishes input source, i.e., either real or generated.
In contrast to prior work (Dupont al. 2022a; Du ct al. 2021| Dupont et al. 2022b), we formulate diffusion generative model over functions in a single-stage approach. This permits efficient end-to-end training without relying initial reconstruction stage or without tweaking the adversarial game, which we empirically find to lead to compelling results. Our contributions are summarized as follows:
We introduce the Diffusion Probabilistic Field (DPF) which extends the formulation of diffusion generative models to field representations.
We formulate a probabilistic generative model over fields in a single-stage model using an explicit field parametrization, which differs from recent work (Dupont al.) 2022a Du
et al. 2021) and simplifies the training process by enabling end-to-end learning.
We empirically demonstrate that DPF can successfully capture distributions over functions across different domains like images, 3D geometry and spherical data, outperforming recent
work (Dupont et al., 2022a Du et al. 2021 Dupont et al. 2022b).
2 BACKGROUND: DENOISING DIFFUSION PROBABILISTIC MODELS
Denoising Diffusion Probabilistic Models (DDPMs) belong the broad family of latent variable models. We refer the reader to Everett 2013) for an in depth review. In short, to learn a parametric data distribution po (0())) from an empirical distribution of finite samples q(xr), DDPMs reverse a diffusion Markov Chain (i.e., the forward diffusion process) that generates latents T1.r by gradually adding Gaussian noise to the data 2ig ~ q(xo) for T time-steps as follows:
q( (x)))= == N ( Vārzo, (1 a.))..
Here, ã is the cumulative product of fixed variances with handcrafted scheduling up time-step t. Ho et al. (2020) highlight two important observations that make training of DDPMs efficient: i) Eq. (1 adopts sampling in closed form for the forward diffusion process. ii) reversing the diffusion
2

---

Published as a conference paper at ICLR 2023
Multidim.
Field
Diffusion Probabilistic Fields (DPF)
M
f:M
Polf--115)
f0
Ruxh x R23
f RR + R
fo
y
Rwxxxd
R
f F 7 IR
M
Suxh x
f f: R
The left panel shows the parametrization of each data domain field. For visualization purposes we use the color red denote the input to the field function (e.g., metrie space M where the field is defined). In the right panel we show the graphical model of DPF, a diffusion generative model to capture distributions over fields. In DPF the latent variables are fields fI: that can be evaluated continuously. By taking field parametrization of data we unify diffusion generative modeling across different domains (images, 3D shapes, spherical images) using the same score network implementation.
Figure 1:
process is equivalent learning a sequence of denoising (or score) networks co. with tied weights. Reparametrizing Eq. T as vā, + vI a.c results in the "simple" DDPM loss
La = (tr-00.)).m000(0)))....
[lle - (eal-calvaa colvarz + v1 - ā,c, +)1l]
(2)
which makes learning of the data distribution (xo) both efficient and scalable.
At inference time, we compute Zo (0(o)) via ancestral sampling (Ho et al. 2020). Concretely, we start by sampling xy ~ (O I) and iteratively apply the score network eg to denoise Ty, thus reversing the diffusion Markov Chain to obtain Tp. Sampling T;-1 po p xx)) is equivalent to computing the update
at co( t))
+
(3)
Vae
vI az.
where at each inference step stochastic component z ~ NN00 I) is injected, resembling sampling via Langevin dynamics (Welling
Teh 201T).
A central part of the learning objective in Eq. is the score network eg, which controls the marginal distribution pe (= Notably, score networks are heavily tailored for each specific data domain. For example, in the image domain, score networks are based on a UNet (Ronneberger et al.) 2015) with multiple self-attention blocks (Nichol & Dhariwal 202T). In contrast, for 3D structures like molecules, score networks are based graph neural nets (Hoogeboom et al.) 2022). To unify the design of score networks across data domains, in this paper, we present the diffusion probabilistic field (DPF), which introduces a unified formulation of the score network that can be applied to multiple domains by representing data samples as fields.
DIFFUSION PROBABILISTIC FIELDS
diffusion probabilistic field (DPF) is a diffusion generative model that captures distributions over fields. We are given observations in the form of an empirical distribution g(fo) over fields (living in an unknown field manifold) where a field fu . + y maps elements from a metric space M to a signal space y. For example, in the image domain an image pixel coordinates to RGB values fo + R², In DPF the latent variables fie are fields that can be continuously evaluated. To tackle the problem of learning diffusion generative model over fields
be defined as a field that maps 2D
3

---

Published as a conference paper at ICLR 2023
Stage 1: reconstruction (ie. learning %0)
Stage 2: learning pe(za)
f (Zo) x R² + +33
Lateat manboold
K0 E
MM-
y
Latent field (Functa, GEM)
Functa Bo po(o))
yp
Latint manifak
GEM alz g(aa)) (1 a.RR, 1.
Fied manifold
Jn fu: Rd
fo: - R..
Single stage: learning score field Fg
Quary
Contest paice
Explicit field (DPF)
ED tc
So r Jo
manifold
Figure 2: In DPF we explicitly parameterize a field by a set coordinate-signal pairs, or context pairs, as opposed to latent vector 2e in Funeta (Dupont et al. 2022a) GEM (Du et al. 2021). This explicit field parametrization allows us to side-step the reconstruction training stage of prior approaches and instead directly model the distribution of fields rather than the distribution of latents that encode fields.
we need to successfully deal with the infinite dimensional nature of the field representation in the forward process as well as in the score network and backward process.
We adopt an explicit field parametrization, where a field is characterized by set of coordinate-signa pairs {me, ((,,o))), m. M, V(c.0) V. which we denote context pairs. For clarity we row-wise stack context pairs and refer the resulting matrix via Co [Me; (cc.))). Here, M, denotes the coordinate portion of all context pairs and Y (c.o) denotes the signal portion of all context pairs at time 1 = O. Note that the coordinate portion does not depend on the time-step by design This is a key difference with respect to Functa (Dupont al., 2022a) or GEM (Du et al. 2021). both of which adopt a latent parametrization of fields, where a learnt field f (22) x M y is parametrized by latent weight vector zu through hypernetwork model L (Ha et al., 2017).
Using a latent parametrization forces a reconstruction stage in which latents 2g are first optimized to reconstruct their corresponding field (Park et al.) 2019) Du et al. (2021) (i.e., defining
empirical distribution of latents ((2) living in an unknown latent manifold). A prior Py(2)) over latents is then learnt independently in a second training stage (Dupont al. (022a).. In contrast, our explicit field parametrization allows us to formulate a score field network, enabling DPF to directly model a distribution over fields, which results in improved performance (cf. Sect. Fig. depicts the differences between latent field parametrizations in Functa (Dupont et al. 2022a) and GEM (Du et al.
2021), and the explicit field parametrization in DPF.
Adopting explicit field parametrization, we define the forward process for context pairs by diffusing the signal and keeping the coordinates fixed. Consequently the forward process for context pairs reads as follows:
C = MM, Y(()t) = (a.Y( + VI ayec].
where e, NO, I) is noise vector of the appropriate size. We now the task of formulating a score network for fields. By definition, the score network needs to take as input the context pairs (i.e., the field parametrization). and needs to accept being evaluated continuously in M in order be
We assume the geometry of fields does not change.
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 Training
Context pairs: Co
1: repeat
Query puirs: Qo
2: (Co, Qo) ~ Uniform(q(fo))
3: Lo Uniform((1 ((.....,((
4: 6, Vo,, I), I
{(oo
C. [M., v (a.0) + v ārEd]
R
6: Q [M,, v Y (0.0) + vI dtcy)
Diffusod contexti pains:
C₁
Diffused query paits: Q
Take gradient descent step
Ve eg co(C,, L, Q1)1
co (Ce, L, Q+)
8: until converged
Figure 3: Left: DPF training algorithm. Right: Visual depiction of training iteration for field in the image domain. See Sect. 3 for definitions.
field. We do this by using query pairs {m.g. O) Equivalently to context pairs, we row-wise stack query pairs and denote the resulting matrix as Qo = [M, (g,,))) Note that the forward diffusion process is equivalently defined for both context and query pairs:
Q: = Mg ((.4) = VarY(ggo) (q.0) + v1- azta];
(5)
where E, ~ IO, is noise vector of the appropriate size. However, the underlying field is solely defined by context pairs, and query pairs merely act as points on which to evaluate the score network. The resulting score field model is formulated as follows, Eg = co(C,, I, Qt).
The design space for the score field model is spans all architectures that can process data in the form of sets, like transformers or MLPs. In particular, efficient transformer architectures offer straightforward way to deal with large numbers of context and query pairs, well as good mechanism for query pairs to interact with context pairs via attention. For most our experiments we use PerceiverIO (Jacgle et al.) 2022), efficient transformer encoder-decoder architecture,
Fig.
11 and Sect. B for details. In addition, in Sect. D we show that other architectures like vanilla Transformer Encoders (Vaswani et al.) (007)) and MLP-mixers (Tolstikhin al. 2021) are also viable candidates offering a very flexible design space without sacrificing the generality of the formulation
of DPF.
Using the explicit field characterization and the score field network, we obtain the training and inference procedures in Alg. D and Alg. respectively, which are accompanied by illustrative examples for field representation of images. For training, we uniformly sample context and query pairs from fo ~ Uniform(q(ff) (fo)) and only corrupt their signal using the forward process Eq. (4) and Eq. 5). We then train the score field network cp to denoise the signal in query pairs, given context pairs (see Fig. [or a visualization using a PerceiverlO implementation). During sampling, to generate field fo po (fo) we first define query pairs QT [,, Y (g.1)) (N 1)] on which the field will be evaluated. Note that the number of points on which the score field evaluated during sampling has to be fixed (e.g., to generate an image with 32 x 32 1024 pixels we define 1024 query pairs Qr at time = T). We then let context pairs be a random subset of the query pairs. We use the context pairs to denoise query pairs and follow ancestral sampling as in the vanilla DDPM (o et al. 20200 Note that during inference the coordinates of the context and query pairs do not change, only their corresponding signal value. The result of sampling is given by Qo which is the result of evaluating fo at coordinates M..
4 EXPERIMENTAL RESULTS
We present results on multiple domains: 2D image data, 3D geometry data, and spherical data. Across all domains we use the same score network architecture. We only adjust the dimensionality of the metric space M and the signal space ) as well the network capacity (i.e., number of layers, hidden
Mooe efficient sampling approaches like DDIM Song al. 2021a trivially adapted DPF.
5

---

Published as a conference paper at ICLR 2023
Algorithm 2 Sampling
Contect pains: Cy
2018 quarry pains
Query palns
I: Qr = M ((.) ~ (0,
2: Cr C Qr
b Random subset R
3: for TT- T.................... do
4: N. ift e,else =0
R
5: Yau 1) yat (Yin.))
1-0g Vi-ai +, t0
2018 pairs
Q M (1.
Contect poins C.-1
7: C--
b Same subset as in step R
Query pairs Qu1
t=t-1
8: end for
9: return evaluated coordinates Ma
R
Figure 4: Left: DPF sampling algorithm. Right: Visual depiction of the sampling process for field in the image domain.
CelebA-HQ 64
FID Pre. Rec.
CIFAR-10 32
FID 1
IS f
VAE (Kingma & Welling
2014)
175.33 0.799 0.001
PixelIQN Ostrovski et al.
2018
49.46
5.29
StyleGANZ Karras
2020a
5.90 0.618 0.481
NCSNv2 Song Ermon
2020
31.75
StyleGANZ (Karras ct al.
2020a)
3.26
9.74
Functa (Dupont al.
2022a)
40.40 0.577 0.397
DDPM (Ho ét al. 2020
3.17
9.46
GEM (Du et al. 2021
30.42 0.642 0.502
GASP Dupont et al.
2022b)
13,50 0.836 0.312
GEM (Du et al. 2021
23.83
8.36
DPF (ours)
13.21 0.866 0.347
DPF (ours)
15.10
8.43
Table Quantitative evaluation of image gen-
Table Quantitative evaluation of image gen- eration on CIFAR-10 (Krizhevsky ).09). The middle bar separates domain-specific (top) from domain-agnostic approaches (bottom).
eration on CelebA-HQ (Karras et al. 2018). The
middle bar separates domain-specific (top) from domain-agnostic approaches (bottom).
units per layer, etc.) when needed. We implement the field score network eæ using a PerceiverIO architecture (Jaegle et al. [2022), an efficient transformer that enables us to scale the number of both context pairs and query pairs. Additional architecture hyperparameters and implementation details are provided in the appendix.
4.1 2D IMAGES
We present empirical results on two standard image benchmarks: CelebA-HQ (Karras al. 2018) 642 and CIFAR-10 (Krizhevsky 2009) 32". All image datasets are mapped to their field representation, where an image E Rxxex is represented as function f R R defined by coordinate- RGB pairs In Tab. T and Tab. 2 we compare DPF with Functa (Dupont al. 2022a). GEM (Dul ct al. 202T) and GASP Dupont et al., 2022b). which are domain -agnosii generative models that employ field representations. For completeness,
also report results for the domain-specific VAE (Kingma & Welling 2014). StyleGAN2 (Karras et al. (0020a) and DDPM (Ho et al. (020).. Similarly, on CIFAR-10 10 data we compare with multiple domain-specific methods including auto-regressive (Ostrovski et al. 2018) and score-based (Song & Ermon 2020) approaches. We report Fréchet Inception Distance (FID) (Heusel al. 2017). Inception Score (IS) (Salimans
2016) and
precision/recall metrics (Sajjadi et al. (018) for different datasets.
We observe that DPF obtains compelling generative performance on both CelebA-HQ 642 and CIFAR-10 data. In particular, DPF outperforms recent approaches that aim to unify generative modeling across different data domains such as Functa (Dupont al. 2022a) and GEM (Du et al.. 2021). In particular, we observe that DPF obtains the best Precision score across all donain-agnostic approaches on CelebA-HQ 64 KKrras et al. 2018), as well as very competitive FID scores. One interesting finding is that GASP Dupont al. (002bb) reports comparable FID score than DPF. However, when inspecting qualitative examples shown in Fig. 5 we observe GASP samples to contain artifacts typically obtained by adversarial approaches. In contrast, samples generated from DPF are more coherent and without artifacts. We believe the texture diversity caused by these artifacts to be the reason for the difference in FID scores between GASP and DPF (similar findings were discussed

---

Published as a conference paper at ICLR 2023
GEM (Du et al. 2021)
Functa (Dupont et al. 2022a)
GASP (Dupont al. 20226)
DPF (ours)
Figure 5: Qualitative comparison of different domain-agnostic approaches on CelebA-HQ 64 (Karras al. 2018). DPF generates diverse and globally consistent samples, without the background
blurriness of two-stage approaches like Functa (Dupont et al. 2022a) and GEM (Du et al. 2021 I) that rely on an initial reconstruction step, or the artifacts of adversarial approaches like GASP (Dupont et al. 2022b). All the images in this figure have been generated by a model, i.e., we do not directly reproduce images from the CelebA-HQ dataset (Karras al. 2018).
GEM (Du et al. 2021)
DPF (ours)
Figure 6: Qualitative comparison of GEM (Du et al. 2021) and DPF on CIFAR-10 Krizhevsky 2009). DPF generates crisper samples than GEM (Du et al. 2021) while also better capturing the empirical distribution.
in (Dupont al. 2022a) Finally, when studying empirical results on CIFAR-10 shown in Tab. 2 we again see that DPF performs better than GEM (Du et al. 2021) both in terms of FID and IS.
Notably, we can observe a gap between domain domain-agnostic and the most recent domain- -ppecffc approaches such as StyleGAN2 (Karras et al., (020a) and DDPM (Ho et al. 2020). This gap is result of two main factors. First, domain-specific approaches can incorporate design choices tailored to their domain in the score network (i.e., translation equivariance in CNNs for images). Second, the training and inference settings are typically different for the domain-agnostic and fomain-speciff approaches reported in Tab. I and Tab. 2 To further study this issue we refer readers to Sect. A where we compare the performance of DPF and DDPM (Ho et al. 2020 using the same training and evaluation settings.
Finally, we show qualitative generation results for domain-agnostee approaches on CelebA-HO 642 (Karras et al., 2018) and CIFAR-10 (Krizhevsky) 2009) in Fig. 5land Fig. respectively. We note that for CelebA-HQ, DPF generates diverse and globally consistent samples, without the blurriness (particularly in the backgrounds) observed in two-stage approaches (Dupont al.) 2022a Du et al. 2021) that rely on an initial reconstruction step, or the artifacts of adversarial approaches (Dupont al. 2022b). When evaluated on CIFAR-10 10 KKrizhessky 2009). we see that DPF generates crisper
and more diverse results than GEM (Du et al. 2021)
4.2 3D GEOMETRY
We now turn to the task of modeling distributions over 3D objects and present results on the ShapeNet dataset (Chang al. 2015). We follow the settings in GEM (Du al. 2021) and train model on
7

---

Published as a conference paper at ICLR 2023
ShapeNet 64
Coverage
MMD
Latent GAN (Chen & Zhang 2019)
0.389
0.0017
GASP (Dupont al. 2022b
0.341
0.0021
GEM Duu al. 2021
0,409
0.0014
DPF (ours)
0.419
0.0016
Table 3: Quantitative evaluation of 3D geometry gener-
ation on ShapeNet (Chang al. 2013). DPF outperforms prior approaches both in terms of Coverage and MMD met- rics.
~ 35k ShapeNet objects, where each object is represented as a voxel grid at a 64 resolution. Each object then represented function f R R. Following the settings of GEM (Du al. 2021), we report coverage and MMD metrics (Achlioptas et al. 2018) computed from sampling 2048 points on the meshes obtained from the ground truth and generated voxel grids. We compare them using the Chamfer distance. In Tab, we compare DPF performance with 3 baselines: Latent
GAN (Chen Zhang 2019). GASP (Dupont al.) (022b) and GEM (Du al. (02T). We train DPF 32 resolution. During sampling evaluate 32" query pairs the score field, then reshape the results to a 3D grid of 32 resolution and tri-linearly up-sample to the final 64 resolution for computing evaluation metrics.
DPF outperforms both GEM (Du et al. 2021 and GASP (Dupont et al. 2022b) in learning the multimodal distribution of objects in ShapeNet, shown by the coverage metric. In addition, while GEM (Du et al. 2021) performs better in terms of MMD, we do not observe this difference when visually comparing the generated samples shown in Fig. 7 We attribute this difference in MMD scores to the fact that MMD over-penalizes fine-grained geometry.
4.3 DATA ON $
Straightforwardly, DPF can also learn fields that are not defined in the Euclidean plane. demon- strate this, we show results on signals defined over the sphere. In particular, following [Chhn et al. (018).. we use a stereographic projection to map image data onto the sphere. Hence, each resulting example is represented
the Driscoll-Healy algorithm (Driscoll & Healy 1994) and sample points at a resolution of 322 and 642 for spherical MNIST (LeCun al.) 1998) and AFHQ (Choi et al. (020)) data, respectively. In Fig. we show the distribution real examples for MNIST (LeCun et al. 1998) and AFHQ (Choi et al. 2020) images projected on the sphere well the samples generated by our DPF. Unsurprisingly, since DPF is agnostic to the geometry of the metric space M, it can generate crisp and diverse samples for fields defined on the sphere.
field fo 32 Rd To uniformly sample points in S we
RELATED WORK
Generative modeling has advanced significantly in recent years with generative adversarial nets (GANs) (Goodfellow et al. 2014 Mao et al.) 2017 Karras et al., 2020h) and variational auto- encoders (VAEs) (Vahdat & Kautz 2020) showing impressive performance. Even more recently, liffusion-based generative modeling has obtained remarkable compelling results (Dickstein et al.
2015 Ho al. 2020 Song et al. 2021b).
GASP
Dupont et al. 2022b
GEM Du et al. 2021
DPF (ours)
Figure 7: Qualitative comparison of different Nomain-agnostic approaches on ShapeNet (Chang
al. 2015
8

---

Published as a conference paper at ICLR 2023
Real samples for spherical MNIST.
Real samples for spherical AFHQ.
DPF samples (ours).
DPF samples (ours).
Figure 8: Qualitative comparison of empirical and generated samples for spherical versions of MNIST and AFHQ (Choi et al. 2020
The formulation developed in DPF is orthogonal to the body of work on Riemannian generative models (Bortoli et al. 2022 Gemici al.,, 2016: Rozen et al. 2021). The goal in Riemannian generative modeling is to explicitly constraint the learned density to Riemannian manifold structure. For example, Riemannian generative model can learn a distribution of points æ E S on the 2D sphere S², explicitly enforcing that any generated samples lie on the sphere. In contrast, DPF learns a generative model over distribution of multiple samples of signals defined on the sphere, or any other metric space.
The DPF formulation also differs from the recently introduced Functa (Dupont et al. 2022a). GEM
Duu et al. (22T)) and GASP (Dupont et al. 20226). The first two approaches adopt a latent field parametrization (Park et al. 2019) and a two-stage training paradigm. However, different from our work, the field network in Functa and GEM is parametrized via a hypernetwork (Ha et al.) (017)) that takes as input a trainable latent vector. During training, a small latent vector for each field is optimized in initial auto-decoding probabilistic model is learned on the latent vectors. GASP (Dupont et al. (022b) leverages a GAN whose generator produces field data whereas point cloud discriminator operates on discretized data and aims to differentiate input source, i.e., either real or generated. Two-stage approaches like Functa (Dupont et al. (022a) GEM (Du et al.) (021) make training the probabilistic model in the second stage more computationally efficient than DPF. This training efficiency of the probabilistic model often comes at the cost of compressing fields into small latent vectors in the first stage, which has non-negligible computational cost, specially for large datasets of fields.
compression) stage (Park et al. [001)). In the second stage,
The formulation introduced in DPF has is closely related to recent work on Neural Processes (NPs) Gamelo et al.) 2018 Kim et al. 2019: Dutordoir et al.] 2022), which also learn distributions over functions via context and query pairs. As opposed the formulation of Neural Processes which optimizes an ELBOKKingma & Welling (014) we formulate DPF as denoising diffusion process in function space, which results in a robust denoising training objective and a powerful iterative inference process. In comparison with concurrent work in Neural Processes (Garnelo et al., 2018 Kim et al. 2019: Dutordoir et al. 2022) we do not explicitly formulate a conditional inference problem and look at the more challenging task of learning
unconditional generative model over fields. We extensively test our hypothesis on complex field distributions for 2D images and 3D shapes, and on distribution of fields defined over non-Euclinn -uc-idelid geometry.
CONCLUSION
In this paper made progress towards modeling distributions over fields by introducing DPF, A diffusion probabilistic model that directly captures a distribution over fields without resorting to a initial reconstruction stage (Dupont et al. 2022al Du et al.) 2021) tweaking unstable adversarial approaches (Dupont et al. 2022b). We show that DPF can capture distributions over fields in different domains without making any assumptions about the data. This enables us to use the same denoising architecture across domains obtaining satisfactory results. In addition, we show that DPF can be used to learn distributions over fields defined on Euclliidn geometry.

---

Published as a conference paper at ICLR 2023
7 ETHICAL STATEMENT
When considering ethical impact of generative models like DPF few aspects that need attention the use generative models for creating disingenuous data, e-g-. "DeepFakes" |Mirsky & Lee (2021), training data leakage and privacy Tinsley et al. (202I), and amplification of the biases present training data Jain al. 2020). For in-depth review of ethical considerations in generative modeling we refer the reader to Rostamzadeh et al. (202T). In addition, we want to highlight that any face images we show are generated from our model. We do not directly reproduce face images from any dataset.
8 REPRODUCIBILITY STATEMENT
We take great care in the reprodueibility of our results. We provide links to the public implementations that can be used replicate our results in Sect. [ and Sect. [ well as describing all training parameters in Tab. 6 All of the datasets we report results on are public and can be freely downloaded.
REFERENCES
P. Achlioptas, P. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and generative models for 3d point clouds. In ICML, 2018.
A. Bansal, Borgnia, H. Chu, Li, H. Kazemi, F. Huang, M. Goldblum, J. Geiping, and T. Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv, 2022.
V. Bortoli, E. Mathieu, M. Hutchinson, J. Thortton, Y. Teh, and A. Doucet. Riemannian score-based generative modeling. arXiv, 2022.
A. Chang, T. Funkhouser, L. Guibas, al. Shapenet: An information-rich 3d model repository, arXiv, 2015.
7. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
Y. Choi, Y. Uh, J. Yoo, and J. Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Cohen, M. Geiger, Köhler, and M. Welling. Spherical cnns. In ICLR, 2018.
Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
J. Driscoll and D. Healy. Computing fourier transforms and convolutions on the 2-sphere. Advances in applied mathematics, 1994.
Y. Du, Li, and Mordatch. Compositional visual generation with energy based models. In NeurlPS, 2020.
Y. Du, K. Collins, J. Tenenbaum, and V. Sitzmann. Learning signal-agnostic manifolds of neural fields. In NeurIPS, 2021.
E. Dupont, Kim, S. Eslami, D. Rezende, and D. Rosenbaum. From data to functa: Your data point is function and you should treat it like one. In ICML, 2022a.
E. Dupont, Y. Teh, and A. Doucet. Generative models distributions of functions. In AISTATS. 2022b.
V. Dutordoir, A. Saul, Z. Ghahramani, and E. Simpson. Neural diffusion processes. arXiv, 2022.
Everett. An introduction to latent variable models. Springer, 2013.
M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. Rezende, SM Eslami, and Teh. Neural processes. ICML workshop, 2018.
10

---

Published as a conference paper at ICLR 2023
M. Gemici, D. Rezende, and S. Mohamed. Normalizing flows riemannian manifolds. arXiv, 2016.
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Neur/PS, 2014.
D. Ha, A. Dai, and Q. Le. Hypernetorrss. In ICLR, 2017.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. Fleet. Video diffusion models. In ICLR, 2022.
E. Hoogeboom, V. Satorras, C. Vignac, and M. Welling. Equivariant diffusion for molecule generation in 3d. In ICML, 2022.
A. Jaegle, S. Borgeaud, J. Alayrac, et al. Perceiver io: A general architecture for structured inputs & outputs. ICLR, 2022.
N. Jain, A. Olmo, S. Sengupta, L. Manikonda, and S. Kambhampati. Imperfect imaganation: Implications of gans exacerbating biases on facial data augmentation and snapchat selfie lenses. CORR, 2020.
T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018.
T. Karras, S. Laine, and T. Aila. style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Karras, M. Aittala, Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. In NeurIPS, 2020a.
T. Karras, Laine, M. Aittala, al. Analyzing and improving the image quality of stylegan. In CVPR, 2020b.
H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. Teh. Attentive neural processes. ICLR, 2019.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
D. Kingma and M. Welling. Auto-encoding variational bayes. In NeurIPS, 2014.
N. Kodali, Abernethy, Hays, and Z. Kira. On convergence and stability of gans. arXiv, 2017.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, 1998.
S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
X. Mao, Q. Li, H. Xie, al. Least Squares Generative Adversarial Networks. In ICCV, 2017.
Y. Mirsky and W. Lee. The creation and detection of deepfakes: A survey. CSUR, 2021.
A. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In JCML, 2021.
G. Ostrovski. W. Dabney, and R. Munos. Autoregressive quantile networks for generative modeling. In ICML., 2018.
Park, Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
11

---

Published as a conference paper at ICLR 2023
Joaquin Quinonero- Cainonerr-CCadela and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression. The Journal of Machine Learning Research, 6:1939-1959) 1959, 2005.
S. Rissanen, M. Heinonen, and A. Solin. Generative modelling with inverse heat dissipation. arXiv, 2022.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
N. Rostamzadeh, E. Denton, and L. Petrini. Ethics and creativity in computer vision. arXiv, 2021.
N. Rozen, A. Grover, M. Nickel, and Y. Lipman. Moser flow: Divergence-baat generative modeling manifolds. In NeurIPS, 2021.
M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. In NeurIPS, 2018.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In NeurlPS, 2016.
Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Y. Song and S. Ermon. Improved techniques for training score-based generative models. In NeurIPS, 2020.
Y. Song, J. Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In /CLR, 2021b.
Tinsley, A. Czajka, and P. Flynn. This face does not exist... but it might be yours! identity leakage in generative models. In WACV, 2021.
Michael Tipping. The relevance vector machine. Advances in neural information processing systems, 12, 1999.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pp. 567-574. PMLR, 2009.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34: 24261-24272, 2021.
A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space. In NeurlPS, 2021.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33: 33:19667-1967, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
M. Welling and Y. Teh. Bayesian learning via stochastic gradient langevin dynamics. ICML, 2011.
Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. Advances in neural information processing systems, 13, 2000.
Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar. Neural fields in visual computing and beyond. In Computer Graphics Forum, 2022.
Yu, A. Seff, Y. Zhang, al. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv, 2015.
S. Zhai, W. Talbott, N. Srivastava, C. Huang, H. Goh, R. Zhang, and J. Susskind. An attention free transformer. In ICML, 2022.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
In this supplementary material, we provide an ablation study (Sect. A implementation details
(Sect. B) additional experiments and visualizations (Sect. C limitations and future work (Sect. G
A ABLATION
In this section we ablate the critical design choices and hyperparameters of DPF. We train all model variants on the CelebA-HQ 64 dataset (Karras et al. 2018).
Performance gap between DPF and DDPM (Ho et al. 2020). We aim to provide a fair comparison between DPF and DDPM (Ho et al. 2020). this we use the implementation of the DDPM score network by Ho et al. 2020 and train both DDPM and DPF with the same settings: batch size, learning rate schedule, number of epochs, etc. During inference we use ancestral sampling and perform 1000 denoising steps. In Tab. we observe the gap between DDPM (Ho et al. 2020) (a domain-specifet approach) and DPF to be much smaller than in Tab. 2 approximately equate the number of parameters of the score network in DDPM and score field model (67.6M vs. 62.4M parameters), we find that DPF outperforms DDPM in both FID and recall metrics. We attribute this to the fact that DPF better captures globally consistent samples and long-range interactions. We show qualitative results generated by DDPM (Ho et al. 2020) and DPF Fig.
Furthermore, when
DDPM (Ho et al. 2020) (120M paramters)
DPF (ours, 62.4M paramters)
Figure 9: Visualization of DDPM (Ho et al., 2020) and DPF results CelebA-HQ KKrrras et al. 2018) dataset. All the images show in this figure are generated by a model, we do not directly reproduce images from the CelebA-HQ dataset (Karras al. 2018
Moreover, we are interested in understanding whether the major bottleneck for DPF is on encoding context pairs or decoding query pairs (e.g., evaluating the field at query coordinates). To verify this, we design two ablation experiments. One experiment is to replace the DPF encoder with UNet and the other one to replace the DPF decoder with UNet. We show quantitative results in Tab, 4 We observe that DPF with a UNet decoder achieves the best performance w.r.t. FID, precision and recall scores. These empirical results suggest that further improvements might be obtained by better decoding of query pairs in DPF.
Resolution-free generation. Importantly, DPF differs from classic DDPM (Ho et al. 2020) in that DPF generates fields that can be evaluated at continuous points using query pairs, i.e context pairs used during training differ from query pairs applied at inference. This enables DPF to learn continuous fields rather than naively memorizing discretized data samples. To verify this, we sample DPF at a different resolution than the one it was trained on. Specifically, we train DPF on CelehA-HQ (Krrras et 2018) a resolution of 642 while using 1282 query pairs to generate high resolution images, as shown in Fig. [0 From Fig. 10 we observe that DPF can generate accurate high-resolution image fields given low-resolution context and query pairs during training. For context, compare
*Code available http:///gthcccccooooooilla/aaaan
13

---

Published as a conference paper at ICLR 2023
Org.
Ours
Nearest
Bi-linear
Org.
Ours
Nearest
Bi-linear
Figure 10: Qualitative results of super-resolution image generation. We generate images 1 1282 resolution (row 1) given 64 context points (top left). We show zoom-in results (row 2). We compare our results with nearest neighbor and bi-linear upsampling. All the images in this figure have been generated by model, i..., do not directly reproduce images from the CelebA-HQ dataset (Karras et al. 2018).
CelebA-HQ 64
Params. # Channel FID
Precision
Recall f
DDPM (Ho et al.
2020
120M
128
11.26
0.916
0,384
DDPM (Ho of al. 2020
67.6M
96
24.93
0.927
0,213
DPF (ours)
62.4M
13.21
0.866
0.347
DPF (w/ UNet encoder)
127M
128
20.79
0.916
0.323
DPF (w/ UNet decoder)
62.5M
128
12.27
0.877
0.457
Table 4: DDPM HHo et al. 2020) vs. DPF under the same setting. The DDPM models are implemented using UNet architecture (Ronneberger et al., (2055) with convolutional channel number of 128 (row 7) and 96 (row 2), respectively. We also use UNet architecture to replace either encoder (row 4) or the decoder (row 5) of DPF.
our results with standard nearest neighbor bi-linear upsampling. In Fig. 10 results enjoy comparable or higher quality w.r.t. texture details than the upsampled results.
show that our
DPF hyperparameters. We provide ablation over different DPF hyperparameters. In Tab. 5 we study the effect of varying the number of context pairs used during training. In Tab. we eva nate different PerceiverIO (Jaegle et al., 2022) hyperparameters. For the comparisons in Tab. 5 and Tab. we train models for 180 epochs and compute FID metrics using 1k samples.
As we in Tab. decreasing the number of context pairs leads to drop in FID scores. This is expected since smaller number of context pairs reduces the fidelity at which the field is represented. Also noteworthy, have not optimized score network context pairs, i.e. they are selected only based on the performance the model trained with the full of context pairs.
for reduced number of
The results reported in Tab. 6Jshow the effects reducing the number of parameters in the PerceiverlO architecture, using different knobs. We observe: decreasing the capacity of the latents (either number of latents or their dimensionality) has a higher impact than decreasing the depth of the model (number of blocks in the encoder, decoder, or the number of self-attention layers in encoder blocks).
CelebA-HQ 64
FID
# context pairs 1028 (50%)
103.86
# context pairs 2048 (75%)
91.12
# context pairs 4096 (100%)
74.89
Table Performance of DPF trained with different number of context pairs. Context pairs are randomly samples during training. The FID score is evaluated after 180 epochs for 1/: samples.
14

---

Published as a conference paper at ICLR 2023
CelebA-HQ 64
Params.
FID
latents = 256
62.3M
93.54
# dim latents = 256
22.8M
96.63
# decoder blocks =
59.7M
77.29
# blocks
35,4M
83.87
# S.A. block =
43.5M
75.52
Base
62.4M
74.89
Table 6: Performance of DPF with different PerceiverlO (Jaegle et al. 2022) hyperparameters. The base hyperparameters 12 encoder blocks with 2 self-attention modules and 4 decoder blocks. The FID score is evaluated after 180 epochs for 1k samples. For more details on hyperparameters and implementation details see Sect. B
use in the experiments are: 512 latents with dimensionality of 512,
B IMPLEMENTATION DETAILS
The design space for the implementation of the score field in DPF spans all architectures that can deal with irregularly sampled data, like Transformers (Vaswani et al. 2017) or MLPs (Tolstikhin et al. 2021 (see Sect. D). Unless otherwise noted, we implement DPF using PerceiverIO (Jaegle et al. 2022). an efficient encoder-decode transformer architecture. Our motivation to choose a PerceiverlO that it provides straightforward way of dealing with a large number context and query pairs, and a natural way for encoding the interaction between context and query pairs via attention. Fig. [T] show how context and query pairs used within the PerceiverlO architecture. Specifically, the encoder maps context pairs to latent arrays (e.g. a set of learnable vectors) via a cross-attention layer. Similarly the decoder maps query pairs latent arrays via cross-attention blocks. We refer readers to Jaegle et al. (2022) for additional discussions around the PerceiverIO architecture.
For conditioning the score computation on the time-step & we simply concatenate positional embedding representation of t to the context and query pairs. PerceiverlO settings for all experiments with quantitative evaluation are show in Tab. In practice, the DPF encoder has 12 transformer blocks, each of which contains cross-attention layer followed by 2 self-attention layers. Both cross-attention and self-attention layers have 4 attention heads. The latent array has a dimension of 512 x 512 for 642 resolution images and 1024 1024 for 322 resolution images. We use Fourier feature position encoding to represent coordinates and time steps with a frequency respectively. We use an Adam (Kingma & Ba 2015) optimizer during training. We set the learning rate to le We set the batch size to 16 for all image datasets. We use 8 A100 GPUs for all experiments. Finally, for PerceiverIO we use modification of the public repository
10 and 64,
C ADDITIONAL EXPERIMENTS
Visualization of denoising process. Fig. 12 we visualize the denoising process over time from left to right on CelebA-HQ (Krrras et al. 2018 and CIFAR-10 (Krizhevsky 2009) data.
Visual results on additional image datasets. Fig. 16 we show qualitative results of DPF on
additional image datasets, including AHFQ (Choi et al. 2020), FFHQ (Karras et al.) 2019) and LSUN church Yuu et al., 2015) data at 32 x 32 resolution. Fig. 17 we show videos generated fields over spherical geometry (view with Adobe Acrobat to see animations).
D ARCHITECTURE ABLATION FOR THE SCORE FIELD NETWORK Eg DPF
The formulation of DPF is independent of the specific implementation of the score network cp, with the design space of the score model spanning a large number options. In particular, all architectures
15

---

Published as a conference paper at ICLR 2023
C MM. Yooi
BB
Context pains: C
Latent arry
R3
Encodo
Query pairs: Q
R
R3
wall
Figure 11: Interaction between context and query pairs in the PerceiverIO architecture. Context pairs C, attend to latent array of learnable parameters via cross attention. The latent array then goes through several self attention blocks. Finally, the query pairs Q, cross-attend to the latent array to produce the final noise prediction Eq.
Figure 12: Visualization of denoising process from / = 1000 (noise) to t (images) on CelebA-HQ (Karras et al. 2018) (leff) and CIFAR-10 (Krizhevsky 2009) (right).
that can process irregular data like transformers MLPs are viable candidates. To support this claim, we perform an evaluation on CelebA-Ho -HQ at 32 x 32 resolution where we compare 3 different architectures: PerceiverlO Jaegle et al.. (2022). vanilla Transformer Encoder Vaswani al.) (2017) and an MI.P-mixer (Tolstikhin et al. (021). We report FID and KID metrics using 1000 samples obtained using the inference process described in Alg. To provide fair comparison we roughly equate the number of parameters (to approximately 55M) and settings (e.g. number of blocks, parameters per block, etc.) each model and train them for 500 epochs. To simplify the evaluation use the same number of context and query pairs. Both the Transformer Encoder and the MLP-mixer process context pairs as input using their respective architectures and the resulting latents then concatenated with the corresponding query pairs and fed to linear projection layer for the final prediction.
In Tab. 8 of the score field model. In particular, we observe relatively similar performance across the board for different architectures, from transformer based MLPs. We also observe the same behaviour when looking at the qualitative samples shown in Fig. 14 This supports our claim that the benefits of DPF are due its formulation and the particular incarnation of the score field model. Each particularly architecture has its own benefits, for example, transformer encoders are simple to implement, and PerceiverlO allows for large and variable number of context and query pairs. We believe that combining the strengths of these different architectures is a very promising future direction progress for DPF. Finally, note that these empirical results are not directly comparable to ones reported in other sections of the paper since these models generally have roughly 50% of the parameters of the models used in other sections.
how the formulation of DPF is amenable different architecture implementations
allow for high throughput,
Furthermore, in Fig. 13 we also provide qualitative comparison of MLP-mixers and PerceiverIO results on the Spherical MNIST dataset used in Sect. In which we can see how both a PerceiverIO and MLP-miee mixer successfully capture the distribution of digits the sphere. This supports the claim that the data-agnostic capabilities for generative modeling introduced DPF are not function of the particular implementation of the score field, but due the formulation.
16

---

Published as a conference paper at ICLR 2023
Hypa-pammettr
CaichA-RQ Korras et
2018
CIFAR-10
Krishessky 2009
Shupehot
Chang ot
2015
642
322
32
eval res.
642
322
64
#cansext pais
4096
2048
827G8
4096
2048
32768
Naum sugnal
3
*frey pos. embed
10
10
4freq pos. embed
64
64
64
Vlatenns
1024
512
512
Adim latents
1024
512
1024
12
Aself attends per block
2
2
Aself attention beads
4
batch size
1e
epochs
1300
1300
1000
Table 7: Hyperparameters and settings for DPF on different datasets.
CelebA-HQ 32
FID KID J. x10
PeceiverlO (Joglee et al.
2022
64.8
7.10
Transf. Enc. Vaswani al. 2017)
70.1
10.27
MLP-mixer Tolstikhın
2021
66.8
7.89
Table 8: Quantitative evaluation of image generation on CelebA-HQ (Karras et al. 2018) at 32x32
resolution for different implementation of the score field 60-
E QUALITATIVE COMPARISON WITH FUNCTA (DUPONT AL.. 2022A AND
GASP DUPONT ET AL.. 2022B ON S
We provide a qualitative comparison with Functa (Dupont et al. 2022a) and GASP (Dupont al. 2022b) on fields defined over the sphere. In order to do this, we used the ERAS temperature dataset released with GASP httpt::t////poo //tp:::///itti huub com/ Emi EmilienDupont. Soural-functions iowputttioodddett..tiiiioooo which available at a 46 x 90 spherical resolution.
important note that this dataset is different from the FRA5 dataset used train Funeta (Dupont et al. 2022a), which is sampled at a higher resolution (181 x 360) and is not publicly available
their repository https::///... / thub. com. dwwpmindnnnne nd/ ftacta In Fig. 15 we show samples generated by Functa (Dupont et al. 2022a), GASP (Dupont et al., 2022b) and DPF. We observe that samples generated by DPF are visually indistinguishable from those Funeta GASP. DPF captures the same patterns as Functa and GASP, where the poles having cooler temperatures (e.g.. dark purple tones) and the equator and tropics having considerably high temperatures shown by the lighter yellow color.
F SAMPLING CONTEXT AND QUERY PAIRS
A natural question that falls from the formulation of DPF via context and query pairs is the following: there an intuition for which context and query pairs to sample from the field?
For query pairs the answer is simple: one should sample many pairs required in the problem definition. For example, the task is that of sampling field representation of an image at 32 32 resolution, one should sample 32 x 32 1024 query pairs. In this sense, the number and distribution of query pairs is entirely problem dependent and flexible and DPF allows for this flexibility.
Selecting which context pairs sample is far more intricate problem. To discuss it we exploit the intimate relationship between the formulation in DPF and Gaussian Processes (GPs). In particular, look at the problem of choosing context pairs equivalent the problem of selecting inducing variables in Sparse GPs QQunonero--Came Candela & Rasmussen (2005). which is a combinatorial problem for which approximations computed. While there exists different approximations the
17

---

Published as a conference paper at ICLR 2023
(a) PerceiverlO (Jaegle et al. 2022).
(b) Transformer Encoder Vaswani et al. 2017)
(c) MLP-mixer (Tolstikhin al. 2021)
Figure 13: Qualitative comparison of different architectures to implement the score field model
(a) PerceiverlO Jaegle et al. 2022
(b) MLP- P-mixer Tolstikhin et al.
2021
Figure 14:
Qualitative comparison of different implementations of the score field for functions
defined on the sphere. (a) PerceiverlO (Jaegle et al. 2022). (b) MLP-mixer (Tostikhinn et al. 2021
problem of selecting inducing variables we highlight 3 classical methods which are representative broader classes of approaches. (Titsias) 2009) a variational approximation that jointly infers the inducing inputs and the kernel hyperparameters by maximizing a lower bound of the true log marginal likelihood is introduced. A key property of this formulation is that the inducing inputs are defined to be variational parameters. On different approach Williams & Seeger (2000) introduce Nystrom approximation of the eigendecomposition
the Gram matrix across inducing variables (i.e.context pairs). Finally, the work of Tipping (1999) introduces the Relevance Vector Machine (RVM) which can be thought of as a Sparse version of the Support Vector Machine embedded within a Bayesian learning framework. Note that since the RVM is finite linear model with Gaussian priors the weights, it can be seen as a Gaussian Process.
An interesting direction of future work is to draw inspiration from Sparse GP approximations to formulate a learning objective that jointly learns the generative model in DPF as well as the distribution of context pairs in hierarchical manner.
G LIMITATIONS AND FUTURE WORK
While DPF makes progress toward learning distributions over fields, there exist some limitations and opportunities for future work. The first limitation is that vanilla implementation of the score network as a transformer becomes computationally prohibitive even at low resolutions, due to the quadratic cost of computing attention over context and query pairs. To alleviate this issue, we leverage the PerceiverIO (Jaegle et al., 2022) architecture that scales linearly with the number of query and context pairs. To further address this, one possible direction of future work is to explore other efficient transformer architectures (Zhai et al.) (022). In addition, similar to DDPM (Ho et al. 2020). DPF iterates over all time steps during sampling produce field during inference, which is slower
18

---

Published as a conference paper at ICLR 2023
(a) Functa (Dupont et al. 2022a (181 x 360 resolution)
(b) GASP (Dupont et
2022b (46 x 90 resolution)
(c) DPF (ours) (46 x 90 resolution)
Figure 15: Qualitative comparison of different implementations of the score field for fields defined
on S (a) Functa (upont et al. 2022a). (b) GASP (Dupont et al. (0022b). (c) DPF (ours).
AFHQ (Choi et al. 2020)
FFHQ KKrras et al. 2019)
LSSUN Church (Yu al. 2015)
Figure 16: Additional results on image datasets. We show generated results using DPF on AHFQ (Choi et al. 2020). FFHQ (Karras et al., 2019) and LSUN church YYu et al. 2015) data at 32 32 resolution. Face images are generated from DPF, i.e., we do not directly reproduce face images from FFHQ (Karras et al. 2019).
compared to GANs. Existing work accelerates sampling (Song et al. 2021a) trading-off sample quality and diversity. We highlight that improved inference approaches like (Song et al.) (021a) trivially integrated with DPF.
19

---

Published as a conference paper at ICLR 2023
Figure 17: Videos of generated spherical results. View with Adobe Acrobat to see animations.
20


Reviews:
[{"review_id": "qqF-SFYUCkO", "reviewer": "Reviewer_joBr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an extension of Diffusion Probabilistic Models using Fields to model data.\nThe basic idea is to model data as functions (fields) e.g. images are functions from $R^2$ to $R^3$, a map between pixel coordinates and color pixel value.\nThis approach is motivated by the need to unify the generative model architecture to cover different modality, thus avoiding the need to construct specific architecture for score functions $\\epsilon_\\theta$ which are the denoisers used in the generation process.\nFields are parameterized using pairs of coordinates in source and target domains and evaluated in a similar manner using pairs of query pairs of coordinates.\n\nResults are compelling considering that the Field Model is domain agnostic.\n\nIn the end the method is using a Transformer (Perceiver) to remap coordinates (implementing the field) (Appendix B)\n", "summary_of_the_review": "Overall the paper is interesting and results are compelling. It has to be noted that the approach claims to be general with respect to the data in input but I feel that the main aspect is that a very flexible and powerful architecture (PerceiverIO) is used as a score function thus enabling this feature. The whole modelling of data as fields seems less relevant than this implementation detail.", "strength_and_weaknesses": "Strenghts\n\n- The method is elegant in principle and domain agnostic yet obtains good performance\n\nWeaknesses \n\n- It is not clear how the gradient at step 7 is computed. $epsilon_Q$ is not defined.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ozLCPWv97k", "reviewer": "Reviewer_E4qq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a diffusion model for an explicit neural fields. They extend the regular diffusion model training and sampling, but instead of assuming an explicit domain for the signal, like 2D grid for images, their model is general enough to work for all kinds of domain through neural field representation. That is possible because the diffusion model is learnt over those fields, $f : \\mathcal{M} \\to \\mathcal{Y}$. Another key feature of the paper is that same score model architecture applies to all kinds of data like images, 3D geometric data or data on $\\mathbb{S}^2$. This is achieved by using transformer as the score function. ", "summary_of_the_review": "The paper is easy to follow and extend the diffusion model to the field representation of data. The use of transformer architecture allows the  common architecture for data of different types. However, the contribution seems to be insufficient. So, I am slightly on the negative side.", "strength_and_weaknesses": "The field of representing data as neural field has recently seen popularity. This work is new iin diffusion models in the sense that it is probably the first to work with field representation of data, which is a positive aspect of the paper. Also, this paper uses the transformer architecture as a score function, which allows it to propose a common architecture for all kinds of data. This is a good contribution and might encourage more use of transformer architecture in the future. \n\nHowever, the technical contribution of the paper seems to be limited. Mainly using coordinate signal pair and having a transformer architecture that can process such a paired information appropriately is the key idea. While training, the authors use context and query pairs. I believe a bit of more analysis, perhaps theoretical, on how to choose these context and query pairs in better way could strengthen the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jTuTvV4Boe9", "reviewer": "Reviewer_NtLL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper adapts a generative diffusion approach to neural fields, allowing simple adaptation to different data domains. A diffusion process is modeled on both context and query pairs and training is done with a score network that incorporates both sets of pairs. This framework is one of a few domain-agnostic approaches, and is the only diffusion-based method which does not require a latent vector to specify the field. \n\nA PerceiverIO architecture is used for the score network, and experiments are done with image data, volumetric data, and spherical data. The method performs well when compared to other domain-agnostic approaches, on both quantitative and qualitative comparisons.", "summary_of_the_review": "The work provides an interesting diffusion-based approach for generating neural fields. The clarity could use some work, but the empirical results are compelling, and I feel it is worth publication.", "strength_and_weaknesses": "Strengths:\n- The approach is the first diffusion-based method that does not use latent vectors to parameterize fields, and hence does not require a step to train for the distributions over latent vectors. \n- The method seems to perform fairly well compared to other domain-agnostic models.\n\nWeaknesses:\n- The clarity of their model explanation with context and query pairs could be improved, I feel.\n- The PerceiverIO framework could also use some explanation, even if brief, given that it's a key part of their method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H3XI3a5Fidp", "reviewer": "Reviewer_6arD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method for learning diffusion models over neural fields. More specifically, the authors perform diffuson directly on coordinate (such as pixel locations) and feature (such as RGB values) pairs, allowing the model to handle a wide variety of data that can be expressed as a neural field. The authors evaluate their model on small image datasets, 3D shapes and spherical images, obtaining favorable results compared to recent baselines.", "summary_of_the_review": "Overall I think this is a fairly strong paper which will be of interest to the ICLR community. The model is timely and the experimental results are quite convincing. While there are some issues around clarity and a few other minor weaknesses, I still believe the strengths outweigh the weaknesses and so I recommend a weak accept.", "strength_and_weaknesses": "\n**Strengths**\n\n- The paper tackles an interesting and timely problem. With the recent surge of interest both in neural fields and diffusion models, I believe this paper will be of interest to the ICLR community\n- The method is fairly novel. Previous works on learning distributions of neural fields have used a two stage approach (embedding neural fields into a latent vector and then learning a distribution over these) which has limitations discussed by the authors. Learning directly on coordinate and feature pairs has been considered in the context of GANs before with GASP, but the results in this paper are considerably stronger\n- The experimental results are generally thorough and interesting. The model performs favorably compared to several recent baselines\n- There is a good discussion of related work and the paper is quite well situated in the literature\n- I like the ablation comparing each element of DDPM and the proposed model in the appendix. It's interesting to see what \"makes the model work\"\n- It is nice to see how seamlessly the model works on spherical data\n\n**Weaknesses**\n\n- While most of the model is fairly well explained, it was not very clear to me exactly how context and query pairs were treated differently until reading the appendix (e.g. in Algorithm 1 and Figure 3, context and query points are treated exactly the same way). I think the paper would benefit from a clearer explanation of this as it is quite crucial to how the model works. It could for example be useful to have a diagram of the score network either in the main text or appendix to clarify this as it's currently not very clear\n- There are no qualitative or quantitative comparisons for the spherical data experiments. As both functa and GASP consider generative modeling on spherical data it would be useful to have these as baselines to compare to\n- The model appears to be computationally expensive which I believe is quite important and deserves to be discussed in more detail in the main text. The two stage process used by GEM and functa for example makes training a generative model fairly cheap, as the model is only fed a relatively small latent vector. Further, the authors only train on fairly small images (up to 64x64) and still require 8 A100 GPUs to train, which is quite heavy. While it is okay for the model to have this weakness, it should be more openly acknowledged in the main text\n- The results do not match SOTA compared to specialized models on images for example. However, as learning generative models on neural fields is a fairly new problem, I don't believe it is very crucial for these models to outperform highly optimized convolutional models yet. This is therefore only a minor weakness in my opinion\n- The proposed model is quite closely related to neural processes (also modeling distributions of fields and using a context/query pair setup). I think the paper would benefit from a more thorough discussion of neural processes in the related work. While I realise there are space constraints, I don't think the extended discussion of GANs is necessary, so this could be replaced with a discussion of neural processes\n- There is some confusion about the naming of baselines. I believe a better naming convention would be functa for Dupont 2022a and GASP for Dupont 2022b. I'm not sure exactly why FDN was chosen as the name for functa. Further, it seems that for some reason in the GEM paper, the authors refer to GASP as FDN, whereas in this paper the authors refer to functa as FDN, leading to even more confusion (I am not sure where FDN comes from as it does not seem to be mentioned in either of Dupont 2022a/b). In addition, in the 3D experiments, the baseline is incorrectly referred to as FDN (Dupont 2022a), when it is actually GASP (Dupont 2022b). In short, I think it would be better to call Dupont 2022a functa (or something similar and more intuitive). In addition, the name of the baseline in the 3D geometry experiments is incorrect and should be changed to GASP (as it's referred to in the rest of the paper).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster