Neural Diffusion Processes
Authors: Vincent Dutordoir
Alan Saul
Zoubin Ghahramani
Fergus Simpson
Venue: Submitted to ICLR 2023 (2023)
Keywords: diffusion models
gaussian processes
neural processes
stochastic processes

Abstract: Gaussian processes provide an elegant framework for specifying prior and posterior distributions over functions. They are, however, also computationally expensive, and limited by the expressivity of their covariance function. We propose Neural Diffusion Processes (NDPs), a novel approach based upon diffusion models, that learns to sample from distributions over functions. Using a novel attention block we are able to incorporate properties of stochastic processes, such as exchangeability, directly into the NDP's architecture. We empirically show that NDPs are able to capture functional distributions that are close to the true Bayesian posterior. This enables a variety of downstream tasks, including hyperparameter marginalisation, non-Gaussian posteriors and global optimisation.

Under review as a conference paper at ICLR 2023
NEURAL DIFFUSION PROCESSES
Anonymous authors
Paper under double-blind review
ABSTRACT
Gaussian processes provide an elegant framework for specifying prior and posterior distributions over functions. They are, however, also computationally expensive, and limited by the expressivity of their covariance function. We propose Neural Diffusion Processes (NDPs), novel approach based upon diffusion models, that learns to sample from distributions over functions. Using a novel attention block we are able to incorporate properties of stochastic processes, such as exchangeability, directly into the NDP's architecture. We empirically show that NDPs are able to capture functional distributions that are close the true Bayesian posterior. This enables a variety of downstream tasks, including hyperparameter marginalisation, non-Gaussian posteriors and global optimisation.
INTRODUCTION
Gaussian processes (GPs) offer a powerful framework for defining distributions over functions [26]. is an appealing framework because Bayes rule allows one to reason consistently about the predictive distribution, allowing the model to be data efficient. However, for many problems GPs are not an appropriate prior. Consider, for example, function that has discontinuity some unknown location. This cannot be expressed in terms of a GP, because it is impossible to express such behaviour by the first two moments of multivariate normal distribution [23].
One popular approach to these problems is to abandon GPs, in favour of Neural network (NN) based generative models. Successful methods include the meta-leaming approaches of Neural Processes (NPs) [8; 12; 2; 21], and VAE-based models [22; 6]. By leveraging a large number of small datasets during training, they able transfer knowledge across datasets at prediction time. Using NNs is appealing since most of the computational effort is expended during the training process, while the task of prediction usually becomes more straightforward. A further major advantage of a NN-based approach is that they are not restricted by the Gaussian assumption.
We seek improve upon these methods by extending an existing state-of-the-arr NN-based generative model. In terms of sample quality, the so-called probabilistic denoising diffusion model [31; 32: 10] has recently been shown to outperform existing methods on tasks such as image [24; 25], molecular structure [40; 11 11], point cloud [20] and audio signal [14] generation. However, the Bayesian inference of functions poses a fundamentally different challenge, one which has not been tackled previously by diffusion models.
Contributions We propose a novel model, the Neural Diffusion Process (NDP), which extends the use case of diffusion models to Stochastic Processes (SPs) and is able to describe a rich distribution over functions. NDPs generalise diffusion models to dimestoalimeniionall function spaces by allowing the indexing of random variables onto which the model diffuses. We take particular care to enforce known symmetries and properties of SPs, including exchangeability, and marginal consistency into the model, facilitating the training process. These properties are enforced with the help of novel attention block, namely the bi-dimensional attention block, which guarantees equivariance over the ordering of (I) the input dimensionality and (2) the sequence (i.e., datapoints). From the experiments we draw the following two conclusions: firstly, NDPs are a clear improvement over existing NN-based generative models for functions such as Neural Processes (NPs). Secondly, NDPs an attractive alternative to GPs for specifying appropriate (i.e., non con-Gaussian) priors over functions. Finally, we present novel global optimisation method using NDPs.

---

Under review as a conference paper at ICLR 2023
(a) GP Regression
(b) Attentive Latent NP
(c) Neural Diffusion Process (ours)
Figure Conditional samples: The blue curves are posterior samples conditioned on the context
dataset (black dots) from different probabilistic models.
2
BACKGROUND
The aim of this section is to provide overview of the key concepts used throughout the manuscript.
2.1 GAUSSIAN PROCESSES
Gaussian Process (GP) f :R R a stochastic process such that, for any finite collection of points y In R the random vector (J)....;J() with fi f(xx), follows a multivariate normal distribution [26]. In the case of regression, GPs offer an analytically tractable Bayesian posterior which provides accurate estimations of uncertainty -as shown in Fig. where we observe that the predictive variance (dashed black line) elegantly shrinks in the presence of data (black dots).
GPs satisfy the Kolmogorov Extension Theorem (KET) which states that all finite-dimensional marginal distributions are consistent with each other under permutation (exchangeability) and marginalisation. Let æ be the permutation of {1....,n} n), then the following holds for the GP's joint:
fa(m) and p(f)) - f fo) ff) f.)
p(f fn)
dfn- (1)
Despite their favourable properties, GPs are plagued by couple of limitations. Firstly, encoding prior assumptions through analytical covariance functions can be extremely difficult, especially for higher dimensions [39; 29; 17J. Secondly, by definition, GPs assume multivariate Gaussian distribution for each finite collect of predictions limiting the set of functions it can model [23; 36; 38; 13; 5; 27]. We will revisit these limitations in the context of our experiments.
2.2 NEURAL PROCESSES AND THE -EAA-ING OF FUNCTIONS
Neural Process (NP) models [7; 8] have recently been introduced as an alternative to GPs addressing the aforementioned problems. NPs utilise deep neural networks to define rich probability distribution over functions. During training, NPs leverage a large number of small datasets, so that knowledge can be transferred across datasets during inference.
Despite being a promising research direction, NPs are severely limited by the fact that they do not produce consistent samples out-of-the-hox. Broadly speaking, NPs have dealt with consistency in two ways: (1) by introducing an additional latent variable per function draw [8; 12), (2) by only modelling the marginals [7]. The former leads to consistent samples, but the likelihood for these models is not analytically tractable which requires crude approximations and ultimately limits their performance [21]. We observe this behaviour in Fig. lb. The latter does allow for sampling coherent functions at all, as no covariance information is available and all function values are modelled independently. We briefly summarise the family of NP models in Appendix D.4.
2.3 DIFFUSION MODELS
Our method relies on denoising diffusion probabilistic models (DPMs) [31], which we briefly summarise here. Diffusion models depend on two procedures: forward and reverse process, as illustrated in Fig. 2. The forward process consists of Markov chain, which incrementally adds random noise to the data. The reverse process is tasked with inverting this chain in order to construct desired data samples from random noise alone.
2

---

Under review as a conference paper at ICLR 2023
d. 1,1)
90 (:TT)
forward process
40
To, yo
11 3/1
ET. YT N(O,1 1)
Teverse process
po(- 1
pol- 1 1,7)
Figure 2: Illustration of the forward and reverse process of NDPs. (Left-to-right) During the forward process, function values (blue dots) from function draw (blue line) are perturbed by increasing levels of noise until the points given by white noise. (Right-to-left) The reverse process consists of gradually removing noise until we arrive at a data point.
The forward process starts from the data distribution ((80) and iteratively corrupts samples by adding small amounts of noise, which leads to a fixed Markov chain ((80:T) - ((8)) II q(0; 81-1) for a total of T steps. The corrupting distribution Gaussian g(*; 8 1) = (st; v 3,8 3.I). The magnitude of the noise in each step controlled by pre-specified variance schedule { (0, 1)} Note that there is no learning involved in the forward process, it simply creates a sequence of random variables (st 1 which progressively look more like white noise ((s)) = (O,
While the forward process is designed to be Markovian, the true reverse probability (($:-1 $;) requires the entire sequence. Therefore, the reverse process learns approximate these conditional probabilities in order to carry out the reverse diffusion process. The approximation relies on the key observation that the reverse conditional probability is tractable when conditioned the initial
state 86: q(8-1 8o, 8:) (s; (50 8t), 3,I). In this work, we follow Ho et al. [10] which approximate jı by estimating the initial state 80 from 8, and t. They propose the following paramterisation t) yau (s. VI-a. co(s;, t)), a = Bt. āg = IT aj for μ. and derive 3g in closed-form (given in Appendix A). We refer to as the noise model and implemented it as neural network. Its architecture is detailed in Sec.
&
The parameters of the noise model are optimised by minimising the objective Et....= co(s:, t)], where the expectation is taken over time corrupted sample trained, generating samples from ((6) ) done by simulating the reverse process. That is, starting from 8T ~ (00,T] I) and iteratively denoising samples using 8 ~ (5 (1 f), 3.).
(1( 2......... T)). data points so ~ q(sp). and 50, + vI - ü₂e created from noise NO, I). Once the network is
3 DIFFUSION PROBABILISTIC MODELS FOR STOCHASTIC PROCESSES
In this section describe Neural Diffusion Processes (NDPs): diffusion model designed for stochastic processes, We focus our attention on the difference between NDPs and traditional diffusion models used for images [24; 25], audio [14] or molecules [40; 11]. We defer the exposition of the NDP's noise model architecture to Sec. 4, where we also prove that NDPs satisfy the exchangeability and consistency conditions of KET.
3.1 FORWARD AND REVERSE PROCESS
Data distribution f: IRD - R, is fundamentally different random variable than, for example, an image. In an image, the pixel values are arranged on a predefined grid (height x width) and have an implicit ordering (e.g., left-to-right, bop-to-bottom). Function values, in contrast, have ordering and do not live predefined grid. Therefore, for NDPs to useful, it is important that one can evaluate the generated samples everywhere in their input domain, i.e. Vr RR..
NDPs, we are interested sampling over the space functions. A function,
3

---

Under review as a conference paper at ICLR 2023
To accommodate for this, define the following data generating process for training data: (1) sample the imput dimensionality
2... Dmax 1) such that the domain is R.," (2) sample the number of point in the dataset N ~ ( ....N max }), (3) sample the N points i..d. from uniform distribution in R and collect them in zo E IR N x (we make the assumption that function inputs originate from uniform measure a GP (i.e.. f ~ GP(O, ky) with ky the covariance function) or any other function class on which one wants to train (e.g., step functions in Sec. 5.1). We obtain two correlated random variables æo and Yo that are, respectively, the function's 's inputs and outputs.
the space), (4) finally, set Y/n ((a), where can be
Diffusion In the forward process, NDPs gradually add noise using factorised distribution
q(
Fit Vr
F2 y
1- VI- 8( x2-1).
As shown in Fig. 2, this diffusion process corresponds to gradually adding Gaussian noise to the function values y, while keeping the input locations æ; fixed across time. At = T the function values y, should be indistinguishable to samples from (O, I), while æт = æT-1 = In Sec. 5.3, we discuss generalisation of this scheme where we perturb both the function inputs and outputs to obtain a diffusion model that can sample the joint p(æ0, yo). For now we are only interested in the conditional p(yo zo) as this is the key quantity of interest in regression, which is why we keep the inputs fixed across time.
Zo-
Backward kernel We parameterise the backward Markov kernel using a neural network that learns de-noise the corrupted function values yr- In contrast, the input locations a, have not been corrupted in the forward process which makes reversing their chain trivial. This leads to parameterised backward kernel po of the form
po
x y
x2 y/ =
No yr No
(ve Vat
3e vI
co( ye: 0)) 1)),8)))) 6(x; Bz xT-1))
(3)
at
In NDPs, the neural network model of the noise Eq : RND x N
+ RN has as inputs the function inputs Hy, the corrupted function values yı- and time t. The network is tasked with predicting the noise that was added to Yo to obtain y.- The design of the network is specific to the task of modelling SP's and such differs from current approaches. In the next section we discuss its particular architecture, but for now we want to stress that is critical for the noise model to have access to the input locations v, make such predictions.
Objective Following Ho al. [10], but substituting the NDP's forward and backward transition densities (Eqs. (2) and (3), resp.) leads to the objective (derived in full detail in Appendix A):
La Et,wo N |||e EØ (xo, Varyo vI-ate,t)1?)
(4)
where we made use of the fact that equals æn for all timesteps I, and ye Voryo + v one
3.2
PRIOR AND CONDITIONAL. SAMPLING
Once the network trained, one can obtain prior function draws from the NDP a specific set of input locations 2g by simulating the reverse process. That is, starting from yy ~ N I) and iteratively denoising samples using Po (Eq. (3)) for time = =T,..., 1. This procedure leads the samples in the left panes of Figs. 4 and 5.
Prior
Conditional NDPs are also able to draw conditional samples from the posterior P(Vo: 2o, D) where = (x MDD yo E M) is the context dataset. Following recent advances in image in-painting [19], adjust the reverse process to account for the information in the context dataset D as we want the conditional samples to be correlated and consistent with the context dataset. The conditional sampling process works as follows: We start by augmenting the intermediate state to contain the perturbed context dataset yf and the de-noised state y.- This results in the augmented intermediate state Ú. (yf yr] (similarly, $o [ar an))). The part of the augmented state ú, that
4

---

Under review as a conference paper at ICLR 2023
Time embookingg
Rlemet-oiio arkiition
Brondcastins
R. batch R: nem. data.
Dr dimensoon
I: latent dimm
Residual comnection
Dense
Multi- Atteatton
D,P5
ReLU
[0,5,13
[3,
Multi-Hoad
Douse
Detoee
Attention
[h,3]
[2,126]
[b.0]
Figure 3: Architecture of the noise prediction model, utilised at each step within the Neural Diffusion Process. The greyed box represents the bi-dimensional attention block, as discussed Section 4.2.
corresponds to the context points yi is sampled using the forward process (left-hand-side), while ye sampled using the backward kernel (right-hand-iid)):
Be vi ----Vvrsi (1 (1- (1-a)))))) I). and y:-1 N (1 V1-
co Tu. Ve t)), (5)
Va
Simulating this scheme from f = T. 1 ensures that for each backward step (rhs of Eq. (5)) leverage the context dataset as well as the randomness the initial draw yr (0, We show conditional samples using this scheme in Figs. Ic, and 5, and list pseudocode in Appendix B.2.
4 ARCHITECTURE OF THE NOISE MODEL
NDPs implement the noise model as neural network. In this section, we review its architecture and key components. principle, any neural network could used. However, we wish for NDPs to mimic SPs, the noise model must learn to generate prior distribution over functions. We expect such a prior to possess several key symmetries and properties, which will heavily influence our choice of architecture. We refer to Fig. 3 for overview.
4.1 INPUT SIZE AGNOSTICITY
Before addressing the neural network invariances and equivariances, we focus our attention to key property of the network: the NDP network agnostic to dataset size and dimension D. That is, the weights of the network do not depend on the size of the inputs (i.e. N nor D). This has as important practical consequences that not required train different NDPs for datasets with different size or dimensionality. This makes it possible to train only single model that handles downstream tasks with different N or D values. To achieve this functionality, NDPs start by reshaping the inputs (xt, R.N x y: RN) ) to Dx2 by replicating the y, outputs D times before concatenating them with Ft. Pseudocode for this operation given Appendix E.
4.2 BI-DIMENSIONAL ATTENTION BLOCK
One of the most fundamental properties of SPs, as specified by the KET, is an equivariance to the ordering of inputs. As such, shuffling the order of data points in the context dataset 'D or the order which we make predictions should not affect the probability of the data (i.e., the data is exchangeable). Secondly, we also expect an invariance in the ordering of the input dimensions. Consider, for example, dataset consisting of two features, the weight and height of people. We would not expect the posterior function to be different if we would swap the order of the columns in the training data. This is an important invariance encoded many GP kernels (c.g., Matérn, RBF) but often overlooked in the neural net literature [18].
We accommodate for both desiderata using our newly proposed bi-dimensional attention block. We denote this block by A, R x Dx its core, the block consists two multi-head self-attention (MHSA) layers [35]. The MHSA layers different axes of the input: attends to the input dimension axis d, while the other attends
+ R N Dx H acts the preprocessed inputs (I,, y,). At
5

---

Under review as a conference paper at ICLR 2023
across the dataset sequence axis n. The outputs of the two are subsequently summed and passed through a non-lineane con-lincartty This process is repeated multiple times by feeding the output back into the next bi-dimensional attention block using residual connections. The 4h block is defined as
- A A
1 + ReLU (MHSAA(s)))) + MHSA,, (s, for (1,.... L),, (6)
with se = st (i.e. the output of the preprocessing component -see diagram) and A O. Kossen et al. [15] introduced a similar block which alternates between attention across datapoints and attributes, whereas our bi bi-dimensional attention block acts on both simultaneously.
As the MHSA layers act on different axes, the component A, is simultaneously equivariant to the order of the dimensions and dataset sequence (i.e., shuffling the rows and/or columns of the input will shuffle the output in the same way).
Let 1N and 1p be the set of all permutations of indices fI....,NN N} and {L..... D}. respectively. Let a E R x and (Tn 8) E N denote a tensor where the ordering of indices the FIRST dimension are given by nn E Hl. Similarly, let (na o 8) denote a tensor where the ordering of indices in the SECOND dimension are given by a E II. Then, VTn, Rd E Hy X NI:
Proposition 1.
WA o Tn As(s) A2 (Td Tn 088.
Proof. Follows from the invariance of MHSA layers, and the commutativity of operators ny and Ti detailed in Appendix C.2.
The noise model's Eo outputs are obtained by element-wise adding the different bi-dimensional attention layers (see purple '+' in Fig. 3). This is followed by (D). These operations introduce an invariance over D, while crucially preserving the equivariance over the dataset ordering N.
the input dimension axis
Proposition Let ny and na be defined as in Proposition 1, then Eg satisfies
nn en (xr v. t) Co na De, Tr y..t).
Proof. The claim follows from Proposition and Zaheer et al. [41] as shown in Appendix C.3,
Crucially, by directly encoding these properties into the noise model, the NDP produces set of random variables .y....., v. at each time step t that are exchangeable.
Consistency The second KET condition, consistency, is achieved when marginalising over the conditional distribution of a random variable yo, we recover its original distribution p(y²) r ))(()"((())))))))))))>))))) dyy When comes consistency, NDPs are unlike NPs, in which the predictive distribution is created directly by the neural network. In NDPs the conditioning step is not performed by the neural network. Instead, conditioning takes place within the inherently probabilistic frame- work of the diffusion model itself. NDPs therefore naturally satisfy consistency, irrespective of the architecture of the NN. We include formal proof in Appendix C.4, which relies on the fact that NDPs gradually transform samples from consistent distribution. N/ I) into the target distribution.
5 EXPERIMENTS
In the experiments we want to answer the following two questions: (1) What do NDPs provide on top of GPs? (2) How do NDPs compare, in terms of performance and applicability, to NPs? We address these questions in Sec. 5.1 and Sec. 5.2. Finally, in Sec. 5.3, we highlight novel approach for global optimisation of black-box functions. All experiments (except Sec. 5.3) share the same model architecture illustrated in Fig. 3 with five bi-dimensional attention blocks. Full model configurations and training times given in Appendix D.1.

---

Under review as a conference paper at ICLR 2023
-0.5
10
0.5
-0.5
0.3
-0.5 0.5 -0.3 0.5
longthscale (E)
lengthscale (E)
lengthscale
(E)
lengthscale (E)
Figure 4: Kernel lengthscale marginalisation: Samples from the NDP, conditioned on an increasing number of data points (black dots), are illustrated in the top row. A sample is coloured according to its most likely lengthscale. As the NDP has no notion of a lengthscale, we infer the most likely lengthscale by retrospectively fitting a set of GPs and matching the lengthscale of the GP corresponding to the highest marginal likelihood to the sample. The bottom row shows a histogram of likely lengthscales from the produced samples. As more data points are provided, the distribution of likely lengthscales converges from the prior over lengthscales (red dashed line) to the lengthscale that was used to produce the data (orange dashed line).
5.1 NEURAL DIFFUSION PROCESSES VS. GAUSSIAN PROCESSES
GP regression models have three major limitations: cubic scaling of complexity ((N5) with dataset size N, difficultly of choosing the right analytical prior covariance by hand, and the constraints associated with Gaussianity. NDPs provide remedy to all three of these issues. Firstly, performing inference on new dataset for a NDP as costly evaluating the noise model multiple times during the backward process. The complexity of this operation is O(TN2), where T is the number of timesteps. However, note that there is no training required to perform Bayesian inference on new dataset after the initial training. This contrasts with the O(N3) for training a GP per dataset. We address the remaining two limitations in the experiments below.
Automatic marginalisation of hyperparameters Conventionally, GP models optimise a point estimate of the hyperparameters. However, it is well known that marginalising over the hyperpa- rameters can lead to significant performance improvements, albeit at an extra computational cost [16;30] A key capability of the NDP is its ability to produce realistic conditional samples across the full gamut of data on which it was trained. It can therefore in effect marginalise over the different hyperparameters it was exposed to, even different kernels. Figure 4 demonstrates how the NDP produces samples from a range of different lengthscales. Indeed, this model was trained on a richer dataset consisting of samples from GP with Matérn- kernel, with variety of lengthscales sampled from ((og((005)) VO.5.. In the lower pane, we can see that the NDP was able to capture this distribution, and that it narrows down the true lengthscale as more data is observed.
Non-Gaussian Posteriors Each finite set of predictions of a GP follows a multivariate normal distribution. While this allows for handy analytic manipulations, is also a restrictive assumption. For example, it is impossible for a GP to represent the covariance arising from 1D step functions when the step occurs at random location its domain [23]. For Fig. we trained a NDP on exactly this prior where the domain is set to [-1, 1J. We observe that the NDP is able to correctly sample from the prior in (a), as well as from the conditional when we introduce datapoints in (b). In (c) show the NDP's posterior of a marginal, which correctly shows the bimodal behaviour. The key takeaway from this experiment is that the NDP can infer data-drived covariance, which need not be Gaussian and thus impossible for GPs model.
7

---

Under review as a conference paper at ICLR 2023
0.0
0.8
0.4
0.4
02
0.2
0.00
(a) Prior
(b) Conditional
(c) (C) == 0.0)
Figure 5: Representing a step function using NDP.
Table 1: Average Negative Log-Likelihood (NLL) across different synthetic datasets and input
dimensions (D.). The performance is measured on held-out functions that are conditioned on random number of points between 1 and (30 Dz). Smaller is better.
Squared Exponential D) = 2
Matérn-5/2 D ==2
D
D 3
D
D 3
NDP (ours)
00.8
1.01
1.20#t0.01
0,13
1.15=0.02
1.19:0..11
ANP [12]
0.29
1.05
1.25=:0.03
0.60
1.14=0.05
1.29:t0.02
NP 181
0.67
1.23:1000
1.35
0.8410.04
1.20:to.03
1.365t0.01
CNP [7]
0.77::0.09
1.26120.05
1.35:t0.02
0.91
1.30:t0.04
1.37:0.002
trivial
1.41++0.03
1.42+o0.02
1.45-+0.02
1.43+0.02
1.43+o.02
1.45++0.02
5.2
COMPARISON TO NEURAL PROCESSES
Neural Processes (NPs) [7; are family of models which closely related to NDPs. NPs use neural network to meta-learn a map from context datasets to predictive distribution over a target set.
Empirical Evaluation In Table 1, empirically evaluate NDPs and NP models synthetic datasets created using Squared Exponential and Matérn-5/2 kernel in different dimensions D. We compare NDP Attentive NPs [12], Latent NP [8], Conditional NPs [7]', and a trivial model which uses the empirical mean and variance as predictive distribution.
NDPs are superior NPs in multiple areas. Firstly, from Table notice that NDPs outperform the NP family in terms of predictive performance. Secondly, NDPs produce consistent samples by design. For conditional NPs this not the case they are only able to predict the marginals. (Attentive) Latent NPs solve this problem by using a latent variable but due to crude approximations in the approximate inference scheme, they typically suffer from underfitting to the context set, leading to overestimated uncertainties, and samples that fail to pass through the context points [21; 22]. We observe this behaviour our experiment, as shown Figure 10 of supplementary.
Bayesian Optimisation In this experiment highlight that NDPs can be used in problems with D 2 3. We tackle 3D and 6D minimisation problems using discrete Thompson sampling BO [28; 4; 33]. In this setting, the probabilistic model is used surrogate of expensive black-box objective. At each iteration the surrogate evaluated at 128 random locations in the input domain and the input corresponding the minimum value is selected the next query point. The objective is evaluated at this location and added to the context dataset. Figure 6 shows the regret (distance from the true minimum) for the different models. We observe that the NDP almost matches the performance of GPR, which is the gold standard model on this type of task. NDPs also outperforms the NPs on both tasks. The important difference between GPR and the NDP is that the NDP requires no training during the BO loop, whereas GPR is retrained at every step.
5.3 NEURAL DIFFUSION PROCESSES TO MODEL THE JOINT p(æ, y)
So far, we have used NDPs to model ply x; D). This is a natural choice as regression is typically the only quantity of interest make predictions. However,
straightforwardly extend
We http:://ggtahb..oomwwssshheee/np
for all NPs it provides SOTA implementations.
8

---

Global optimisation By taking advantage of the NDP's ability to model the full joint distribution p(x", yy (D) we could conceive of a new global optimisation strategy. Consider conditioning the NDP model on our current belief about the minimum value y" This allows us to obtain samples from p(z" y which provide information about where the minima lies in the input domain according to the model. In Fig. 9 we illustrate global optimisation routine using this idea. In each step, we sample target from p(y"). which describes sample y" from truncated-normas function values, the variance corresponds to the variance of the observations. The next query point is selected by sampling p(x" " y" D), thereby systematically seeking out the global minimum. This experiment showcases NDP's ability to model the complex interaction between inputs and outputs of Under review as a conference paper at ICLR 2023
NDP (ours)
GPR
ANP [12]
NP [8]
Random
16
2
0
0
10
20
30
20
40
20
40
60
60
step
step
step
step
(a) Hartmann 3D
(b) Rastrigin 4D
(c) Ackley 5D
(d) Hartmann 6D
Figure 6: The regret of several probabilistic models used in Thompson sampling based Bayesian Optimisation on Hartmann 3D, Rastrigin 4D, Ackley 5D and Hartmann 6D optimisation problems.
NDPs to model the joint p(æ, y D). For this, during the forward process we corrupt both the function inputs and outputs with additive Gaussian noise. The task of the reverse process now consists of denoising both the corrupted inputs æ, and outputs y2, which leads to the objective
(9) where we highlighted the differences with Eq. (4) in orange. Importantly, in this case we require a noise model for both the inputs and outputs: Eg and cØ. resp. We detail the architecture of this NN in the supplementary (Fig. 7). We design it such that eg and e/ share most of the weights. The inputs the NN are now the corrupted inputs H, vā,ao + /I ā₁E and outputs y, Vā,yo + a,ey. This is in contrast to the previous NDP in which
Co Er,... Mo.E
||EE (z, Mer t)l + ll,y - (a Ve: [:]
Io for all t.
belief about the minima. For the experiment, where the mean corresponds to the minimum of the observed
function -a task on which it was not trained.
6 CONCLUSION
Limitations As with other diffusion models, we found that the sample quality from a NDP improves with the number of diffusion steps T. This does however lead slower inference times relative to other architectures such as GANs. Techniques for accelerating the inference process could be incorporated ameliorate this issue. The method proposed by Watson al. [37] is of particular interest for NDPs as we have a principled and differentiable metric to assess sample quality, namely the corresponding GP's marginal likelihood. Secondly, as is common with neural networks we found that it is important for test input points to lie within the training range, as going beyond leads to poor performance. This issue also well-known for NP models Gordon et al. [9].
We proposed Neural Diffusion Processes (NDPs), denoising diffusion model approach for learning probabilities on function spaces, and generating prior and conditional samples functions. NDPs generalise diffusion models to random variables. We introduced a new neural network building block, namely the bi-dimensional attention block, which wires dimension and sequence equivariance into the neural network architecture such that it behaves like a stochastic process. We empirically show that NDPs are able to capture functional distributions that richer than GPs and more accurate than NPs.
function spaces by allowing the indexing of

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Joel Berkeley, Henry B. Moss, Artem Artemev, Sergio Pascual-Diaz, Uri Granta, Hrvoje Stojie, Ivo Couckuyt, Jixiang Qing, Nasrulloh Loka, Andrei Paleyes, Sehastian W. Ober, and Victor Picheny. Trieste, 2022. URL https: /ttp:://gittiie /ggthub.... com/ hoom/scconee edd www.ooomiiddanno/oooooeeeeeeeeee labs/trieste,
[2] Wessel Bruinsma, James Requeima, Andrew YK Foong, Jonathan Gordon, and Richard Turner. The Gaussian neural process. In Advances in Approximate Bayesian Inference.
[3] Yann Dubois, Jonathan Gordon, and Andrew YK Foong. Neural process family. http: //yanndubs. hww.mcuui...ttt.....eeeaaaaaaeae
[4] Vincent Dutordoir, Nicolas Knudde, Joachim van der Herten, Ivo Couckuyt, and Tom Dhaene. Deep gaussian process metamodeling of sequentially sampled non-stationary response surfaces. In Winter Simulation Conference, 2017.
[5] Vincent Dutordoir, Hugh Salimbeni, James Hensman, and Mare Deisenroth. Gaussian process conditional density estimation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL http:: httpttpppppppppppeeeeeeeo ngs neurips cc/paper/2 pdf.
[6] Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. GP-VAE: Deep probabilistic time series imputation. In International Conference on Artificial Intelligence and Statistics, 2020.
[7] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Jimenez Rezende, and S. M. Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, 2018.
[8] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. 2018.
[9] Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes. International Conference on Learning Representations, 2019.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems, 2020.
[11] Emiel Hoogeboom, Víctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3D. In International Conference on Machine Learning, pp. 1867-8887, 2022.
[12] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosen- baum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
[13] Y.-L. Kom Samo and S. J. Roberts. Generalized Spectral Kernels, arXiv preprint, 2015. URL http:// arxiv. com abs 1506. 02236. D.Xiv:1506.0225
[14] Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2020.
[15] Jannik Kossen, Neil Band, Aidan N. Gomez, Clare Lyle, Tom Rainforth, and Yarin Gal. Self- attention between datapoints: Going beyond individual input-output pairs in deep learning. arXiv:2106.02584 2021.
[16] Vidhi Lalchand and Carl Edward Rasmussen. Approximate inference for fully Bayesian Gaussian process regression. In Advances Approximate Bayesian Inference, 2020.
[17] Sulin Liu, Xingyuan Sun, Peter J Ramadge, and Ryan P Adams. Task-agnostic amortized inference of Gaussian process hyperparameters. Neural Information Processing Systems, 2020.
10

---

Under review as a conference paper at ICLR 2023
[18] James Robert Lloyd, Peter Orbanz, Zoubin Ghahramani, and Daniel M. Roy. Random function priors for exchangeable arrays with applications to graphs and relational data. In Neural Information Processing Systems, 2012.
[19] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. 2022.
[20] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Computer Vision and Pattern Recognition, 2021.
[21] Stratis Markou, James Requeima, Wessel Bruinsma, and Richard Tumnr. Efficient Gaussian neural processes for regression. arXiv preprint vxxiv:2008.0967. 2021.
[22] Swapnil Mishra, Seth Flaxman, Tresnia Berah, Mikko Pakkanen, Harrison Zhu, and Samir Bhatt. pivae: Encoding stochastic process priors with variational autoencoders. arXiv preprint arXiv:2002.06873 2020.
[23] R. M. Neal. Regression and classification using gaussian process priors. In J. M. Bernardo, J. O Berger, J. W. Dawid, and A. F. M. Smith (eds.), International Conference on Learning Representations. Oxford University Press, 1998.
[24] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 10741, 2021.
[25] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 2022.
[26] Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. MIT Press, 2006.
[27] Hugh Salimbeni, Vincent Dutordoir, James Hensman, and Marc P. Deisenroth. Deep Gaus- sian Processes with Importance- Weighted Variational Inference. In Proceedings of the 36th International Conference on Machine Learning (ICML).
[28] Bobak Shahriari. Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: review of bayesian optimization. Proceedings of the IEEE, 2015.
[29] Fergus Simpson, Ian Davies, Vidhi Lalchand, Alessandro Vullo, Nicolas Durrande, and Carl Ed- ward Rasmussen. Kernel identification through transformers. In Neural Information Processing Systems, 2021.
[30] Fergus Simpson, Vidhi Lalchand, and Carl Edward Rasmussen. Marginalised Gaussian pro- cesses with nested sampling. In Neural Information Processing Systems, 2021.
[31] Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: 1rXiv:1503.03585 2015.
[32] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Neural Information Processing Systems, volume 33, 2020.
[33] Sattar Vakili, Henry Moss, Artem Artemev, Vincent Dutordoir, and Victor Picheny. Scalable thompson sampling using sparse gaussian process models. In Neural Information Processing Systems, 2021.
[34] Mark van der Wilk, Vincent Dutordoir, ST John, Artem Artemev, Vincent Adam, and James Hensman. A framework for interdomain and multioutput Gaussian processes. arxiv preprint arXiv:2003.01115, 2020.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
11

---

Under review as a conference paper at ICLR 2023
[36] Chunyi Wang and Radford M Neal. Gaussian process regression with heteroscedastic non-gaussian residuals. arXiv preprint arXiv:1212.6246, 2012.
[37] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations, 2021.
[38] Andrew G. Wilson and Ryan Adam. Gaussian process kernlss for pattern discovery and extrapolation. International Conference on Machine Learning. 2013.
[39] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Arthur Gretton and Christian C. Robert (eds.). Proceedings of the 19th International Conference on Artificial Intelligence and Statisties, volume 51 Proceedings of Machine Learning Research, pp- 370-378, Cadiz, Spain, 09-11 May 2016. PMLR.
[40] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff:: geo- metric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.0299 2022.
[41] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Neural Information Processing Systems, 2017.
12


Reviews:
[{"review_id": "rv6zxMV9pLz", "reviewer": "Reviewer_d5b3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use a diffusion based approach to model stochastic processes and thus enabling sampling from distributions over functions. A novel attention mechanism is proposed to build exchangeability and marginal consistency into the neural network model architecture. The model is trained to capture prior predictive distributions and to obtain posteriors, conditioning information is incorporated at inference time. The proposed approach reduces the computational complexity with respect to Gaussian processes and enables non-Gaussian priors. Experiments demonstrate automatic hyperparameter marginalization properties and applications to Bayesian optimization.\n", "summary_of_the_review": "Diffusion models are beating state of the art in various problems and this paper proposes their application to stochastic processes, which is a timely and novel contribution. The paper demonstrates the approach through interesting applications in hyperparameter marginalization and joint distribution modeling. It is a good paper, I recommend acceptance. The only reason why I am not recommending a strong acceptance is due to the lack of real data experiments.", "strength_and_weaknesses": "Strengths:\n- The paper makes a novel contribution combining diffusion modeling with stochastic processes.\n- Gaussian processes’ cubic scaling with dataset size is a big limitation and this paper offers a technique reducing computational complexity to O(TN^2). With the recent advancements in diffusion modeling, T is reduced to as few as 4 steps (Salimans and Ho, 2022), which might be an important point for this technique’s adoption.\n- The proposed technique also offers unique properties ranging from automatic marginalization of hyperparameters without added computational cost, and modeling the joint distribution p(x,y) enabling interesting global optimization schemes.\n\nWeaknesses:\n- The experiments are conducted using simulated data. It would be interesting to see the performance of the proposed technique in real data settings. As an example, neural processes (NP) (Garnelo et al. 2018) explore image completion as a regression task with application to MNIST and CelebA datasets. It would be interesting to see comparisons between the proposed approach and NP.\n\nOther comments:\n- Kossen et al. (2022) proposes non-parametric transformers where the architecture consists of layers of attention mechanism across datapoints and attributes to enable reasoning using an entire dataset. Can the authors comment on the similarities between this and the proposed bi-dimensional attention block?\n- Why do you show q(x_t|x_{t-1}) and p_\\theta(x_{t-1}|x_t) in Eqns 2 and 3 since x’s are treated deterministically?\n\n\nSalimans, T. and Ho, J., 2022. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512.\n\nGarnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D.J., Eslami, S.M. and Teh, Y.W., 2018. Neural processes. arXiv preprint arXiv:1807.01622.\n\nKossen, J., Band, N., Lyle, C., Gomez, A.N., Rainforth, T. and Gal, Y., 2021. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. Advances in Neural Information Processing Systems, 34, pp.28742-28756.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ekYOTBFE5H", "reviewer": "Reviewer_YxyB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed Neural Diffusion Processes to specify distribution over functions. It uses a similar learning approach to diffusion models, but generalized the dimensions to infinities. Prior and conditional sampling from NDP were discussed. The authors designed a novel bi-dimensional attention block in the architecture of the noise model, to ensure dimension and sequence equivariance, and therefore making it satisfy the properties of stochastic processes. Experiments on synthetic datasets compared NDP with Gaussian Process and Neural process, and presented a way for global optimization of black-box functions.", "summary_of_the_review": "The paper provides a new of using the diffusion model, ie to model the distribution of functions. This is not a trivial extension, and the paper provides rounded support to justify its correctness. Experiment results are well-rounded to show the proof of concept, but not very significant from the practical point of view. Writing could be improved to put more emphasis on the key parts of the contributions. Given these, I would give it a recommendation of marginally above the acceptance threshold.", "strength_and_weaknesses": "The submission mentioned the complexity of GP is $O(N^3)$ and NDP is $O(TN^2)$:\n- How was $T$ selected in the experiments in practice? In many plots $N$ is small, how did $T$ compare to $N$?\n- I assume the proposed method used auto diff and gradient updates to minimize the objective. How long did it take to converge? Compared to GP regression, which has closed-form updates, is the computation of NDP really faster?\n\nI can see Sec. 5.3 also added noise to the input $x$ to model the joint distribution $p(x, y)$. Could you explain the difference between the Bayesian optimization in Sec. 5.2 and the global optimization in Sec. 5.3?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZojXzPEJBTw", "reviewer": "Reviewer_6nma", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a new class of stochastic processes that they think of as a generalization of diffusion models to infinite-dimensional inputs. \n", "summary_of_the_review": "The attempted generalization of diffusion models to functions is interesting, but the work requires more polishing before it is ready for publication.", "strength_and_weaknesses": "- **Strength:** Any attempt to mary the pragmatism, effectiveness and scalability of deep learning with the \"principledness\" of Bayesian nonparametrics is always laudable. This paper is no exception.\n\n- **Weaknesses:**\n\n**1- Incorrect Construction:** Why are the sample size and the input dimension drawn rather than fixed, let alone from a uniform distribution? As a stochastic process, what space are NDPs indexed in? \n\n**2- Wrong Consistency Proof:** One would expect that what makes NDPs a generalization of diffusion models (DMs) is that the forward and reverse processes operate on functions, not simply fixed and finite dimensional inputs, through an arbitrary (but deterministic) number of inputs and function values. \n\nIn this case, consistency ought to be proven not across time, but with respect to the choice of inputs. For instance, you need to show that if you have 3 inputs $(a, b, c)$ and associated outputs $(y_a, y_b, y_c)$ at time $0$, whether you consider all 3 inputs, $a$ and $b$, $b$ and $c$, or only $b$, at time $T$ the $b$-marginals are all the same. The same goes for the reverse process.\n\n**3- Insufficient Motivation:** How does the infinite dimensional extension of DMs improve on DMs?\n\n**4- Incomplete Literature Review:** There is an extensive literature on expressive kernels (e.g. Sparse Spectrum Kernels, Spectral Mixture Kernels, Generalized Spectral Kernels (stationary and non-stationary), etc.) that is worth reviewing when comparing GPs with NDPs. Additionally, at the end of the day the forward process should map a structured process into a Gaussian white noise, and the reverse process should map a Gaussian white noise into a structured process. Many such approaches have already been proposed that would have been useful to review here (e.g. the Karhunen–Loève expansion, etc.).\n\n**5- Other Mistakes:**  E.g. The section \"Non-Gaussian Posteriors\" contains several mistakes. First, the covariance function of any $L^2$ process is also the covariance function of a mean-zero GP. So one cannot say that \"it is impossible for a GP to represent the covariance arising from 1D step functions when the step occurs at a random location in its domain\". Similarly, the sentence \"The key takeaway from this experiment is that the NDP can infer a data-driven covariance, which need not be Gaussian and thus impossible for GPs to model.\" is incorrect. There is no such a thing as a Gaussian covariance, and expressive covariance functions (e.g. GSKs) should have been considered here.\n\n \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4umGTfa5Hf", "reviewer": "Reviewer_sTEa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a diffusion model, Neural Diffusion Process (NDP), to represent a distribution over function spaces by parameterizing the score model with a neural network. To guarantee the essential properties of stochastic processes, e.g., exchangeability, the authors propose a bi-dimensional attention block, which ensures equivariance over the input dimensionality and sequence order.", "summary_of_the_review": "I think the paper provides an interesting idea and experimental results (marginalization, Bayesian optimization) on synthetic data demonstrate several interesting features of the proposed method.  ", "strength_and_weaknesses": "**Strength**\n- The paper provides novel contributions (NDP formulation, bidirectional attention module) and properly enforces important properties of stochastic processes into the neural architecture. The model is invariant to the input dimension, which is a desired property for neural net families to model function spaces.\n\n**Weakness**\n- The empirical comparison seems interesting but the authors should consider a broader family of BO objectives beyond the Hartmann function, which is quite easy to imitate the inference of GP. How does NDP perform on the Rastrigin function or the Ackley function in high-dimensional space?  \n\n**Question**\n- NDP provides an interesting idea to model the joint distribution of a given function space in Section 5.3. However, it is confusing to me since I think it is nothing more than changing the delta distribution in equations (2) and (3). Why do you separate those contents from Section 3?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject