Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a diffusion model, Neural Diffusion Process (NDP), to represent a distribution over function spaces by parameterizing the score model with a neural network. To guarantee the essential properties of stochastic processes, e.g., exchangeability, the authors propose a bi-dimensional attention block, which ensures equivariance over the input dimensionality and sequence order.
strength_and_weaknesses: **Strength**
- The paper provides novel contributions (NDP formulation, bidirectional attention module) and properly enforces important properties of stochastic processes into the neural architecture. The model is invariant to the input dimension, which is a desired property for neural net families to model function spaces.

**Weakness**
- The empirical comparison seems interesting but the authors should consider a broader family of BO objectives beyond the Hartmann function, which is quite easy to imitate the inference of GP. How does NDP perform on the Rastrigin function or the Ackley function in high-dimensional space?  

**Question**
- NDP provides an interesting idea to model the joint distribution of a given function space in Section 5.3. However, it is confusing to me since I think it is nothing more than changing the delta distribution in equations (2) and (3). Why do you separate those contents from Section 3?
clarity,_quality,_novelty_and_reproducibility: **Novelty**: The proposed method theoretically generalizes diffusion models to function spaces, which seems quite interesting and looks novel. NDPs model the distribution of input-output pairs and address it with a specially-designed architecture of neural networks. The novelty seems to be the main strength of this paper.

**Quality**: The proposed method is technically sound, and the theoretical results are solid to demonstrate the strength of NDPs compared to previous NP models. 

**Clarity**: The paper is well-written, and the figures are illustrative. Notations are clean.

**Reproducibility**: The paper seems to provide sufficient information in the appendix to reproduce the results although a code is not provided.
summary_of_the_review: I think the paper provides an interesting idea and experimental results (marginalization, Bayesian optimization) on synthetic data demonstrate several interesting features of the proposed method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors introduce a new class of stochastic processes that they think of as a generalization of diffusion models to infinite-dimensional inputs. 

strength_and_weaknesses: - **Strength:** Any attempt to mary the pragmatism, effectiveness and scalability of deep learning with the "principledness" of Bayesian nonparametrics is always laudable. This paper is no exception.

- **Weaknesses:**

**1- Incorrect Construction:** Why are the sample size and the input dimension drawn rather than fixed, let alone from a uniform distribution? As a stochastic process, what space are NDPs indexed in? 

**2- Wrong Consistency Proof:** One would expect that what makes NDPs a generalization of diffusion models (DMs) is that the forward and reverse processes operate on functions, not simply fixed and finite dimensional inputs, through an arbitrary (but deterministic) number of inputs and function values. 

In this case, consistency ought to be proven not across time, but with respect to the choice of inputs. For instance, you need to show that if you have 3 inputs $(a, b, c)$ and associated outputs $(y_a, y_b, y_c)$ at time $0$, whether you consider all 3 inputs, $a$ and $b$, $b$ and $c$, or only $b$, at time $T$ the $b$-marginals are all the same. The same goes for the reverse process.

**3- Insufficient Motivation:** How does the infinite dimensional extension of DMs improve on DMs?

**4- Incomplete Literature Review:** There is an extensive literature on expressive kernels (e.g. Sparse Spectrum Kernels, Spectral Mixture Kernels, Generalized Spectral Kernels (stationary and non-stationary), etc.) that is worth reviewing when comparing GPs with NDPs. Additionally, at the end of the day the forward process should map a structured process into a Gaussian white noise, and the reverse process should map a Gaussian white noise into a structured process. Many such approaches have already been proposed that would have been useful to review here (e.g. the Karhunen–Loève expansion, etc.).

**5- Other Mistakes:**  E.g. The section "Non-Gaussian Posteriors" contains several mistakes. First, the covariance function of any $L^2$ process is also the covariance function of a mean-zero GP. So one cannot say that "it is impossible for a GP to represent the covariance arising from 1D step functions when the step occurs at a random location in its domain". Similarly, the sentence "The key takeaway from this experiment is that the NDP can infer a data-driven covariance, which need not be Gaussian and thus impossible for GPs to model." is incorrect. There is no such a thing as a Gaussian covariance, and expressive covariance functions (e.g. GSKs) should have been considered here.

 




clarity,_quality,_novelty_and_reproducibility: The paper is fairly easy to read, the topic is interesting, but the paper contains a few mistakes.
summary_of_the_review: The attempted generalization of diffusion models to functions is interesting, but the work requires more polishing before it is ready for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to use a diffusion based approach to model stochastic processes and thus enabling sampling from distributions over functions. A novel attention mechanism is proposed to build exchangeability and marginal consistency into the neural network model architecture. The model is trained to capture prior predictive distributions and to obtain posteriors, conditioning information is incorporated at inference time. The proposed approach reduces the computational complexity with respect to Gaussian processes and enables non-Gaussian priors. Experiments demonstrate automatic hyperparameter marginalization properties and applications to Bayesian optimization.

strength_and_weaknesses: Strengths:
- The paper makes a novel contribution combining diffusion modeling with stochastic processes.
- Gaussian processes’ cubic scaling with dataset size is a big limitation and this paper offers a technique reducing computational complexity to O(TN^2). With the recent advancements in diffusion modeling, T is reduced to as few as 4 steps (Salimans and Ho, 2022), which might be an important point for this technique’s adoption.
- The proposed technique also offers unique properties ranging from automatic marginalization of hyperparameters without added computational cost, and modeling the joint distribution p(x,y) enabling interesting global optimization schemes.

Weaknesses:
- The experiments are conducted using simulated data. It would be interesting to see the performance of the proposed technique in real data settings. As an example, neural processes (NP) (Garnelo et al. 2018) explore image completion as a regression task with application to MNIST and CelebA datasets. It would be interesting to see comparisons between the proposed approach and NP.

Other comments:
- Kossen et al. (2022) proposes non-parametric transformers where the architecture consists of layers of attention mechanism across datapoints and attributes to enable reasoning using an entire dataset. Can the authors comment on the similarities between this and the proposed bi-dimensional attention block?
- Why do you show q(x_t|x_{t-1}) and p_\theta(x_{t-1}|x_t) in Eqns 2 and 3 since x’s are treated deterministically?


Salimans, T. and Ho, J., 2022. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512.

Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D.J., Eslami, S.M. and Teh, Y.W., 2018. Neural processes. arXiv preprint arXiv:1807.01622.

Kossen, J., Band, N., Lyle, C., Gomez, A.N., Rainforth, T. and Gal, Y., 2021. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. Advances in Neural Information Processing Systems, 34, pp.28742-28756.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and organized. Experiments are provided in detail to enable reproducibility of the results. Empirical results are presented with error bars. To my knowledge, this is the first study using diffusion models for stochastic processes, so the contributions are quite unique.
summary_of_the_review: Diffusion models are beating state of the art in various problems and this paper proposes their application to stochastic processes, which is a timely and novel contribution. The paper demonstrates the approach through interesting applications in hyperparameter marginalization and joint distribution modeling. It is a good paper, I recommend acceptance. The only reason why I am not recommending a strong acceptance is due to the lack of real data experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed Neural Diffusion Processes to specify distribution over functions. It uses a similar learning approach to diffusion models, but generalized the dimensions to infinities. Prior and conditional sampling from NDP were discussed. The authors designed a novel bi-dimensional attention block in the architecture of the noise model, to ensure dimension and sequence equivariance, and therefore making it satisfy the properties of stochastic processes. Experiments on synthetic datasets compared NDP with Gaussian Process and Neural process, and presented a way for global optimization of black-box functions.
strength_and_weaknesses: The submission mentioned the complexity of GP is $O(N^3)$ and NDP is $O(TN^2)$:
- How was $T$ selected in the experiments in practice? In many plots $N$ is small, how did $T$ compare to $N$?
- I assume the proposed method used auto diff and gradient updates to minimize the objective. How long did it take to converge? Compared to GP regression, which has closed-form updates, is the computation of NDP really faster?

I can see Sec. 5.3 also added noise to the input $x$ to model the joint distribution $p(x, y)$. Could you explain the difference between the Bayesian optimization in Sec. 5.2 and the global optimization in Sec. 5.3?


clarity,_quality,_novelty_and_reproducibility: **Clarity**: several parts of the paper was a bit vague to me and thus may not be very clearly presented
 - In Sec. 2.2, what is the math details behind Neural Processes, especially attentive neural processes? Considering that it generated quite comparable results to NDP in Table 1 and Figure 6, some discussions about its strengths, weakness and similarities to NDP would be beneficial.
- The bi-dimensional attention block seems to be a key contribution of the paper. A clear definition in the main paper about it as well as the entire noise prediction model would help with understanding. Now it everything is in the Appendix except the diagram in Figure 3.

**Quality**: the paper provided detailed derivations on the following aspects, making it a coherent work
- forward and backward computation
- the loss derivation and training algorithm
- prior and conditional sampling
- permutation equivariance and dimensionality invariance

**Novelty**: the idea of applying the diffusion model to distributions of functions is relatively new and not trivial
summary_of_the_review: The paper provides a new of using the diffusion model, ie to model the distribution of functions. This is not a trivial extension, and the paper provides rounded support to justify its correctness. Experiment results are well-rounded to show the proof of concept, but not very significant from the practical point of view. Writing could be improved to put more emphasis on the key parts of the contributions. Given these, I would give it a recommendation of marginally above the acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper propose to extend diffusion models to define distributions over functions. The paper works on pairs of input , function output , where a score function is learned to denoise noisy output of the function given the position of the input.

A special care is added to ensure permutation invariance w.rt to input /output pairs, and invariance to dimensionality when learning the score function. A new attention is devised for this purpose. The proposed method  is then evaluated on synthetic datasets for hyperparameter marginalisation, non-Gaussian posteriors and baysian  optimisation.

The paper was discussed at length between authors , reviewers and AC. 

Reviewers thought that the idea is promising nevertheless it has few issues on fundamental level and on the applications level proposed in the submission. 

* consistency of the construction needs to be revised and explained in which sense it is meant in the paper. Especially regarding the backward process as pointed out by reviewer 6nma and acknowledged by the authors. 
* The lack of real world data experiments was also pointed out as limiting for the impact of the work by all reviewers. 



justification_for_why_not_higher_score: Better justification for the consistency of the "process" and better experimentation on real world problem.N/A
Decision: Reject