Deep Stochastic Processes via Functional Markov Transition Operators
Authors: Jin Xu
Emilien Dupont
Kaspar Märtens
Tom Rainforth
Yee Whye Teh
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural Processes
Bayesian Nonparammetric Models
TL;DR: We introduce a new class of expressive stochastic process models which are constructed by stacking sequences of neural parameterised Markov transition operators in function space.

Abstract: We introduce Markov Neural Processes (MNPs), a new class of Stochastic Processes (SPs) which are constructed by stacking sequences of neural parameterised Markov transition operators in function space. We prove that these Markov transition operators can preserve the exchangeability and consistency of SPs. Therefore, the proposed iterative construction adds substantial flexibility and expressivity to the original framework of Neural Processes (NPs) without compromising consistency or adding restrictions. Our experiments demonstrate clear advantages of MNPs over baseline models on a variety of tasks.

Deep Stochastic Processes via Functional Markov Transition Operators
Jin Xu
Emilien Dupont
Kaspar Märtens
Tom Rainforth'
Yee Whye Teh 18
Department Statistics, University Oxford, UK. Big Data Institute, University of Oxford, UK.
Abstract
We introduce Markov Neural Processes (MNPs), new class of Stochastic Processes (SPs) which are constructed by stacking sequences of neural parameterised Markov transition operators in function space. We prove that these Markov transition operators can preserve the exchangeability and consistency of SPs. Therefore, the proposed iterative construction adds substantial flexibility and expressivity to the original framework of Neural Processes (NPS) without compromising consistency or adding restrictions. Our experiments demonstrate clear advantages MNPS over baseline models on variety of tasks.
Introduction
Stochastic Processes (SPs) are widely used in many scientific disciplines, including biology [3]. chemistry [57]. neuroscience [34], physics [45] and control theory [1]. They are formed by (typically infinite) collection of random variables and can be used to model data by considering the conditional distribution of target variables given observed context variables. machine learning, SPs in the form of Bayesian nonparametric models-such Gaussian Processes (GPS) [47] and Dirichlet processes [56]-are are used in tasks such as regression, classification, and clustering. SPS parameterised by neural networks have also been used for meta-learning [21, 60] and generative modelling [15, 39].
With the increasing amount of data available, and the complex patterns arising many applications, more flexible and scalable SP models with greater learning capacity are required. The Neural Process (NP) family [18, 21, 22, 25, 30] meets this demand by parameterising SPS with neural networks, and enjoys greater flexibility and computational efficiency compared to traditional nonparametrie models.
Unfortunately, the original version of NPs [22] lacks expressivity and often underfits the data in practice [30]. Various extensions have therefore been proposed- such as Attentive Neural Processes (ANPS) [30], Convolutional Neural Processes (CONVNPS) [18, 25] and Gaussian Neural Processes (GNPS) [4] to improve expressivity. However, these models which we refer to as predictive SPS-are based around directly constructing mappings from contexts to predictive distributions, and therefore forgo the fully generative nature of the original NP formulation. This can be problematic as means that they are no longer consistent under conditioning: their predictive distributions no longer correspond to the conditional distribution of underlying SP prior. In turn, this can cause variety of issues, such as conflicts or inconsistencies between predictions and miscalibrated uncertainties.
To address these issues, we propose an alternative mechanism to extend the NP family and provide increased expressivity while maintaining their original fully generative nature and consistency. We
"Corresponding author: <jin.xudstats xutstats.ccmmm ac. uk>
'ED is now Google DeepMind; this work done while ED Oxford. 4YWT is at both Google DeepMind and Oxford: this work was done at Oxford.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Step (trivial SP)
Step 1
Step 2
Step 3 (final SP)
MNPs to model distributions over monotonic functions. We start with trivial initial SPs and gradually transform them into more complex SPs conditioned observed context points marked as black crosses. The blue lines represent sampled mean functions and the shaded region indicates one standard deviation.
Figure 1: MIPS construct expressive SPs via iterative transformations. Here
begin by generalising the marginal density functions of NPs. The generalised form can be viewed as Markov transition operators with functions as states. This lays the foundation for stacking these operators to construct more powerful generative SP models that call Markov Neural Processes (MNPS). MNPS can be seen as Markov chains in function space, parameterised by neural networks: they make iterative transformations from simple initial SPs into more flexible, yet properly defined, SPS (as illustrated Figure 1), without compromising consistency or introducing additional assumptions.
To empirically demonstrate the value of our proposed approach, we first benchmark MNPs against baselines on ID function regression tasks, and conduct ablation studies in this controlled setting. We then show that they can be used as nigh-performing surrogate for contextual bandit problems. Finally, we apply them to geological data, for which they demonstrate encouraging performance.
2 Background
SP a (typically infinite) collection of random variables defined on a common probability space. We can consider SP as a random function F X + y where inputs can be regarded as indexing the output random variables. With a relaxed use notation, we employ p(J)) ()) denoting SP, where f maps inputs E X' to outputs y E y. Kolmogorov's Consistency Theorem shows that SP can be indirectly defined via collection of marginal distributions, (Pr... (y):.) Ex. (we drop X from here on for conciseness) if they, for any permutation and all possible sets of inputs Tinn X satisfy the exchangeability condition:
Pri. (yim) Px(2).)) (((())))(((()))))) = Precs)em (Un(1): -.. (n(n)))
and the (marginal) consistency condition:
[ PPr... (y)va) dymm+Im S m < n.
Prism (y)mm)
(2)
If none of the random variables in a SP are observed, we call it prior SP. For any two distinet subsets of datapoints C {2,, 34) )m) (the context) and T {(x,, 9:) y=)) (the target), one can use this prior SP to compute the conditional density Pzm-timlaa... (ym++lal (1:mm via Bayesian inference. inference is exact, it can be proved that the collection of conditional distributions (Pam+ Fm (ym++ 1mm [/:mm) Jm also satisfy exchangeability and consistency, and hence define a valid posterior SP, PC((C. We discuss the impact of approximate inference in Section 4.3.
Based conditional version of de Finetti's Theorem, NP [22] defines SP by providing collection of exchangeable and consistent marginal densities parameterized by neural networks. Specifically, they set up their marginal densities to have the form:
Pei... (yim: 8) = po (2) IIT N(uluolE, 2), oz 2))
(3)
where is a latent variable which captures dependencies across different input locations, yi = (xx), [4g and ag are deep neural networks, and po(2) is a, typically Gaussian, prior distribution on the latents. Note that this form is more general than the one in [22] where both pe and dg are not learnt.

---

3 Generative versus predictive stochastic process models
Before introducing our MNP approach, we first delve into the distinctions between conventional, consistent, SP models like NPS, and the predictive SP models corresponding to popular NP extensions, such as ANPS, CONVNPS, and GNPS, highlighting some of the potential drawbacks with the latter.
We introduce the term generative SP model to refer to the standard case where one first specifies a prior SP, p(f). and then relies on Bayesian inference to compute posterior SP, p(f(C). The context and the target are only distinguished for inference, and they are treated equally in model specification. Traditional nonparametric models such as GPS, Student-t processes and the original NPs belong to this category. Since posterior SPS are inferred under the same prior SP, under the condition of exact inference, for two different contexts C {(1,, y:) and C' = =[( 31) Nee', we have
f p(J(C'p=e (VC)))peeee)dg) = f D(((C"))p (vC* )dye",
where æe := fee and ze. := hee are taken as fixed, and Pxe (ye) and Pxcr (yer) are defined through the prior SP p(f). We call this property conditional consistency to set apart from marginal consistency defined in Equation (2). Note that conditional consistency a prerequisite for marginal consistency to hold for the prior ():)) if the two integrals in Equation (4) are not equal, at least one must be different the prior distribution p(f).
By contrast, predictive SP models directly construct mappings from context C to a predictive SP C(f;C) and make clear distinction between the context and the target in model specification. Consequently, for these models p(ʃ; C) no longer corresponds to a posterior derived from a certain prior p(f), they longer need to satisfy conditional consistency. As such, they no longer form valid conditioning of SP: though they are typically constructed to ensure p(f: f: C)) is itself valid sp for new evaluations of f with C held fixed, this is longer itself derived as the conditional of prior SP. As such, predictive SP models are not consistent under updating and no longer update their uncertainties in Bayesian manner as new data is observed. In short, they are no longer treating the data itself as being drawn from an SP.
Though predictive models have proven to be effective tools for meta- meta-learning tasks, such as 1-D regression, image regression, and few-shot image classification [4, 18, 30, 60), this lack of consistent updating can be problematic in scenarios where the context is not fixed, such when performing sequential updating. In particular, their uncertainties will be mismatched with how the model is updated in practice as new data observed. This has the potential to be problematic for wide variety of possible applications involving sequential decision-making, such as Bayesian experimental design [7, 46], active learning [2. 27, 51], Bayesian optimization [19, 23], and contextual bandit problems [35, 50]. There is also the potential for such models to fall foul of so-called Dutch Book scenarios [26], wherein the inconsistencies of the model can lead to conflicting predictions.
4 Markov Neural Processes
To correct the shortfalls of NPs while maintaining their conditional consistency, we now introduce a more expressive family of generative SP models termed Markov Neural Processes (MNPs). Our starting point is to extend NP density functions into generalised form that can be viewed as transition operator which transforms a trivial SP to more flexible one. MNPS are then formed by stacking sequences of these transition operators to form a highly expressive and flexible model class. The training and inference procedure for MNP mirrors that of NP, but we also introduce a novel inference model that allows for efficient learning in our scenario.
4.1 A more general form of Neural Process density functions
Recall the form of the NP marginal density functions from Equation (3). One can draw joint samples from Pri. (Ni.n: f) via reparameterization using:
(0) Vin
N(00
s po(s), y1 = ==)2 vf uo (2(
(5)
(0)
The key starting point for MNPS is to show that this can be generalised to the case where Vi.. is drawn from any given SP of its own, p()( and each y is any invertible transformation, Fe, of
3

---

" parameterized by and Specifically, we introduce the following result (see Appendix A for proof).
Proposition 4.1. Let Fo(:xx2) y + y denote an invertible transformation between outputs,
parameterized by the input and latent. If (PZu.. (olinnan is a valid SP (i.e. it satisfies Equation (1) and Equation (2)) and
Vim {Pr... (iin Famn 2~ p~)o()).
(0)
y/ Fo(u; :24, 2).
(6)
then {Pri. (Bh:a) Dar. is also a valid SP.
Our next step is to realise that Equation (6) can be viewed Markov transition in function space, denoted as p(f f(0)), which transforms simpler SP p(f f() ) to more expressive one p(f). We can thus generalise things further by stacking sequences of Markov transition operators function spaces (FMTOs), denoted by p ( .... ) f(r-1)), together to form a Markov chain f(0)
((T) function space. This will be the basis of MNPS.
4.2 Markov chains in function space
Analogously to defining SPS through its marginals, we indirectly specify FMTOs using collection of marginal Markov transition operators (MMTOs), denoted by {Pzz,. (V1:n (tt::Ilii.) (1th)) Jm where each MMTO is just Markov transition operator over a finite sequence of function outputs.
To adapt the definitions of consistency and exchangeability for SPs to the transition operator setting, we say that the MMTOs are consistent if and only if, for any 1 < m < n and sequence Iim X"",
f | (yi:n wi dymm-1:n Par... (y1.m
( Pr. (11 li
Similarly, MMTOs are exchangeable if, and only if, for all possible permutations n,
Prim (y (iin) (1 (v1())).
Note that, if we consider (Ira, vi (0)) inputs, and y; function outputs, the transition operator
Pri... (Ni. (0) can be seen as the finite marginals of a random function F" : X x ) + y whose
distribution is SP, and the conditions (7) and (8) correspond (2) and (1).
Provided that these MMTOs are consistent and exchangeable, the FMTOs in function space will also be well-defined indirectly-i.e the transition produces a well-defined given input random functions from a Sp-as per the following result (see Appendix A for proof):
Proposition 4.2. the collection of marginals before transition {Pei.. (yim))) is consistent and
exchangeable (as per Equations (1) and (2)) and the collection of MMTOs (Pmi.. (yr.u (y Jan is also consistent and exchangeable (as per Equations (7) and (8)), then the collection of marginals after transition {P.... (yr.)} is also consistent and exchangeable, hence defining a valid sP.
Furthermore, a Markov chain in function space can be constructed by a sequence of FMTOs p( p(f(())f())) )
where p(f p(f(0)((t))) = (Pe... (v (0---""))))))- ) With repeated applications of Proposition 4.2, if the initial state {P.... (v1.) 2 is exchangeable and consistent, {Pa,... ( )a.... at time T is also exchangeable and consistent, hence defining SP pT)(T).
Equation (6) then provides valid construction of consistent and exchangeable MMTOs by introduc- ing an auxiliary latent variable z. The transition operator is written as
12 ff po(2)) II 6(m Fo(y; (0) x,, 2), dz,
Pri. (/1:n 0 8)
where both po and Fg in Equation (9) be parameterised by neural networks. Critically, we can extend Proposition 4.2 to cover these auxiliary settings in Equation (9) as well, as per the following result (see Appendix A for proof):
Proposition 4.3. MMTOs in the form of Equation (9) are consistent and exchangeable.

---

4.3 Parameterisation, inference and training
We can now define a MNP as sequence of neural FMTOs, with each FMTO indirectly specified via a collection of MMTOs in the form of Equation (9). If we specify a distribution over the sequence of auxiliary latent variables p=(=(1:)) along with an initial SP with marginals Pri.. (wim) ). we can
write down the marginal distribution over function outputs Vim := Mi:n for a sequence of inputs Tim under the MNP model as (see Figure 2a for illustration and Appendix A for derivation):
T
Pri: (Hi.xi 0)= = (0( PeI...
(p(2(1))))p21. 11))))
Pg
(1
9; 11-1)
I ,,,, doy (0:7-1) dz(TT) (10)
1=1
=f po(2(1:T)) Prinn 1m
det OF (v. (1-1);; ; 24, 2(1))
(0)
dz
Oy. (1-1)
According to Propositions 4.2 and 4.3, {Pa... ()im::)) 2)).. defines a valid SP parameterised by 0.
The initial SP can be arbitrarily chosen, long experiments, we use a trivial SP where all the outputs are i.i.d. standard normal distributed. We adopt
evaluate its marginals Pæ1:. (v1 In our
normalising flows [16, 43, 49] to parameterise the invertible transformations F(())
To integrate over latent variables 2(1:T) in Equation (10), we introduce a posterior approximation Qua.n (z (z(1 Iy:m 6). For many applications, we need to learn and query the conditional distributions of a target
{ y/) { To better reflect the desired model behaviour at test time, similar to NPs [22], we train the model by maximising the following approximation of the conditional log- liglliillhood w.r.t. both model @ and variational @ parameters:
(zi 3))) given context
Lo (Mizi Fimm)
E gesin (2(1:) (n)) log Pam4i:n (Um++ 1:m (2(1)T), 0) + log Io (2(1:T)
3/ ())
(11)
2umn (2(1:T) 1/1:n: Φ)
where log Pe fram (ym+ton M1 0) 2 La F1mn). Here Qa (2 (1:2)) "ismm ②) can be seen an approximate prior conditioned on the context (0 9z) } while Zus... (2(1:T) V/:ni o)
is the approximate posterior after the target {<x; ys m is observed. Thus the prior and the posterior share the same inference model with parameters ø. The training objective is optimised by stochastic gradient descent, and the gradients Vo.aCo, (Fisni mn)) can be efficiently estimated with low variance using the reparameterisation trick [40].
However, different from NPS, the latent variables (1:7T) for MNPS are a sequence. Therefore, we propose a different inference model which shares parameters with the generative model above. Firstly, we write 2r (2(1:2) ylomi Φ) in factorised form:
9x (2(::) Vi:i Φ) II'
2 (2(0))(((11) (t) Φ)
(12)
where & are variational parameters and we set ((T+1) O, Vl.n = Vinn In our implementation.
we use a Gaussian distribution for each factor qei.. with mean p 20() (2(t+1), (1)
(8) (2(++1) (1) Fim)) and variance
1/9...11.. r1,)),, Here (!) and xn(() are parameterised functions invariant to the permutation of data points { 9:) i.e.
M (0) (2((t)) (c) 31m) = u/
(1) (2((t)) (1)
Vz(1::)> f+(1=))
100 (2((+1), (t) xim)) = 21 (2(++1), w )T(1=)))
We can parameterise them by first having
(1)) SETENCODDR (y F1:a))
(13)
where SETDDCDDER could be a Set Transformer [36], Deep Sets [61], then taking
2 (=(t+1),, , , xim) = MLP, /u (1,++,1),00)))) (t) o (2(1+1), yi x1sa) = MLP, (2 (1(11) ((+1),,()))
(14)
(15)
5

---

F
33)
y
y
2
2/
(0)
F (3)
2
F (2)
3/2
(2)
F (3)
2
(3)
Y/
F 4
y
(3)
y
Hidden
Stochastic
(1)
(2)
3
Observed
Deterministic
(a) Generative models
(0) y
F
91
F (1)-1
Y
F (3)
Y1
(0)
(1)
(2)
(3)
2/
11 F
Y2
F 2 1
Y
F
Y2
x2
step
2nd step
2
3rd step
(b) Inference models
Figure 2: (a) Consistent generative models on finite sets. Observed variables are shown in white, and hidden variables are shaded. Stochastic paths are indicated with dashed lines while deterministic paths solid lines. Conditioned the function inputs Fin and the auxiliary latent variables 2(1:T), the function outputs are transformed independently using instance-wise conditional normalising flows (CNFs). If the initial states {(14, y=) come from a SP, the construction will ensure that the collection of marginals p(y1:z (Ii:)) are consistent under marginalisation for any t. (b) Permutation- invariant inference models. Auxiliary latent variables z(1-T) are inferred in reverse order in our inference model. We reuse the parameters of CNFs in the generative model to compute function values vi.,. at all time steps, where each step has a different colour. We parameterise the conditional (t) distribution Cram (2((1), (t) Φ) with permutation- -ivaatiiiiinnnninntt neural networks.

---

In Equation (12), the intermediate function outputs Vim visn (1:7-1) are not observed. However, we can sample them autoregressively by iterating between sampling 2 and calculating y," (F(+t)))="( (++1); :z;; 2(++1) for each i (see Figure 2b). Therefore, we share the parameters of the normalising flows Fg. F=(121) between the generative and inference models, leading to better perfor- mance. This idea was originally proposed by Cornish et [1]], except that our normalising flows are applied to sequence of function outputs yi:n given the inputs Timz
In practice, the inference model q=1.. (z(1:2) 1yim; Φ) provides an approximate prior/posterior. Ideally, if it gives the exact posterior, conditional consistency would hold perfectly. However, when the inference model is approximate, the degree of conditional consistency depends on the discrepancy between the inference model and the true posterior. Predictive SP models also have stochastic mapping conditioned on the context, represented as Zz (y1mm: +). However, it is important not to confuse this with the inference model, as they do not serve to approximate the posterior.
5 Related work
Bayesian nonparametric models Bayesian nonparametric models such as GPs [24, 48] and Student-t processes [5, 10, 52] provide common classical approaches for modelling distributions over functions. Under these models, any conditional distribution of target given context can be directly evaluated, and both marginal/conditioall consistency is guaranteed by construction (if all computation is exact). However, they be too restrictive for some applications, e.g. any conditional or marginal distribution of a GP is also Gaussian. Further, the evaluation of conditional densities is typically computationally intensive (cubic w.r.t. the context due to matrix inversion). Deep GPS [13, 59] use GPS as building blocks to construct deep architectures, designed to be flexible enough to model wide range of complex systems. However, only the hyperparameters for each GP layer are tunable, which means they can still be restrictive when modelling highly non- Gan-Gassiann data.
Copula-based processes In multivariate statistics, a copula function refers to multivariate function that describes the dependence between random variables and links the marginal distributions of each variable to the joint distribution of all the variables [28, 41]. Similarly, copula process [58] describes the dependency between infinitely many random variables independent of their marginal distributions. [32, 33, 38] exploited this independence to specify the more flexible Copula- -ased Processes (CBPs) by combing the copula processes of existing SPS with flexible models of marginal distributions based on normalising flows [16, 43, 49]; the consistency of CBPs is guaranteed long as the base SPS are consistent. However, CBPS are still restrictive in terms of modelling relationship between variables because the underlying copula processes still come from known SPs. For example, BRUNOs [32, 33] have the same underlying copula processes as GPs, so they cannot be used to model data with -on--Gasiian dependencies.
Neural process family NPS [22] are generative SP models which specify a prior SP and rely on Bayesian inference to compute conditional densities. To improve expressivity of the original NPs, [4, 18, 30] explore predictive SP models that directly learn mappings from context to predictive SPS. More specifically, ANPS [30] incorporate cross-attention modules to model the interaction between context and target. CONVNPS [18] produce a functional representation with a convolutional archi- tecture which parameterizes stationary predictive SP over the target, given the context. In GNPs [4], predictive SPS are modelled by GPS where the mean/kernel functions are directly produced by neural networks conditioned on the context. However, simply setting the context to an empty set in these mod- els does not yield more expressive generative SP models. In the absence of context, ANPS and GNPS become standard NPS and GPS respectively, without any additional expressivity. CONVNPS can indeed be adjusted for generative SP models where the latent variables are random functions. However, it re- mains unclear how to perform Bayesian inference in function space [18]. Conditional NP families, e.g. Conditional Neural Processes (CNPs) [21], Convolutional Conditional Neural Processes (CONVCNPS) [25) are another category of predictive SP models, which make a strong assumption that all targets are independent given the context. Recently, Transformer Neural Processes (TNPs) [42], Neural Diffusion Processes (NDPs) [17] provide an alternative to the NP models above, showing promising predictive performance. However, both marginal and conditional consistency are sacrificed in these models.
7

---

Table 1: Estimated marginal log-likelihood of SPS 1D function regression problems. We compare (a) Oracle models (when available). (b) GPS with optimised hyperparameters for additive kernels with three component kernels including RBF kernel, Matern kernel and periodic kernel. (c) CBPS which combine Gaussian copula processes with neural spline flows [16]. (d) NPs. (e) MNPS with 7 transition steps. Each experiment repeated 5 times and report the mean and standard errors of the marginal log-likelihood normalised by the number of points in the target. When latent models such as MNPs, NPs are used, we obtain estimations of marginal log- likelihooooss on test data using the IWAE objective [6] with 20 latent samples.
Samples from GPs
Non-GP Data Convex
Model
RBF Kernel
Matern Kemel
Periodic Kernel
Monotonic
SDEs
Oracle
2.846:0.012
2.70910.013
0.641 1141::0.000 008
GPs
2.84440.013
2. 2.708
0.41940.013
0..39
2.976+0.224
1. 1.719
CBPs
2.628:to.u16
2..01
0.169 +0.022
1.776=u.088
4,268++o.as
1.842
NPs
0.935
1.115+0.022
0.356 020
1.823=0.006
1.956
1.621 10.000
MNPs
2.491 20.044
2.290#2.022
0.594420.0s8
2.755=20.000
5.58220.081
1.942=0.019
0.5
= 0.7 = 0.9
0.95 = 0.99
Figure 3: Wheel contextual bandits with varying exploration parameter The optimal actions are a1....,.0 05 when the context point are in blue, yellow, red, green and black regions respectively.
Experiments
Our experiments aim answer the questions: 1) Can the proposed framework of MNPS offer better expressivity than NPs? 2) Compared to Bayesian nonparametric models, do neural parameterised SPS have advantages, especially on highly non-Gaussian data? How well do MNPs perform scientific problems? All experiments are performed using PyTorch [44]. For details about datasets, please see Appendix B.I. For additional experimental details such as hyperparameters and architectures, please refer to Appendix B.2 and our reference implementation at https: //gs::///iiie github. com/ 1iixu066mmma mmn..
6.1 ID function regression
We first consider ID function regression problems to perform controlled comparisons between GPS, CBPS, NPs, and MNPS. Table shows results across a range of datasets, including samples from the GP prior (we consider three different kernels) well as challenging non-GP data.
For datasets sampled from GPs, we also include the performance of oracle GPs with the right hyperpa- rameters for generating the data. As shown, GPs and CBPS are close to the oracle except for samples generated using a periodic kernel, where CBPS struggle to learn the right kernel hyperparameters due to the difficulty of optimisation. For neural parameterised models, the performance gaps between the MNPS and the oracle GPS are much smaller than for NPS.
For non-GP samples such as monotonic functions, convex functions and samples from certain SDEs, MNPS perform the best across all models. CBPS obtain better marginal likellhoo than GPs because the pointwise marginals are transformed using normalising flows, but their underlying copula processes (which capture the dependence between data points) are still Gaussian, restricting their ability to model highly non-Gaussian data compared to neural parameterised models e.g. MNPs, NPs.
6.2 Contextual bandits
Contextual bandits are a class of problems where agents repeatedly choose from set actions based on context and receive rewards. The goal is then to learn policy that maximizes expected cumulative reward over time. Contextual bandits find applications in real-time decision making problems such as resource allocation and online advertising.

---

Table 2: Results wheel contextual bandit problems. We use increasing value of 6, where more exploration is needed with higher 5. We report the mean and standard deviation both cumulative and simple regrets (a performance measure of the final policy, estimated by the mean cumulative regrets in the last 500 steps) over 100 trials. Results are normalised by the performance of a uniform agent which chooses actions with equal probability.
0.5
0.7
0.9
0.95
0.99
Cumulative regrets
Uniform
100.0+0.00
100.010.00
100.0+0.00
100.0=0.00
100.0=0.00
MAML
2.95:10.12
3.11:1..16
0.84
7.01::0.35
22.93
NPs
1.60.1.06
1.75:10.05
3.:1100:10
5.7110.24
12.13:1.20
MNPs
1.08=0.00
1.23=0.01
2.10#t..01
2.07+0.01
5.46=#000
Simple regrets
Uniform
100.0
100.0+0.00
100.0+0.00
100.0=0.00
100.0
MAML
2.49+0.11
3.00
4.75++0.48
7.10=0.77
22.89+1.41
NPs
1.04
1.26-0.21
2.90
5.4520.47
21.45:1.3
C-BRUNO
1.32.4.06
1.43:0.07
3.44
6.17=0.21
21.5220.41
MNPs
1.14 10.06
1.29: 20.08
2.12 10.16
2.24
5.22 2.338
Ground truth
20
40
80
160
Top
Left
Center
Figure 4: Inferred geology conditioned on measurements. Given a small number of physical measurements (red pixels indicate positive measurements while blue pixels indicate negative ones), we show the predictive mean (first row in each panel) and standard deviation (second row in each panel). Different columns correspond to different context sets. As can be seen, MNPs make predictions that are consistent with the data while showing larger uncertainty when fewer context points are available (or when prediction is made far from a context point).
Similarly to Gareelo et al. [22], we use the wheel bandit problem [50] (see Appendix B. to evaluate our approach: unit circle is partitioned into 5 regions (see Figure 3) whose sizes are controlled by an exploration parameter 5. There are 5 actions a1, 02, 03, a4. as, and their associated reward depend on 2D contextual coordinate X (X,, X2) uniformly sampled from within the circle. If ||X|| < 6, a1 is the optimal action with reward sampled from (1.2, 0.012), and taking any other action would yield a reward r ~ (1.0, 0.01 3). ,. If ||X| > 6. the optimal action depends on which of the remaining region X falls into and choosing it would yield a reward
(50, 0.01²). Under this circumstance, any other action yield reward (1.0, 0.012) except that ai receives (1.2, 0.01²). We follow the experimental setup from Garnelo et al. [22] and only include models with a pre- training procedure (see Appendix B.2 for details). As can be seen in Table 2, MNPS significantly outperform basclines (taken from the results of [22]) across different exploration rates 8.
6.3 Geological inference
In geostatisties, one often interested inferring the geological structure of an area given only sparse set of measurements. This problem has traditionally been tackled with variants of GP regression

---

Table 3: Test marginal log-likelihood on geology data. Uniform context points uniformly sampled from within the 2D square, while regional context includes the top/left half or the central square with half the size.
Uniform Context
Regional Context Left
|C|
N = 20
40
80
160
Top
Center
NPs
00.3520.30
20.3
20.
-0.25
-39. -39.63
- 22.003
-1.1824.20
MNPs
0.67=0.31
0.80=2.23
0.98
1.12
0.44
0.22
0.65
[12] (often referred as Kriging in this context). However, several geological patterns (e.g. fluvial patterns) are highly complex and cannot be properly captured by these methods. To address this, several geostatistical models use a single training image to extract geological patterns and match these to the measurements [53, 63]. However, these approaches generally fail to produce realistic patterns capturing the variability of real geology.
More recently, deep learning approaches have been applied to this problem. [8, 14, 62] train GANs on large geological datasets and use this as a prior for inferring geological structure given sparse measurements. However, the resulting models do not provide any meaningful uncertainty estimates, which are crucial for decision making in several applications. As MNPS provide uncertainty estimates they may therefore be a compelling approach for this problem.
To evaluate MNPP on this problem, we introduce the GeoFluvial dataset, containing more than 25k simulations of fluvial patterns as 128x x 128 grayscale images. The dataset was generated using the open source meanderpy [54] package, which itself simulates the geological patterns as SPS (see Appendix B. for details). We train our model on a training set of 20k simulations and evaluate it on test set of 5k simulations. We our trained model on variety of context point configurations. corresponding to geological measurements taken at various spatial locations. Specifically, we consider uniformly sampled measurements (at 20, 40, 80, and 160 locations) as well as scenarios where measurements are spatially restricted to a certain area (top, left, and in the center of the square). Quantitative results are shown Table 3 and qualitative results Figure As can be seen, our model achieves better likelihoods than NPs on this problem for all context point configurations, while also generating patterns that match the geological context. Further, as can be seen in Figure 4, the uncertainty estimates of the model are consistent with expectations, i.c. the model is more uncertain far from mcasurement locations or when there is ambiguity in the direction of the fluvial pattern.
Discussion
Limitations and future work Because apply only finite number of Markov transitions, the entire computational graph containing intermediate states is held in memory for backpropagation. To alleviate this, interesting direction for future work would be to consider continuous time Markov transitions, i.e. stochastic differential equations in function space, which require less memory by using adjoint methods [9, 37]. While Equation (9) provides valid general construction of exchangeable and consistent MMTOs with latent variables, it may be feasible to investigate other forms of MMTOs that do not necessitate latent variables, thereby eliminating the need for variational approaches.
There are often inherent symmetries in data (c.g. translational symmetries for stationary SPS) and may be possible to improve empirical performance of MNPS using (group equivariant) convolutions that preserve certain symmetries [18, 25, 29]. These improvements are, however, orthogonal to our contributions and combining them is an exciting direction for future work.
Conclusions We have introduced Markov Neural Processes (MNPs), framework for constructing flexible neural generative SP models. MNPs use parrmettzzad Markov transition operators on function spaces to gradually transform simple initial SP into a flexible one. We proved that our proposed neural transitions preserve exchangeability and consistency necessary to define valid SPS. Empirical studies demonstrate that we can obtain expressive models of SPS with promising performance on contextual bandits and geological data.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Hyunjik Kim for valuable discussions. We also thank Peter Tilke for useful discussions around generating the geological dataset. JX gratefully acknowledges funding from Tencent AI Labs through the Oxford-Tencent Collaboration Large Scale Machine Learning.
References
[1] Dimitri P. Bertsekas and Steven E. Shreve, Stochastic optimal control: the discrete time case. 2007.
[2] Bickford Smith, Kirsch, Farquhar, Gal, Foster, and Rainforth. Prediction-oriented Bayesian active learning. International Conference on Artificial Intelligence and Statistics, 2023.
[3] Paul C. Bressloff. Stochastic processes in cell biology. Interdisciplinary Applied Mathematics, 2021.
[4] Wessel Bruinsma, James Requeima, Andrew YK Foong, Jonathan Gordon, and Richard E Turner. The gaussian neural process. In Third Symposium on Advances in Approximate Bayesian Inference, 2021.
[5] Tu Dinh Bui, Dinh Tran, and Richard E Turner. Student-t processes alternatives to gaussian processes, In International Conference on Machine Learning, pages 1232-1241, 2016.
[6] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
[7] Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical science, pages 273-304, 1995.
[8] Shing Chan and Ahmed H Elsheikh. Parametric generation of conditional geological realizations using generative neural networks. Computational Geosciences, 23(5):925-955 2019.
[9] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[10] Muyang Chung and Richard E Turner. Non-gaussian process regression with student-t processes.
In International Conference Machine Learning, pages 4607-4616, 2019.
[11] Rob Corishh, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International conference on machine learning, pages 2133-2143 PMLR, 2020.
[12] Noel Cressie. The origins of kriging. Mathematical geology, 22(3):239-25 222, 1990.
[13] Andreas Damianou and Neil D Lawrence. Deep gaussian processes. In Artificial intelligence and statistics, pages 207-215. PMLR, 2013.
[14] Emilien Dupont, Tuanfeng Zhang, Peter Tilke, Lin Liang, and William Bailcy. Generating realistic geology conditioned physical measurements with generative adversarial networks. arXiv preprint 1X2v:1802.0306. 2018
[15] Emilien Dupont, Yee Whye Tch, and A. Doucet. Generative models as distributions of functions. In AISTATS, 2022.
[16] Conor Durkan, George Papamakarios, lain Murray, and Jascha Sohl-Dickstein. Neural spline flows. International Conference on Learning Representations, 2019.
[17] Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes, arXiv preprint arXiv:2206.0327 2022.
[18] Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33:8284-8295 2020.

---

[19] Peter Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv: 1807008800.081/1 2018.
[20] Fred N. Fritsch and Judy Butland. method for constructing local monotone piecewise cubic interpolants. Siam Journal on Scientific and Statistical Computing, 5:300-304, 1984.
[21] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pages 1704-1713, PMLR, 2018.
[22] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Jimenez Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. ArXiv, abs/1807 .01622, 2018.
[23] Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
[24] Zoubin Ghahramani. A tutorial on gaussian processes. Summer School on Machine Learning. 6:67-80, 1999.
[25] Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. In International Conference on Learning Representations, 2019.
[26] Alan Häjek. Dutch book arguments. In The Handbook of Rational and Social Choice, 2009.
[27] Houlsby, Huszár, Ghahramani, and Lengyel. Bayesian active learning for classification and preference learning. arXiv, 2011.
[28] Harry Joe. Multivariate models and dependence concepts. Chapman & Hall/CRC, 1997.
[29] Makoto Kawano, Wataru Kumagai, Akiyoshi Sannai, Yusuke Iwasawa, and Yutaka Matsuo. Group equivariant conditional neural processes. In International Conference on Learning Representations, 2020.
[30] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Tch. Attentive neural processes. In International Conference Learning Representations, 2018
[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
[32] Iryna Korshunova, Jonas Degrave, Ferenc Huszár, Yarin Gal, Arthur Gretton, and Joni Dambre. Bruno: deep recurrett model for exchangeable data. Advances in Neural Information Processing Systems, 31, 2018.
[33] Iryna Korshunova, Yarin Gal, Arthur Gretton, and Joni Dambre. Conditional bruno: A neural process for exchangeable labelled data. Neurocomputing, 416:305-309, 2020. ISSN 0925-2312. doi: http://dbb.or///.1111666..eeeeee 201.................0.... URL https: com/ hom/sccince/// htt.///c/arrccceee///////2222220
[34] Carlo R. Laing and Gabriel J. Lord. Stochastic methods in neuroscience, 2009.
[35] John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits, Advances in neural information processing systems, 20(1):96-1, 2007.
[36] Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention- permutation-invariaat neural networks. In International conference on machine learning, pages 3744-3753. PMLR, 2019.
[37] Xuechen Li, Ricky Tian Qi Chen, Ting Tim Leonard Wong, and David Duvenaud. Scalable gradients for stochastic differential equations. In Artificial Intelligence and Statistics, 2020.
[38] Juan Maroñas, Oliver Hamelijnck, Jeremias Knoblauch, and Theodoros Damoulas. Trans- forming gaussian processes with normalizing flows. In International Conference on Artificial Intelligence and Statistics, pages 1081-1089. PMLR, 2021.
12

---

[39] Emile Mathieu, Adam Foster, and Yee Whye Teh. On contrastive representations of stochastic processes. In NewrIPS, 2021.
[40] Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. J. Mach. Learn. Res., 21(132):1-62, 2020.
[41] Roger B Nelsen. An introduction to copulas. Springer Science & Business Media, 2007.
[42] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. International Conference on Machine Learning, pages 16569-16594, PMLR, 2022.
[43] George Papamakarios, lain Murray, Matthew Gorham, and Jascha Sohl-Dickstein. Normalizing flows for probabilistic modeling and inference. arXiv preprint 1011v:19/..02766, 2019.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances neural information processing systems, 32, 2019.
[45] Wolfgang Paul and Jörg Baschnagel. Stochastic processes: From physics finance. 2000.
[46] Tom Rainforth, Adam Foster, Desi R Ivanova, and Freddie Bickford Smith. Modern bayesian experimental design. arXiv preprint arXiv:2302. 14545, 2023.
[47] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. In Adaptive computation and machine learning, 2009.
[48] Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. MIT press, 2006.
[49] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[50] Carlos Riquelme, George Tucker, and Jasper Snock. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint /8X 2018.
[51] Settles. Active learning literature survey. Technical report, University of Wisconsin, Madison, 2009.
[52] Amar Shah, Andrew Wilson, and Zoubin Ghahramani. Student-t Processes as Alternatives to Gaussian Processes. In Samuel Kaski and Jukka Corander, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, volume 33 of Proceedings of Machine Learning Research, pages 877-885, Reykjavik, Iceland, 22-25 Apr 2014. PMLR.
[53] Sebastien Strebelle. Conditional simulation of complex geological structures using multiple- point statisties. Mathematical geology, 34(1):1-21, 2002.
[54] Zoltán Sylvester, Paul Durkin, and Jacob A Covault. High curvatures drive river meandering. Geology, 4763):263-266, 2019.
[55] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Keill, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[56] Yee Tch, Michael Jordan, Matthew Beal, and David Blei. Sharing clusters among related groups: Hierarchical dirichlet processes. Advances in neural information processing systems, 17, 2004.
[57] Nico G. van Kampen and William P. Reinhardt. Stochastie processes in physics and chemistry. 1981.
13

---

[58] Andrew G Wilson and Zoubin Ghahramani. Copula processes. Advances in Neural Information Processing Systems, 23, 2010.
[59] Andrew Gordon Wilson, Matthew Hoffman, Cheng Wang, Eric Fox, and Zoubin Ghahramani. Deep gaussian processes with stochastic backpropagation. In International Conference Machine Learning, pages 1436-1444 1445, 2016.
[60] Jin Xu, Jean-Francois Ton, Hyunjik Kim, Adam Kosiorek, and Yee Whye Teh. Metafun: Meta- learning with iterative functional updates. In International Conference Machine Learning. pages 10617-10627. PMLR, 2020.
[61] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[62] Tuan-Feng Zhang, Peter Tilke, Emilien Dupont, Ling-Chen Zhu, Lin Liang, and William Bailey, Generating geologically realistic 3d reservoir facies models using deep learning of sedimentary architecture with generative adversarial networks. Petroleum Science, 16(3):541-549 2019.
[63] Tuanfeng Zhang, Paul Switzer, and Andre Journel. Filter-based classification of training image patterns for spatial simulation. Mathematical Geology, 38(1))63-80, 2006.
14

---

Appendix of Deep Stochastic Processes via Functional Markov Transition Operator
A Proofs
A.1 Proof of proposition 4.1 (See page 4)
Proposition 4.1. Let Fo(:;r,2) y + y denote an invertible transformation between outputs,
parameterized by the input and latent. If (P
(0)) ( is a valid Stochastic Process (SP) (i.e.
satisfies Equation (1) and Equation (2)) and
Vim (P (Vim) Jains 2~ pop) y/1 Foly) (0) Jzs z).
(0)
(6)
then {PZ... (y1:)) Jay. is also a valid SP.
Proof. Given an imput 2, and latent variable 2, Yi is obtained by applying an invertible transforma- tion Fg to Ui (0) Therefore, we can represent p(3 y/ (0), 24 2) using the Dirac delta:
p(y wi ;, =) = ö(y Fo(y; (0) Fi, =))
(16)
Furthermore, according to Equation (6), these transformations are independently applied given the latent variable 2. So we have:
p(ynn Win Ul:n 2) 5(ys Fo(v: (0)
x4, =))
(17)
Hence
Poim (8/mm wind
pe (e)pp((imm) 11o0 2in. 2)d2
(18)
/ pa (2) II 5(y; - Po :x, ;x,, 2))
(19)
As per Proposition 4.3, {Pe... (11mm ((hh mm is exchangeable and consistent. Furthermore, because 22... is valid SP, (Pa.... (f/mm) 2.... also a valid SP
(Pa... )2..
A.2 Proof of proposition 4.2 (See page 4)
2z is consistent and exchangeable (as per Equations (1) and (2)) and the collection of marginal Markov transition operators (MMTOs) {pru. (y1mm (is)) 32.. is also consistent and exchangeable (as per Equations (7)
Proposition 4.2. If the collection of marginals before transition (p.... (0)) )
(0)
and (8)), then the collection of marginals after transition {Poim (Him) is also consistent and exchangeable, hence defining a valid sP.
Proof. According to the definitions of consistency and exchangeability for both the collection of marginals (Px,.. (1))))))))))))))))))))) and the collection of transition operators {P (V1.n /V/ in 2 have the following (Equations (1), (2), (7) and (8)):
(0)
(0) f P.... (Vimn dr m- l:n Pai (V1.
(0)
( = Pri. (v
(0)
I Pai.. (Vi ( dymm- 1:m Pri... (11.m (0)
Prim (Vimm (in = Pa(znn)) (1i (T ()
15

---

where 1 < m < n.
The Markov transition the function outputs described by:
(0) / Pri. (wi (aln)rrr... (y (y Jdy (0)
Prim (yim))
For any finite sequence 2im we have Pei:n (y1m Idym+
f
/ (ʃ Pei. ( (11.0 dd (0) dym++in
f Prim (Vim) (ʃ
(Seen Pen. (dmml dyii.
(0)
(0)
f Prism (wion Par... (11:m Wi Jdy (0)
(0)
/
[f Pe (wi Jyy 1:m Prim (1)
f Prism (Vim) Pamm (V1:m l Vi.m )dy
f Prim (Vim)dyl..
(20)
Furthermore, we have
/ Pr(ain) (x (0) Pr(ee.)) ( (0)
Pr(eL)) (z(im))
f Pri. (vin Par (y1.m ly (0) dy
(0)
Par. (W1:n)
(21)
Therefore, the collection of marginals {Pai.. (31 (iim))m.. Jm are both consistent and exchangeable (Equa- tions (20) and (21)), hence defining valid SP.
A.3 Proof of proposition 4.3 (See page
Proposition 4.3. MMTOs in the form of Equation (9) are consistent and exchangeable.
Proof. Recall that MMTOs in Equation (9) write
Pei. (vmm Vien
(0)
f po(2))
II
Fo(u; (0) ; 2)) dz,
(22)
a special case of more general form: 1=1
/ (0(2) II po (vt v. 20,
Pen. (1imm Vi.n: 0)
22,,2))0 2)d2,
(23)
Equation (23) becomes Equation (22) when po(w 1y 24, 2) 5(v, Fo(y! 24, xx,2)) 2)) is 8-
distribution. Below we prove that MMTOs are consistent and exchangeable for the general form. We have
Pai.. (Vim (0); 0) dymmtin
[( (o(=)) II Po (y Vs x,, =) dym+1:m
= - po II
m
pe (y:
yi (0)
Its 2)
II i=++1
Po(y: Bi (0) 24 2)dym+1m) dz
i=1
-f po(2)) I
m
Po (y:
wy (0)
mee 2) dz
(0) Parim (21.m 1iimm 0)
(24)
16

---

and
x(m²
Pa(ma (T(V1)) = po(z)
II (=((1)
pa (yi 3/ 24, 2)d2
f po(2)
po (ys 1 3, 2)dz
Prin (21mm
(25)
Therefore, the collection (Pr(an...) (1 (11())))) 0) Iei. is both consistent (Equation (24)) and exchangeable (Equation (25)).
A.4 Derivation of Markov Neural Process marginal densities
T step Markov Neural Processes (MNPS) have marginal densities in the following form (as in Equation (10)):
T T
Prim (M 0) = / (o(2 )
Par (1im)
IIII
(Ui
(1-1)
Ni; = 21)) d'))d 1) dz:13)
= - =footaaio (o(2(1:1)) Prin
aF (y; ,
(1-1), Iity z(1))
d2 (2:1:T)
(11m))
det
Oy, (1-1)
Proof. Given the sequence of latent variables 2(1m)) our model becomes normalising flows on the finite sequence of function outputs Vim. with a prior distribution Pri.. (vim). and the invertible transformation at each step Fl(() (1)). According Papamakarios et al. [43], Rezende and
Mohamed [49], we have
Pri. (11m0/////1122/000/1 (1:T)); Pmim (Vim (2 (1:T), 0)
OF(T) (v.
(T-1) xii , ({))
Par (Vim
det
Oy (1-1)
n
OF
(v) (t (t-1), xz 2(1))
(0)
Prim
(Vie))
det
Oy: (1-1) 1)
(=1=)
The marginal density Pan. (yimi f) can computed as:
Pri... (y1:n; 0) =) = / PEE. (y)ml (1:T); (1:7)
OF(1) (v. (t-1), xz, 2(1))
/ po(=(1:T:00)p)) 00)pz (v)))
III
det
de (1:T)
(26)
Oy." 1)
B Implementation details
B.1 Data
B.1.1 1D synthetic data
Gaussian Process (GP) samples Three ID function datasets were created, each comprising samples from Gaussian processes (GPs) with different kernel functions: RBF (length scale 0.25), Matern-2.5 (length scale 0.5), and Sxp-Sine-Squared (length scale 0.5 and periodicity 0.5). Observation noise
17

---

variances were set at 0.0001 for RBF and Matern kernels, and 0.001 for the Exp-Sine-Squae Sine-Squared kernel. Function inputs spanned from -2. 2.0. To accelerate sampling, identical input locations were employed for every 20 function instances. For this dataset, context size varies randomly from 2 to 50.
Monotonic functions Our generation of monotonic functions starts by sampling N ~ Poissonn5500 (5.0) to determine the number of interpolation nodes. We then sample N + increments Xinerements sampled from Dirichlet distribution. These increments are increased by 0.01 to avoid excessively small values, and are then normalized such that their sum is 4.0. The final X values for interpolation nodes are obtained by adding -2.0 to the cumulative sum of these increments so that these X values are within the range 2-2.0, 2.0]. For each X value, a corresponding Y value is sampled from a Gamma distribution Y ~ Gamma 2, 1). The cumulative sum of Y values ensures monotonicity. A PCHIP interpolator [20] is then created using these interpolation nodes (X and Y values) to generate function outputs. Given the functions, we randomly sample 128 X values and compute their corresponding function values. Note that these X values are now used to evaluate the functions, rather than serving as interpolation nodes. The function values are normalized to the range [-1.0, 1.0]. Finally, Gaussian observation noise with standard deviation of 0.01 is added to these function values. For this dataset, context size varies randomly from 2 to 20.
Convex functions To create a dataset of convex functions, we compute integrals of the monotonic functions previously created. These convex functions are then randomly shifted and rescaled to increase diversity. The function values are normalized to the range [-1.0, 1.0). Finally, Gaussian observation noise with a standard deviation of 0.01 is added to these function values. Context sizes varied randomly from 2 to 20.
Stochastic differential equations samples We create dataset of 1D functions, each of which represents a solution to a Stochastic Differential Equation (SDE). This SDE is defined by the drift function fax, t) -a + x 6²). (1 x²) and the diffusion function g( t) b- (1 1---- =²). with constants and b both set to 0.1. The function sets up a time span that includes 128 uniformly distributed points within the range of [-5.0, 5.0). We then uniformly sample an initial condition, Top between 0.2 and 0.6. We use the sdeint. stratKP2iS function from the sdeint library to generate a solution to the SDE. This solution forms ID function that depicts trajectory of the SDE across the defined time span, originating from the initial condition To- Lastly, we randomly alter the context sizes between 2 and 50.
For all the aforementioned datasets, we use the following set sizes: 50000 for the training set, 5000 for the validation set, and 5000 for the test set.
B.1.2 Geological data
We generate the GeoFluvial dataset using the meanderpy [54] package. We first run simulations using the numerical model of meandering in meanderpy with default parameters except for channel depth which we change from 16m to 6m. The resulting simulations correspond to images of shape (800, 4000). We then extract 3 random non-overlapping crops of shape (700, 700) from these images, which are resized to (128, 128) and are used as data. We ran [25000/3] simulations, resulting in a dataset of 25,000 images.
B.2 Model architectures and hyperparameters
Permutation equivariant/ivvaiiant neural networks on sets We implement two versions of neural modules which operate on sets, both of which preserve the permutation symmetry of the sets. They are known as deep sets [61] and set transformers [36]. To obtain an invariant representation, we used sum pooling for deep sets, and pooling by multi-head attention (PMA) [36] for set transformers. Our experiments primarily employed set transformers, chosen for their stronger ability to model interactions between datapoints. However, for the wheel contextual bandit experiments, the context set often expanded to tens of thousands. To circumvent memory issues, we use deep sets in this instance. For set transformers, we stack two layers of set attention blocks (SABs) with a hidden dimension of 64 and 4 heads. This is followed by single layer of PMA, which was subsequently followed by a linear map. the case of using deep sets, we use a shared instance-wiss Multi-Layer Perceptron (MLP) that has two hidden layers and a hidden dimension of 64. This MLP processes the concatenation of function inputs and outputs. Following this, we add sum aggregation which is then succeeded by single-layer MLP with ReLU (Rectified Linear Unit) activation.
18

---

Conditional normalising flows The instance-wise invertible transformation F/8 at each time step I is parameterised as a rational quadratic spline flow [16]. Note that we do not share parameters among iteration. To condition on an input 2; and a latent variable 2, we use a Multi-Layer Perceptron (MLP) which takes in 24, 2 and produces the parameters for configuring a one-layer spline flow with 10 bins.
Multi-Layer Perceptrons (MLPs) Except for the deep sets, we use MLPs to parameterise continuous functions in two places. Firstly, we use as conditioning component in conditional normalising flows F/ (as we mentioned above). It has two hidden layers and hidden dimension
of 128. Secondly, we use to parameterise ( and (f) o in the inference model (see Section 4.3).
It has two hidden layers and a hidden dimension of 64.
We adopt the same pretraining approach as Garnelo et al. [22] for the contextual bandits problem, pretraining the model on 64 wheel problems {6, Jam where 8; ~ (4 1). Each wheel 6; has context size ranging from 10 to 512 and a target size varying between 1 and 50. Here each data point tuple (X, r). Because the context size can grow tens of thousands during test time, for computational efficiency, we use deep sets to implement SETENCODER and a linear flow rather than
spline flow to parameterise F
For optimisation, we use Adam [31] optimiser with learning rate of 0.0001. We use a batch size of 100 for ID synthetic data and a batch size of 20 for the geological data. In experiments, we find that is often beneficial to encode x; with Fourier features [55], and we use 80 frequencies randomly sampled from a standard normal.
For more details, please also see our reference implementation at https: htaps:://githbb... htatthu..coom/3.. pom/jinnx066 mnp.
B.3 Computational costs and resources
In our current implementation.e the invertible transformations F/ in the generative model and the (9) (t) in the inference model do not share parameters across iterations. mean/variance function 1o &
However, we do share the SETODRD across iterations. Consequently, with our MNP, both memory usage and computing time increase linearly with the number of steps. If the SETENCODER employs set transformers, the computational cost becomes 0(m²), where m stands for the context attention blocks (SABs) with
size. This cost, however, can be decreased to (mk) by replacing induced set attention blocks (ISABs), where & denotes the number of inducing points. If deep sets are used to implement SETENCODER, the computational cost reduces to ((m). despite the inference model being less expressive. The computationnl ad advantage of using deep sets becomes prominent when we have large number of observed datapoints in the context. For instance, in the contextual bandits experiments (in Section 6.2). our model needs to process 80K datapoints, which is intractable for a transformer-basse NP on a single GPU.
To further enhance scalability, one might consider sharing some parameters across transition steps. Additionally, in the case of space complexity, as discussed in Section 7, we can explore continuous- time Markov transitions, i.e., stochastic differential equations in function space. Such an approach would potentially require less memory by leveraging adjoint methods.
Training MNP is indeed resource-intensive; however, inference in MNP simply requires a forward pass. On single GeForee GTX 1080 GPU card, a standard 7-step MNP takes approximately one day to train for 2004 steps on ID functions. If we use 20 latent samples to evaluate the marginal log-likelihood using the IWAE objective [6], inference runs typically in few seconds for batch of 100 functions.
C Ablation studies
The better performance of MNPs compared to standard Neural Processes (NPs) can roughly be attributed to two reasons: Firstly, instead of directly constructing the SP as in NPs, MNPs iteratively transform from a trivial SP gradually to a more expressive one. Secondly, MNPS employ more expressive invertible transformations on function outputs, parameterized by normalizing flows. To verify that multiple steps are indeed advantageous, we evaluated various models based on their estimated marginal log- likellioooo GP data using an RBF kernel. The results are follows:
19

---

MNP, step, spline flow: 0.941
MNP, steps, spline flow: 1.950
MNP, steps, spline flow: 2.266
MNP, step, spline flow, latent dimension x2: 0.850
MNP, step, spline flow, latent dimension x2, MLP hidden dimension x2: 1.476
MNP, step, linear transformation: 0.990
MNP, steps, linear transformation: 1.977
MNP, steps, linear transformation: 2.140
From these results, observe that performance generally improves with an increase in the number of steps. Using higher-dimensional latent variables or wider neural networks in models with only one step doesn't achieve the performance seen in models with multiple steps. Furthermore, replacing the spline flow with a linear transformation (keeping mind that stacking multiple linear transformations remains linear) still showed consistent performance improvement with increase in steps.
D Broader impacts
Our work presents MNPs, novel approach to construct SP using neural parameterised Markov transition operators. The broader impacts of this work have many aspects. Firstly, the proposed models are more flexible and expressive than traditional SP models and NPs, enabling them to handle more complex patterns that arise in many applications. Moreover, the exchangcability and consistency in MNPs could improve the robustness and reliability of neural SP models. This could lead to more trustworthy systems, which is critical aspect high-stakes applications. However, with any machine learning system, there potential risks. For example, the complexity of these models could exacerbate issues related to interpretability and transparency, making it more difficult for humans to understand and control their behaviour.
20


Reviews:
[{"review_id": "l2yz5ACLqs", "reviewer": "Reviewer_ipFc", "summary": "The paper introduces Markov Neural Processes (MNPs), a new variant of Stochastic Processes (SP). The paper points out that predictive SP models that construct context-to-target mappings no longer satisfy conditional consistency property and also are not fully generative. An expressive NP variant called MNP is proposed, the basis of which is stacking sequences of Markov transition operators in the function space. The paper adapts the definitions of consistency and exchangeability to MNPs, and show that they are consistent and exchangeable by construction. The paper shows the potential of MNPs in a variety of benchmarks from 1D function regression to geological inference.\n", "questions": "Figure 2: I could not understand clearly why the auxiliary latent variables should be inferred in the reverse order. Can you expand further?\n\nThe paper uses a specific variant of normalizing flows in its experiments called spline flows. It would be useful to justify the usage of the specific variant of flows. More flexible variants such as deep sigmoidal flows can also be tried out.\n", "limitations": "The training of MNPs is computationally intensive as the authors state; this limitation can be addressed in future works.\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper introduces the framework from a very generic point of view, and also provides context in the specific choices made for the experiments. For instance, the usage of instance-wise conditional normalizing flows to parameterize the MNP is very interesting; as flows are universal approximators, this gives the MNPs high flexibility. Similarly, the usage of permutation-invariant neural networks is also interesting.\n\nThe idea is simple but powerful, and the exposition of the idea is very neat.\n", "weaknesses": "The empirical evaluation of the paper is somewhat limited. More empirical results may attract readers to this paper - for instance, few-shot image classification results would be interesting to see. Results in more large-scale datasets will also be useful.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their feedback on our manuscript. Below are our responses to the questions raised.\n\n> *“Figure 2: I could not understand clearly why the auxiliary latent variables should be inferred in the reverse order. Can you expand further?”*\n\nThe auxiliary latent variables are inferred in reverse order primarily to enable parameter sharing between the generative model and the inference model. During inference, function outputs $y_{1:n}^{(T)}$ are observed. Upon inferring $z^{(T)}$ first, we can directly compute $y_{1:n}^{(T-1)}$ by inverting the transformation $F_{\\theta}^{(T)}$ which is already learned in the generative model. Recursively, for other steps, $F_{\\theta}^{(t)}$ can be reused in the inference model in a similar way. This design was initially proposed by [1] and is a critical aspect that contributes to the effectiveness of their methods.\n\nMoreover, it should be noted that the prior over auxiliary latents $z^{(1:T)}$ can be specified in an arbitrary order. Therefore, we can express $p(z^{(1:T)})$ as $p(z^{(1:T)})=p(z^{(T)})p(z^{(T-1)} | z^{(T)}) \\cdots p(z^{(1)} | z^{(2:T)})$. This concept is illustrated in Figure 2(a) (which is updated and uploaded in the 1-page PDF above); specifically in the direction of the arrows for the latent variables. As a result, the inferred posterior over the latent variables can actually have the same factorization as the prior over these latent variables.\n\n[1] Cornish, Rob, et al. \"Relaxing bijectivity constraints with continuously indexed normalising flows.\" International conference on machine learning. PMLR, 2020.\n\n> *“The paper uses a specific variant of normalizing flows in its experiments called spline flows. It would be useful to justify the usage of the specific variant of flows. More flexible variants such as deep sigmoidal flows can also be tried out.”*\n\n\nApplying different normalizing flows to our model would indeed be very interesting. We did not exhaustively optimize our model and it may be possible that other normalizing flows perform better. Our main motivation for using spline flows is that they are very expressive and flexible even on low-dimensional spaces (including in 1D) - as our normalizing flows operate on the space of $\\mathcal{Y}$ which is often low-dimensional, this made spline flows a simple yet effective choice for our purposes. Compared to spline flows, many advanced normalizing flows excel particularly in high-dimensional domains. That said, more suited and powerful alternatives to spline flows may indeed exist and it would be interesting to explore this further in future work.\n"}, {"review_id": "96t1KzVljU", "reviewer": "Reviewer_nP7k", "summary": "The paper presents Markov Neural Processes (MNPs), a novel class of Stochastic Processes (SPs) that leverage neural networks for data modeling. MNPs enhance the flexibility and expressivity of the Neural Processes (NPs) framework without sacrificing consistency or imposing restrictions. The proposed iterative construction is fully generative and maintains consistency under conditioning. MNPs demonstrate superior performance over baseline models across various tasks.\n\nThe primary contribution is the MNP model, which employs a neural process as the transition operator, thereby adding flexibility to the stochastic process. This method increases expressivity while preserving the fully generative nature and consistency of NPs. The conditional consistency defined in this paper plays a crucial role in preventing uncertainty mismatches.", "questions": "How is the scalability of this method? Were any tricks used to make the method more scalable? Can you provide an analysis on the time/space complexity?", "limitations": "Yes. ", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "he model is theoretically proven to satisfy the valid conditions. The experiments are well-designed to demonstrate the model's better expressivity, advantage over traditional models on non-Gaussian data, and better performance on scientific problems. The method has high theoretical novelty, especially in the use of Markov operators that lifts the Gaussian restrictions of the previous models.", "weaknesses": "The experiments are run on a few small-scale synthetic datasets. The model's performance would be better evaluated if there are experiments on a larger variety of, especially large real-world, datasets. Besides, experiments that evaluates the time consumption among the models would be helpful. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the time and effort the reviewer has dedicated to evaluating our paper. Please see below for our responses to the questions raised.\n\n> *“How is the scalability of this method? Were any tricks used to make the method more scalable? Can you provide an analysis on the time/space complexity?”*\n\nIf we employ deep sets to parameterize the permutation-invariant set encoders in the inference model (line 176), MNPs with $T$ steps are roughly $T$ times more expensive than the original neural processes in terms of both time and space complexity. However, it's important to note that MNPs still represent a scalable and expressive model compared to attention-based models such as ANPs and transformer neural processes: Our model scales linearly with respect to the context size. In contrast, for transformer-based NP architectures, the cost is quadratic in the context size. This advantage becomes prominent when we have a large number of observed datapoints in the context. For instance, in the contextual bandits experiments (Sec 6.2), our model needs to process 80K datapoints, which is intractable for a transformer-based NP on a single GPU.\n\nTo further enhance scalability, one might consider sharing some parameters across transition steps. Additionally, in the case of space complexity, as discussed in Section 7, we can explore continuous-time Markov transitions, i.e., stochastic differential equations in function space. Such an approach would potentially require less memory by leveraging adjoint methods.\n\nWhen making predictions with trained $T$-step MNPs, the time complexity with deep sets parameterized set encoders stands at $\\mathcal{O}(BT(N_{\\mathcal{C}}+N_{\\mathcal{T}}))$ where $B$ is batch size, $N_{\\mathcal{C}}$ is context size (the observed number of datapoints) and $N_{\\mathcal{T}}$ is target size (the predicted number of datapoints). In contrast, with set transformers parameterized set encoders, it is $\\mathcal{O}(BT(N_{\\mathcal{C}}+N_{\\mathcal{T}})+BN_{\\mathcal{C}}^2)$. Note that the set transformer is used to compute a representation shared across steps so there is no multiplier of $T$ for the quadratic term. The space complexity follows a similar pattern: $\\mathcal{O}(BT(N_{\\mathcal{C}}+N_{\\mathcal{T}}))$ for deep sets and $\\mathcal{O}(BT(N_{\\mathcal{C}}+N_{\\mathcal{T}})+BN_{\\mathcal{C}}^2)$ for set transformers. It's worth noting that in this analysis, we've excluded the complexity related to the depth and width of each MLP component for simplicity.\n"}, {"review_id": "z2tTq9fn1c", "reviewer": "Reviewer_kRke", "summary": "The paper attempts to improve the flexibility of Neural Processes without sacrificing exchangeability and consistency properties of classical stochastic processes. The paper proposes a diffusion-like iterative refinement of the predicted values, however, ensuring exchangeability and consistency.", "questions": "1. I feel Eq. (11) is somehow incomplete considering that $y_i^{(t)}$ (for $t=0, \\ldots, T-1$) are also random variables that do not appear here. Are they absorbed in the reconstruction term (i.e., the first term)?\n2. Is there an empirical way to test how big of a problem marginal consistency poses, for instance, by measuring KL between LHS and RHS of Eq. (2). What I am getting at is: Is marginal consistency really a big issue empirically speaking, especially considering the flexibility of the state of the art models like TNPs or NDPs.\n3. The paper does not cite or discuss Transformer Neural Processes (TNPs). Perhaps one may also compare with Neural Diffusion Processes (NDPs). It may help position the performance of MNPs even if it cannot outperform these considering that these models relax the consistency requirement.\n4. One aspect I think is that MNPs do not require pairwise interaction between the query points required by the transformer-based NP architectures. I wonder if one might expect some computational benefits of MNPs with respect to this class of models.\n5. It would be nice to provide a derivation of the second line of equation (10).", "limitations": "Yes, the paper discusses the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Theoretically ensures exchangeability and consistency.\n- Performs better than the main baseline in this category i.e., NPs.", "weaknesses": "I discuss this in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their detailed comments. Below are our responses:\n\n> *“I feel Eq. (11) is somehow incomplete considering that y_i^{(t)} (for t=0,..., T-1) are also random variables that do not appear here. Are they absorbed in the reconstruction term (i.e., the first term.”*\n\nThanks for pointing out the confusion here. Note that our model is intentionally constructed such that, while $y_{1:n}^{(t)}$ are indeed random variables, they become deterministic functions of $y_{1:n}^{(t-1)}$ once the corresponding latent variables $z^{(t)}$ are provided (see Eq. (9)). This construction ensures that while the Markov transitions from $y_{1:n}^{(t-1)}$ to $y_{1:n}^{(t)}$ are stochastic, all the stochasticity is encapsulated in the latent variables $z^{(t)}$. Consequently, there is no need to integrate over $y_{1:n}^{(t-1)}$ in Eq. 11. This design choice is crucial in making our model practical by requiring integration only over low-dimensional latent variables with variational inference.\n\nIn Eq. (11), when the sequence of auxiliary latent variables $z^{(1:T)}$ is provided, the sequence of transition steps from $y_{1:n}^{(0)}$ to $y_{1:n}^{(T)}$ become deterministic. Therefore, $p(y_{m+1:n} | z^{(1:T)})$ can be directly computed with a change of variables, in the same way as normalizing flows. The primary difference is that now the invertible transformations are applied to a set rather than a single vector. Hence, the intermediate outputs transitioning from $y_{1:n}^{(t-1)}$ to $y_{1:n}^{(t)}$ are indeed absorbed in the first term, as they become deterministic variables once all the latent variables $z^{(1:T)}$ are provided.\n\nTo provide a better illustration of the generative process, we also updated Figure 2 (Please see the uploaded PDF above).\n\n> *“Is marginal consistency really a big issue empirically speaking, especially considering the flexibility of the state-of-the-art models like TNPs or NDPs.”*\n\nThere are several practical reasons why marginal consistency may be desirable - we highlight some of these below.\n\n* Data Efficiency: Models lacking consistency guarantees, such as TNPs and NDPs, require direct learning of consistent properties from the data, potentially making these models more data-hungry than consistent SP models.\n* Out-of-Distribution Generalization: Without consistency guarantees, model properties become entirely dependent on the training data distribution. As an example, if the model consistently encounters target sizes greater than $M$ during training, its performance is likely to deteriorate when faced with tasks where the target size is less than $M$. Conversely, models maintaining marginal consistency are expected to naturally generalize to smaller target sizes.\n* Predictive Reliability: When using MNPs for predictions, there's an assurance that the predictions won’t contradict each other. However, such reliability cannot be guaranteed with TNPs and NDPs.\n\nWe are planning to verify the aforementioned conjectures via experiments, but we may not be able to finish them within the rebuttal period.\n\nGenerally speaking, the choice of model largely depends on the application. If the sole objective is to attain the best possible fit to the data, then models without consistency or exchangeability properties, such as TNPs and NDPs, may be more appropriate. However, if the goal is to construct a model that offers better generalizability, interpretability, and robustness, models that have both exchangeability and (marginal and conditional) consistency properties might be preferred.\n\n> *“The paper does not cite or discuss Transformer Neural Processes (TNPs). Perhaps one may also compare with Neural Diffusion Processes (NDPs).”*\n\nThanks for providing this reference. We will cite and discuss TNPs in our related work section. We will also discuss NDPs in more detail in the related work section.\n\n> *“One aspect I think is that MNPs do not require pairwise interaction between the query points required by the transformer-based NP architectures. I wonder if one might expect some computational benefits of MNPs with respect to this class of models.”*\n\nThe reviewer's observation is entirely accurate. When we use deep sets to parameterize our permutation-invariant set encoders in the inference model (line 176), our model scales linearly with respect to the context size. In contrast, for transformer-based NP architectures, the cost is quadratic in the context size. This advantage becomes prominent when we have a large number of observed datapoints in the context. For instance, in the contextual bandits experiments (Sec 6.2), our model needs to process 80K datapoints, which is intractable for a transformer-based NP on a single GPU.\n\n> *“It would be nice to provide a derivation of the second line of equation (10).”*\n\nThe detailed derivation for equation 10 can be found in Appendix A.4.\n\nIn particular, from the first to the second line of Equation 10, it's crucial to recognize that while both $y_{1:n}^{0:T-1}$ and $z^{(1:T)}$ are random variables, once the latent variables $z^{(1:T)}$ and the initial states of the function outputs $y_{1:n}^{0}$ are conditioned, $y_{1:n}^{1:T-1}$ becomes deterministic (see Equation 9). For deterministic transformations, the resulting density functions can be computed as the product of the original density functions and the determinant of the inverse Jacobian of the transformations, and we do not need to integrate over intermediate variables.\n"}, {"review_id": "uKzXS90MVe", "reviewer": "Reviewer_wA6N", "summary": "This paper proposes Markov neural process, a new class of stochastic processes. The work is based on prior art on neural process, which was published in 2018. The paper is reasonably written. The effectiveness of the new method is very clear from the experimentations. ", "questions": "1. please elaborate the statement \"the exchangeability and consistency in MNPs could improve the robustness and reliability of neural SP model\".", "limitations": "No potential negative societal impact. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed markov neural process adds more expressiveness to the traditional neural process, which potentially can open the door to new applications;\n2. The theoretical analysis appears sounds\n2. The proposed method appears to be competitive when compared to the baselines.", "weaknesses": "1. The explanation of the overall MNP is hard to follow. Fig 2 is crucial to explain the overall structure, plus the training and inference flow. But it's unclear. For example, the three type of the arrows in Fig 2(b) aren't explained well.\n2. The experiments are impressive in the tables. But the manuscript didn't provide any intuitive explanation why MNP works better. In the contextual bandit example, MNP shows significant improvement over NP when $\\delta = 0.99$. Some explanation the reason or an ablation study with respect to the number of markov stages is needed for the readers to get a better feeling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for dedicating their time to evaluating our manuscript. Below are our responses:\n\n> *“The explanation of the overall MNP is hard to follow. Fig 2 is crucial to explain the overall structure, plus the training and inference flow. But it's unclear. For example, the three type of the arrows in Fig 2(b) aren't explained well.”*\n\nWe have revised Figure 2 and included the updated version in the uploaded 1-page PDF above. We believe that the figure is now much clearer.\n\n> *“The experiments are impressive in the tables. But the manuscript didn't provide any intuitive explanation why MNP works better.”*\n\nThe better performance of MNPs can be attributed to the following reasons:\n* Instead of directly constructing the SP as in NPs, MNPs iteratively transform from a trivial SP gradually to a more expressive one.\n* MNPs employ more expressive invertible transformations on function outputs, parameterized by normalizing flows.\n\nThese two aspects correspond to the extensions mentioned in Section 4.1, transitioning from NPs to a more generalized form, which serves as the foundation for MNPs. We will include these discussions in the experimental section.\n\n> *“Some explanation the reason or an ablation study with respect to the number of markov stages is needed for the readers to get a better feeling”*\n\nTo address this point, we ran a simple ablation study. We compared MNPs with different numbers of Markov transitions in terms of the estimated marginal log-likelihood on GP data with a RBF kernel:\n* MNP (T=1), spline flows, 0.941\n* MNP (T=2), spline flows, 1.950\n* MNP (T=3), spline flows, 2.266\n\nAs the results show, in this setting the performance improves as the number of steps increases. We will run a more extensive ablation study and include the detailed results in the next version of the paper.\n\n> *“Please elaborate the statement \"the exchangeability and consistency in MNPs could improve the robustness and reliability of neural SP model\"*\n\nThis statement appears in the broad impact section in the appendix. While there are many recent developments to construct neural SP models, we highlight that many of these models do not consider conditional consistency, which is problematic, especially for sequential decision-making. There is also the potential for such models to fall foul of the so-called Dutch Book scenarios. Additionally, models such as NDPs do not meet marginal consistency, so the distribution over functions is not properly defined, and conflicting predictions can be made.\n\nMNPs, on the other hand, provide a framework to extend the original NPs and enhance their expressivity without compromising consistency. When using MNPs for predictions, there's a guarantee that the predictions won’t contradict each other and will hence be more reliable. MNPs also enjoy better generalization because of the consistency properties, and are less sensitive to changes to the data such as the observation order in sequential decision-making. In this regard, MNPs contribute to improving the robustness and reliability of neural SP models.\n"}, {"review_id": "g5jrq6Lbz6", "reviewer": "Reviewer_UK93", "summary": "The paper introduces a new class of Bayesian nonparametric  models by extending neural processes. The fundamental idea is to create a Markov chain of stochastic processes, culminating in a flexible enough stochastic process.", "questions": "None", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper introduces a simple but a clever modification of vanilla neural processes. The idea allows for more expressive representation as demonstrated by the experiments. I believe the idea is significant enough for it to be of interest to the community.", "weaknesses": "The paper uses an unclear notation for probabilistic notions. It’s not just that this choice is unconventional, forcing the reader to keep a mental map of how things are redefined, but also that it needlessly complicates exposition. For example, what is $p$? It is ubiquitous in the paper but is never clearly defined. This makes the discussion too hand-wavy — what is $p_{x_{1:n}}(y_{1:n})$? Is it the distribution function of the random vector $(F(x_1), \\ldots, F(x_n))$? If so, what even is $y_{1:n}$? Is the density function of the random vector $(F(x_1), \\ldots, F(x_n))$? What is $f$? Since $\\mathcal X$ is simply the indexing set, and for real-valued random variables we should have $\\mathcal Y = \\mathbb R$, is $f$ simply a real-valued function defined on the indexing set? The notation for transition operators is also very troublesome. My point in raising such questions is that I would have preferred if the authors used standard measure-theoretic notation (it is standard for a good reason) to write much of the paper (I really like the treatment in Ghosal and van der Vaart’s book).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s constructive feedback and have provided our responses below:\n\n> *“The paper uses an unclear notation for probabilistic notions. It’s not just that this choice is unconventional, forcing the reader to keep a mental map of how things are redefined, but also that it needlessly complicates exposition.”*\n\n> *“I would have preferred if the authors used standard measure-theoretic notation (it is standard for a good reason) to write much of the paper (I really like the treatment in Ghosal and van der Vaart’s book).”*\n\nWe appreciate the reviewer's suggestions in terms of notation. While the referenced book indeed offers a nice notation system based on measure theory, we are worried about several potential issues that could arise if we alter our notation:\n* Our current notation aims to align as closely as possible with the original neural process (NP) paper [1]. This system is widely adopted within the NP community, and adopting a different one may cause confusion to some readers.\n* We intentionally write everything using density functions so that it is more accessible to the more application-oriented reader. For example, we denote stochastic processes (SPs) as a collection of density functions $\\\\{p_{x_{1:n}}(y_{1:n})\\\\}\\_{x_{1:n}}$. In this context, '$p$' represents densities over a finite sequence $y_{1:n}$ in the traditional sense. It is indexed by input sequences $x_{1:n}$ because each input sequences correspond to a different density. Furthermore, we express the marginal transition operators $p(y_{x_{1:n}}^{(t)}|y_{x_{1:n}}^{(t-1)})$ as a conditional density function, avoiding the use of operator notations. However, we appreciate that a measure-theoretic notation might be preferable to readers with a more theoretical background.\n\nBelow, we provide a more detailed explanation of our notation, which we hope will improve clarity. We are entirely open to further suggestions on improving our notation while maintaining consistency with [1] and accommodating our non-theoretical audience.\n\n[1] Garnelo, Marta, et al. \"Neural processes.\" arXiv preprint arXiv:1807.01622 (2018).\n\n> *“What is p_{x_{1:n}}(y_{1:n})? Is it the distribution function of the random vector (F(x_1), F(x_2), …, F(x_n))? If so, what even is y_{1:n}? Is the density function of the random vector (F(x_1), F(x_2), …, F(x_n))? ? What is f?”*\n\nEvery $p$ in our paper is a density function in the traditional sense, representing a probability distribution. We use the colon notation to denote a sequence, i.e. $x_{1:n}=[x_1,x_2,\\dots,x_n]$, $y_{1:n}=[y_1,y_2,\\dots,y_n]$ (see Equation 1 in our paper and Equation 1 in the NP paper [1]). We will add a few sentences to our paper to make it clear.\n\n$F$ represents a random function, while $f$ stands for a specific realization of that function. This is analogous to the convention in Euclidean space where a random variable is often symbolized by a capital letter, such as X. The density function, denoted as $p(x)$, describes the probability of X assuming the value of $x$, where $x$ is a specific realization of the variable. In a similar vein, $[F(x_1), F(x_2), \\dots, F(x_n)]$ denotes a sequence of random variables, resulting from evaluating the random function $F$ at respective points $x_i$. The sequence $y_{1:n}=[f(x_1), f(x_2), \\dots, f(x_n)]$ represents the specific realization of this sequence.\n\n> *“Since $\\mathcal{X}$ is simply the indexing set, and for real-valued random variables we should have $\\mathcal{Y}=\\mathbb{R}$, is $f$ simply a real-valued function defined on the indexing set?”*\n\nIn our paper, we treat functions as infinite-dimensional vectors (each element corresponds to a function value at a certain point). A stochastic process then becomes a distribution in the infinite-dimensional space. Through this lens, the outputs of functions are considered random variables, while their inputs serve as the continuous indices for these variables. Essentially, in our context, the term 'indexing set' is synonymous with the function input domain.\n\n> *“The notation for transition operators is also very troublesome.”*\n\nTransition operators represent a conditional distribution of the next states given the current states. Therefore, we express all transition operators in terms of their corresponding conditional distributions, without introducing the operator notation.\n\n* Marginal Transition Operators: $p_{x_{1:n}}(y_{1:n}^{(t)} | y_{1:n}^{(t-1)})$ denotes the conditional density function of the function outputs at time $t$ given the function outputs at time $t-1$. The subscript $x_{1:n}$ indicates that these conditional density functions are different for different input sequences $x_{1:n}$.\n\n* Functional Transition Operators: In our paper, we use $p(f^{(t)})$ to denote a stochastic process (i.e. a distribution over functions). Thus, the functional transition operator $p(f^{(t)} | f^{(t-1)})$ refers to the conditional distribution of the current functions (a stochastic process) given the prior one. In our paper, functional transition operators are defined indirectly via marginal transition operators. The notation here is only for ease of discussion.\n\n"}]

Author rebuttal:
Here we updated Figure 2 in the uploaded PDF, which illustrates the generative and inference models of MNPs. In the reviewer rebuttals below, we address each reviewer's questions and comments separately.

Meta-review: This paper proposes a new class of stochastic processes called Markov Neural processes that stacks sequences of Markov transition operators in the function space and satisfies the requirements of proper stochastic processes. There is agreement among the reviewers on the novelty and technical strength of the paper as well as in its limited experimental evaluation. Overall, I believe the first two aspects (novelty and technical strength) are contributions worth of presentation at NeurIPS.
Decision: Accept (poster)