SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In practice, the inference model  $q_{x_{1:n}}(z^{(1:T)}|y_{1:n};\phi)$  provides an approximate prior/posterior. Ideally, if it gives the exact posterior, conditional consistency would hold perfectly. However, when the inference model is approximate, the degree of conditional consistency depends on the discrepancy between the inference model and the true posterior. Predictive SP models also have a stochastic mapping conditioned on the context, represented as  $q_{x_{1:m}}(z^{(1:T)}|y_{1:m};\phi)$ . However, it is important not to confuse this with the inference model, as they do not serve to approximate the posterior.
Bayesian nonparametric models Bayesian nonparametric models such as GPs [24, 48] and Student-t processes [5, 10, 52] provide common classical approaches for modelling distributions over functions. Under these models, any conditional distribution of a target given a context can be directly evaluated, and both marginal/conditional consistency is guaranteed by construction (if all computation is exact). However, they can be too restrictive for some applications, e.g. any conditional or marginal distribution of a GP is also Gaussian. Further, the evaluation of conditional densities is typically computationally intensive (cubic w.r.t. the context due to matrix inversion). Deep GPs [13, 59] use GPs as building blocks to construct deep architectures, designed to be flexible enough to model a wide range of complex systems. However, only the hyperparameters for each GP layer are tunable, which means they can still be restrictive when modelling highly non-Gaussian data.
Copula-based processes In multivariate statistics, a copula function refers to a multivariate function that describes the dependence between random variables and links the marginal distributions of each variable to the joint distribution of all the variables [28, 41]. Similarly, a copula process [58] describes the dependency between infinitely many random variables independent of their marginal distributions. [32, 33, 38] exploited this independence to specify the more flexible Copula-Based Processes (CBPs) by combing the copula processes of existing SPs with flexible models of marginal distributions based on normalising flows [16, 43, 49]; the consistency of CBPs is guaranteed as long as the base SPs are consistent. However, CBPs are still restrictive in terms of modelling relationship between variables because the underlying copula processes still come from known SPs. For example, BRUNOs [32, 33] have the same underlying copula processes as GPs, so they cannot be used to model data with non-Gaussian dependencies.
Neural process family NPs [22] are generative SP models which specify a prior SP and rely on Bayesian inference to compute conditional densities. To improve expressivity of the original NPs, [4, 18, 30] explore predictive SP models that directly learn mappings from context to predictive SPs. More specifically, ANPs [30] incorporate cross-attention modules to model the interaction between context and target. CONVNPs [18] produce a functional representation with a convolutional architecture which parameterizes a stationary predictive SP over the target, given the context. In GNPs [4], predictive SPs are modelled by GPs where the mean/kernel functions are directly produced by neural networks conditioned on the context. However, simply setting the context to an empty set in these models does not yield more expressive generative SP models. In the absence of context, ANPs and GNPs become standard NPs and GPs respectively, without any additional expressivity. CONVNPs can indeed be adjusted for generative SP models where the latent variables are random functions. However, it remains unclear how to perform Bayesian inference in function space [18]. Conditional NP families, e.g. Conditional Neural Processes (CNPs) [21], Convolutional Conditional Neural Processes (CONVCNPs) [25] are another category of predictive SP models, which make a strong assumption that all targets are independent given the context. Recently, Transformer Neural Processes (TNPs) [42], Neural Diffusion Processes (NDPs) [17] provide an alternative to the NP models above, showing promising predictive performance. However, both marginal and conditional consistency are sacrificed in these models.
Table 1: Estimated marginal log-likelihood of SPs on 1D function regression problems. We compare (a) Oracle models (when available). (b) GPs with optimised hyperparameters for additive kernels with three component kernels including an RBF kernel, a Matern kernel and a periodic kernel. (c) CBPs which combine Gaussian copula processes with neural spline flows [16]. (d) NPs. (e) MNPs with 7 transition steps. Each experiment is repeated 5 times and we report the mean and standard errors of the marginal log-likelihood normalised by the number of points in the target. When latent models such as MNPs, NPs are used, we obtain estimations of marginal log-likelihoods on test data using the IwAE objective [6] with 20 latent samples.
Model	Samples from GPs	Non-GP Data
RBF Kernel	Matern Kernel	Periodic Kernel	Monotonic	Convex	SDEs
Oracle	2.846±0.012	2.709±0.013	0.641±0.006	—	—	—
GPs	2.844±0.013	2.708±0.014	0.419±0.013	0.633±0.059	2.976±0.224	1.719±0.034
CBPs	2.628±0.016	2.604±0.015	0.169±0.022	1.776±0.088	4.268±0.035	1.842±0.024
NPs	0.935±0.019	1.115±0.021	0.356±0.020	1.823±0.006	1.956±0.004	1.621±0.009
MNPs	2.491±0.024	2.290±0.021	0.594±0.032	2.755±0.010	5.582±0.081	1.942±0.019
$\delta = 0.5$
Figure 3: Wheel contextual bandits with varying exploration parameter  $\delta$ . The optimal actions are  $a_1, \ldots, a_5$  when the context point are in blue, yellow, red, green and black regions respectively.
$\delta = 0.7$
$\delta = 0.9$
$\delta = 0.95$
$\delta = 0.99$
Our experiments aim to answer the questions: 1) Can the proposed framework of MNPs offer better expressivity than NPs? 2) Compared to Bayesian nonparametric models, do neural parameterised SPs have advantages, especially on highly non-Gaussian data? 3) How well do MNPs perform on scientific problems? All experiments are performed using PyTorch [44]. For details about datasets, please see Appendix B.1. For additional experimental details such as hyperparameters and architectures, please refer to Appendix B.2 and our reference implementation at https://github.com/jinxu06/mnp.
6.1 1D function regression
We first consider 1D function regression problems to perform controlled comparisons between GPs, CBPs, NPs, and MNPs. Table 1 shows results across a range of datasets, including samples from the GP prior (we consider three different kernels) as well as more challenging non-GP data.
For datasets sampled from GPs, we also include the performance of oracle GPs with the right hyperparameters for generating the data. As shown, GPs and CBPs are close to the oracle except for samples generated using a periodic kernel, where CBPs struggle to learn the right kernel hyperparameters due to the difficulty of optimisation. For neural parameterised models, the performance gaps between the MNPs and the oracle GPs are much smaller than for NPs.
For non-GP samples such as monotonic functions, convex functions and samples from certain SDEs, MNPs perform the best across all models. CBPs obtain better marginal log-likelihood than GPs because the pointwise marginals are transformed using normalising flows, but their underlying copula processes (which capture the dependence between data points) are still Gaussian, restricting their ability to model highly non-Gaussian data compared to neural parameterised models e.g. MNPs, NPs.
6.2 Contextual bandits
Contextual bandits are a class of problems where agents repeatedly choose from a set of actions based on context and receive rewards. The goal is then to learn a policy that maximizes expected cumulative reward over time. Contextual bandits find applications in real-time decision making problems such as resource allocation and online advertising.
Table 2: Results on wheel contextual bandit problems. We use an increasing value of  $\delta$ , where more exploration is needed with higher  $\delta$ . We report the mean and standard deviation of both cumulative and simple regrets (a performance measure of the final policy, estimated by the mean cumulative regrets in the last 500 steps) over 100 trials. Results are normalised by the performance of a uniform agent which chooses actions with equal probability.
δ	0.5	0.7	0.9	0.95	0.99
Cumulative regrets
Uniform	100.0±0.00	100.0±0.00	100.0±0.00	100.0±0.00	100.0±0.00
MAML	2.95±0.12	3.11±0.16	4.84±0.22	7.01±0.33	22.93±1.57
NPs	1.60±0.06	1.75±0.05	3.31±0.10	5.71±0.24	22.13±1.23
MNPs	1.08±0.00	1.23±0.01	2.10±0.01	2.07±0.01	5.46±0.05
Simple regrets
Uniform	100.0±0.00	100.0±0.00	100.0±0.00	100.0±0.00	100.0±0.00
MAML	2.49±0.12	3.00±0.35	4.75±0.48	7.10±0.77	22.89±1.41
NPs	1.04±0.06	1.26±0.21	2.90±0.35	5.45±0.47	21.45±1.3
C-BRUNO	1.32±0.06	1.43±0.07	3.44±0.13	6.17±0.21	21.52±0.41
MNPs	1.14±0.06	1.29±0.08	2.12±0.16	2.24±0.16	5.22±0.38
Figure 4: Inferred geology conditioned on measurements. Given a small number of physical measurements (red pixels indicate positive measurements while blue pixels indicate negative ones), we show the predictive mean (first row in each panel) and standard deviation (second row in each panel). Different columns correspond to different context sets. As can be seen, MNPs make predictions that are consistent with the data while showing larger uncertainty when fewer context points are available (or when a prediction is made far from a context point).
Similarly to Garnelo et al. [22], we use the wheel bandit problem [50] (see Appendix B.1) to evaluate our approach: a unit circle is partitioned into 5 regions (see Figure 3) whose sizes are controlled by an exploration parameter  $\delta$ . There are 5 actions  $a_1, a_2, a_3, a_4, a_5$ , and their associated reward depend on a 2D contextual coordinate  $X = (X_1, X_2)$  uniformly sampled from within the circle. If  $||X|| \leq \delta$ ,  $a_1$  is the optimal action with reward sampled from  $\mathcal{N}(1.2, 0.01^2)$ , and taking any other action would yield a reward  $r \sim \mathcal{N}(1.0, 0.01^2)$ . If  $||X|| > \delta$ , the optimal action depends on which of the remaining 4 region  $X$  falls into and choosing it would yield a reward  $r \sim \mathcal{N}(50, 0.01^2)$ . Under this circumstance, any other action yields a reward  $\mathcal{N}(1.0, 0.01^2)$  except that  $a_1$  receives  $r \sim \mathcal{N}(1.2, 0.01^2)$ . We follow the experimental setup from Garnelo et al. [22] and only include models with a pre-training procedure (see Appendix B.2 for details). As can be seen in Table 2, MNPs significantly outperform baselines (taken from the results of [22]) across different exploration rates  $\delta$ .
6.3 Geological inference
In geostatistics, one is often interested in inferring the geological structure of an area given only a sparse set of measurements. This problem has traditionally been tackled with variants of GP regression
Table 3: Test marginal log-likelihood on geology data. Uniform context points are uniformly sampled from within the 2D square, while regional context includes the top/left half or the central square with half the size.
|C|	Uniform Context	Regional Context
N=20	40	80	160	Top	Left	Center
NPs	-0.35±0.30	-0.31±0.29	-0.28±0.27	-0.25±0.27	-39.63±304.08	-22.03±107.25	-1.18±4.20
MNPs	0.67±0.31	0.80±0.23	0.98±0.18	1.12±0.15	0.44±0.59	0.22±0.61	0.65±0.39
[12] (often referred to as Kriging in this context). However, several geological patterns (e.g. fluvial patterns) are highly complex and cannot be properly captured by these methods. To address this, several geostatistical models use a single training image to extract geological patterns and match these to the measurements [53, 63]. However, these approaches generally fail to produce realistic patterns capturing the variability of real geology.
More recently, deep learning approaches have been applied to this problem. [8, 14, 62] train GANs on large geological datasets and use this as a prior for inferring geological structure given sparse measurements. However, the resulting models do not provide any meaningful uncertainty estimates, which are crucial for decision making in several applications. As MNPs provide uncertainty estimates they may therefore be a compelling approach for this problem.
To evaluate MNPs on this problem, we introduce the GeoFluvial dataset, containing more than 25k simulations of fluvial patterns as  $128 \times 128$  grayscale images. The dataset was generated using the open source meanderpy [54] package, which itself simulates the geological patterns as SPs (see Appendix B.1 for details). We train our model on a training set of 20k simulations and evaluate it on a test set of 5k simulations. We test our trained model on a variety of context point configurations, corresponding to geological measurements taken at various spatial locations. Specifically, we consider uniformly sampled measurements (at 20, 40, 80, and 160 locations) as well as scenarios where measurements are spatially restricted to a certain area (top, left, and in the center of the square). Quantitative results are shown in Table 3 and qualitative results in Figure 4. As can be seen, our model achieves better likelihoods than NPs on this problem for all context point configurations, while also generating patterns that match the geological context. Further, as can be seen in Figure 4, the uncertainty estimates of the model are consistent with expectations, i.e. the model is more uncertain far from measurement locations or when there is ambiguity in the direction of the fluvial pattern.
Limitations and future work Because we apply only a finite number of Markov transitions, the entire computational graph containing intermediate states is held in memory for backpropagation. To alleviate this, an interesting direction for future work would be to consider continuous-time Markov transitions, i.e. stochastic differential equations in function space, which require less memory by using adjoint methods [9, 37]. While Equation (9) provides a valid general construction of exchangeable and consistent MMTOs with latent variables, it may be feasible to investigate other forms of MMTOs that do not necessitate latent variables, thereby eliminating the need for variational approaches.
There are often inherent symmetries in data (e.g. translational symmetries for stationary SPs) and it may be possible to improve empirical performance of MNPs using (group equivariant) convolutions that preserve certain symmetries [18, 25, 29]. These improvements are, however, orthogonal to our contributions and combining them is an exciting direction for future work.
Conclusions We have introduced Markov Neural Processes (MNPs), a framework for constructing flexible neural generative SP models. MNPs use neural-parameterized Markov transition operators on function spaces to gradually transform a simple initial SP into a flexible one. We proved that our proposed neural transitions preserve exchangeability and consistency necessary to define valid SPs. Empirical studies demonstrate that we can obtain expressive models of SPs with promising performance on contextual bandits and geological data.
We would like to thank Hyunjik Kim for valuable discussions. We also thank Peter Tilke for useful discussions around generating the geological dataset. JX gratefully acknowledges funding from Tencent AI Labs through the Oxford-Tencent Collaboration on Large Scale Machine Learning.
[1] Dimitri P. Bertsekas and Steven E. Shreve. Stochastic optimal control: the discrete time case. 2007.
[2] Bickford Smith, Kirsch, Farquhar, Gal, Foster, and Rainforth. Prediction-oriented Bayesian active learning. International Conference on Artificial Intelligence and Statistics, 2023.
[3] Paul C. Bressloff. Stochastic processes in cell biology. Interdisciplinary Applied Mathematics, 2021.
[4] Wessel Bruinsma, James Requeima, Andrew YK Foong, Jonathan Gordon, and Richard E Turner. The gaussian neural process. In Third Symposium on Advances in Approximate Bayesian Inference, 2021.
[5] Tu Dinh Bui, Dinh Tran, and Richard E Turner. Student-t processes as alternatives to gaussian processes. In International Conference on Machine Learning, pages 1232-1241, 2016.
[6] Yuri Burda, Roger B Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
[7] Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical science, pages 273-304, 1995.
[8] Shing Chan and Ahmed H Elsheikh. Parametric generation of conditional geological realizations using generative neural networks. Computational Geosciences, 23(5):925-952, 2019.
[9] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[10] Muyang Chung and Richard E Turner. Non-gaussian process regression with student-t processes. In International Conference on Machine Learning, pages 4607-4616, 2019.
[11] Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International conference on machine learning, pages 2133-2143. PMLR, 2020.
[12] Noel Cressie. The origins of kriging. Mathematical geology, 22(3):239-252, 1990.
[13] Andreas Damianou and Neil D Lawrence. Deep gaussian processes. In Artificial intelligence and statistics, pages 207-215. PMLR, 2013.
[14] Emilien Dupont, Tuanfeng Zhang, Peter Tilke, Lin Liang, and William Bailey. Generating realistic geology conditioned on physical measurements with generative adversarial networks. arXiv preprint arXiv:1802.03065, 2018.
[15] Emilien Dupont, Yee Whye Teh, and A. Doucet. Generative models as distributions of functions. In AISTATS, 2022.
[16] Conor Durkan, George Papamakarios, Iain Murray, and Jascha Sohl-Dickstein. Neural spline flows. In International Conference on Learning Representations, 2019.
[17] Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes. arXiv preprint arXiv:2206.03992, 2022.
[18] Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33:8284-8295, 2020.
[58] Andrew G Wilson and Zoubin Ghahramani. Copula processes. Advances in Neural Information Processing Systems, 23, 2010.
[59] Andrew Gordon Wilson, Matthew D Hoffman, Cheng Wang, Eric Fox, and Zoubin Ghahramani. Deep gaussian processes with stochastic backpropagation. In International Conference on Machine Learning, pages 1436-1445, 2016.
[60] Jin Xu, Jean-Francois Ton, Hyunjik Kim, Adam Kosiorek, and Yee Whye Teh. Metafun: Meta-learning with iterative functional updates. In International Conference on Machine Learning, pages 10617-10627. PMLR, 2020.
[61] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[62] Tuan-Feng Zhang, Peter Tilke, Emilien Dupont, Ling-Chen Zhu, Lin Liang, and William Bailey. Generating geologically realistic 3d reservoir facies models using deep learning of sedimentary architecture with generative adversarial networks. *Petroleum Science*, 16(3):541-549, 2019.
[63] Tuanfeng Zhang, Paul Switzer, and Andre Journel. Filter-based classification of training image patterns for spatial simulation. Mathematical Geology, 38(1):63-80, 2006.
Appendix of Deep Stochastic Processes via Functional Markov Transition Operator
A.1 Proof of proposition 4.1 (See page 4)
Proposition 4.1. Let  $F_{\theta}(\cdot ;x,z):\mathcal{Y}\mapsto \mathcal{Y}$  denote an invertible transformation between outputs, parameterized by the input and latent. If  $\{p_{x_{1:n}}(y_{1:n}^{(0)})\}_{x_{1:n}}$  is a valid Stochastic Process (SP) (i.e. it satisfies Equation (1) and Equation (2)) and
Proof. Given an input  $x_{i}$  and a latent variable  $z$ ,  $y_{i}$  is obtained by applying an invertible transformation  $F_{\theta}$  to  $y_{i}^{(0)}$ . Therefore, we can represent  $p(y_{i}|y_{i}^{(0)};x_{i},z)$  using the Dirac delta:
$$
p \left(y _ {i} \mid y _ {i} ^ {(0)}; x _ {i}, z\right) = \delta \left(y _ {i} - F _ {\theta} \left(y _ {i} ^ {(0)}; x _ {i}, z\right)\right) \tag {16}
$$
Furthermore, according to Equation (6), these transformations are independently applied given the latent variable  $z$ . So we have:
$$
p \left(y _ {1: n} \mid y _ {1: n} ^ {(0)}; x _ {1: n}, z\right) = \prod_ {i = 1} ^ {n} \delta \left(y _ {i} - F _ {\theta} \left(y _ {i} ^ {(0)}; x _ {i}, z\right)\right) \tag {17}
$$
$$
\begin{array}{l} p _ {x _ {1: n}} \left(y _ {1: n} \mid y _ {1: n} ^ {(0)}\right) = \int p _ {\theta} (z) p _ {\theta} \left(y _ {1: n} \mid y _ {1: n} ^ {(0)}; x _ {1: n}, z\right) d z (18) \\ = \int p _ {\theta} (z) \prod_ {i = 1} ^ {n} \delta \left(y _ {i} - F _ {\theta} \left(y _ {i} ^ {(0)}; x _ {i}, z\right)\right) d z (19) \\ \end{array}
$$
As per Proposition 4.3,  $\{p_{x_{1:n}}(y_{1:n}|y_{1:n}^{(0)})\}_{x_{1:n}}$  is exchangeable and consistent. Furthermore, because  $\{p_{x_{1:n}}(y_{1:n}^{(0)})\}_{x_{1:n}}$  is a valid SP,  $\{p_{x_{1:n}}(y_{1:n})\}_{x_{1:n}}$  is also a valid SP
A.2 Proof of proposition 4.2 (See page 4)
Proposition 4.2. If the collection of marginals before transition  $\{p_{x_{1:n}}(y_{1:n}^{(0)})\}_{x_{1:n}}$  is consistent and exchangeable (as per Equations (1) and (2)) and the collection of marginal Markov transition operators (MMTOs)  $\{p_{x_{1:n}}(y_{1:n}|y_{1:n}^{(0)})\}_{x_{1:n}}$  is also consistent and exchangeable (as per Equations (7) and (8)), then the collection of marginals after transition  $\{p_{x_{1:n}}(y_{1:n})\}$  is also consistent and exchangeable, hence defining a valid SP.
Proof. According to the definitions of consistency and exchangeability for both the collection of marginals  $\{p_{x_{1:n}}(y_{1:n}^{(0)})\}_{x_{1:n}}$  and the collection of transition operators  $\{p_{x_{1:n}}(y_{1:n}|y_{1:n}^{(0)})\}_{x_{1:n}}$ , we have the following (Equations (1), (2), (7) and (8)):
$$
\begin{array}{l} \int p _ {x _ {1: n}} (y _ {1: n} ^ {(0)}) \mathrm {d} x _ {m + 1: n} = p _ {x _ {1: m}} (y _ {1: m} ^ {(0)}) \\ p _ {\pi (x _ {1: n})} (\pi (y _ {1: n} ^ {(0)})) = p _ {x _ {1: n}} (y _ {1: n} ^ {(0)}) \\ \int p _ {x _ {1: n}} \left(y _ {1: n} \mid y _ {1: n} ^ {(0)}\right) d y _ {m + 1: n} = p _ {x _ {1: m}} \left(y _ {1: m} \mid y _ {1: m} ^ {(0)}\right) \\ p _ {x _ {1: n}} \left(y _ {1: n} \mid y _ {1: n} ^ {(0)}\right) = p _ {\pi \left(x _ {1: n}\right)} \left(\pi \left(y _ {1: n}\right) \mid \pi \left(y _ {1: n} ^ {(0)}\right)\right) \\ \end{array}
$$
where  $1 <   m <   n$
The Markov transition on the function outputs are described by:
$$
p _ {x _ {1: n}} (y _ {1: n}) = \int p _ {x _ {1: n}} (y _ {1: n} ^ {(0)}) p _ {x _ {1: n}} (y _ {1: n} | y _ {1: n} ^ {(0)}) d y _ {1: n} ^ {(0)}
$$
For any finite sequence  $x_{1:n}$ , we have
$$
\begin{array}{l} \int p _ {x _ {1: n}} \left(y _ {1: n}\right) \mathrm {d} y _ {m + 1: n} = \int \left(\int p _ {x _ {1: n}} \left(y _ {1: n} ^ {(0)}\right) p _ {x _ {1: n}} \left(y _ {1: n} \mid y _ {1: n} ^ {(0)}\right) \mathrm {d} y _ {1: n} ^ {(0)}\right) \mathrm {d} y _ {m + 1: n} \\ = \int p _ {x _ {1: n}} \left(y _ {1: n} ^ {(0)}\right) \left(\int p _ {x _ {1: n}} \left(y _ {1: n} \mid y _ {1: n} ^ {(0)}\right) d y _ {m + 1: n}\right) d y _ {1: n} ^ {(0)} \\ = \int p _ {x _ {1: n}} \left(y _ {1: n} ^ {(0)}\right) p _ {x _ {1: m}} \left(y _ {1: m} \mid y _ {1: m} ^ {(0)}\right) d y _ {1: n} ^ {(0)} \\ = \int \left(\int p _ {x 1: n} \left(y _ {1: n} ^ {(0)}\right) \mathrm {d} y _ {m + 1: n} ^ {(0)}\right) p _ {x 1: m} \left(y _ {1: m} \mid y _ {1: m} ^ {(0)}\right) \mathrm {d} y _ {1: m} ^ {(0)} \\ = \int p _ {x _ {1: m}} \left(y _ {1: m} ^ {(0)}\right) p _ {x _ {1: m}} \left(y _ {1: m} \mid y _ {1: m} ^ {(0)}\right) d y _ {1: m} ^ {(0)} \\ = \int p _ {x _ {1: m}} \left(y _ {1: m}\right) \mathrm {d} y _ {1: m} ^ {(0)} \tag {20} \\ \end{array}
$$