Linear-Time Probabilistic Solution of Boundary Value Problems

Abstract: We propose a fast algorithm for the probabilistic solution of boundary value problems (BVPs), which are ordinary differential equations subject to boundary conditions.  In contrast to previous work, we introduce a Gauss-Markov prior and tailor it specifically to BVPs, which allows computing a posterior distribution over the solution in linear time, at a quality and cost comparable to that of well-established, non-probabilistic methods.  Our model further delivers uncertainty quantification, mesh refinement, and hyperparameter adaptation. We demonstrate how these practical considerations positively impact the efficiency of the scheme. Altogether, this results in a practically usable probabilistic BVP solver that is (in contrast to non-probabilistic algorithms) natively compatible with other parts of the statistical modelling tool-chain.


Reviews and author responses:

 Thank you for your response! After reading other reviews, I keep my score. 

 Thanks for your review and for your concise summary.

With all due respect, we were surprised by some of your evaluations. This is probably a classic misunderstanding between reviewers and authors, so please let us take a position on your points of criticism. We are confident that the confusion can be cleared up and that we can get on the same page.

1. We agree that the presented algorithm is restricted to solving ODE boundary value problems, a domain that has received significant study and is widely accepted as sufficiently distinct from initial value problems to require completely separate algorithms. Whether this is a limited scope is subjective and certainly up for discussion (we describe a number of potential applications in Section 2). We are not entirely sure what makes the results on error estimation, calibration, and initialisation “boilerplate”. It is important for efficient BVP solvers to consider such practical aspects (see e.g. the role of error estimation, mesh refinement and similar concepts in [1, 18, 25]). 

2. We agree that there is a connection to the work by Bosch et al. [6], but we respectfully disagree on how strong this connection is. Bosch et al. solve initial value problems, which is a  task that implies algorithms that are fundamentally different to boundary value problem solvers (it seems to be a misunderstanding that BVP solvers are only slight variations of IVP solvers). In our view, the papers [10-12] are more closely related. Please refer to the introduction and conclusion of the paper, as well as to our reply to the reviewer JUUq for a thorough distinction.

3. We understand and acknowledge your confusion. From a purely technical perspective, $Y=[Y_0, ..., Y_\nu]$ models the integrated Wiener process $Y_0$ as well as its $\nu$ derivatives (jointly). For this stack of states, there exists an SDE formulation [Chapter 12, 23]. The terminology of referring to the stack of vectors as the integrated Wiener process is aligned with contemporary work on probabilistic ODE solvers [4-9, 13], which includes [6]. In fact, if you look at Equation (2a,b) in [6], you will see how their description coincides with our description (through the choice $A = (\sum_{i=1}^\nu e_i e_{i+1}^\top) \otimes I_d$ and $B=e_{\nu+1} \otimes I_d$). Not only do we use the same prior (integrated Wiener processes) and the same terminology, but also the same SDE!

4. It is only a minor correction, but the method is not restricted to solving first-order BVPs without transforming them into first-order problems (Figure 7 contains results on a fourth-order ODE; see also Appendix D).

Did these comments help clear up the results, and hopefully improve your evaluation of the paper? If not, please let us discuss further.

 Thank you for your review! We agree with you in the observation that the paper by John et al. [12] is one of the most closely related works. Please allow us to also include the papers by Hennig and Hauberg [10] and Arvanitidis et al. [11] in the following explanation of what we do differently.

The present work contains at least two improvements over those manuscripts:

1) We use a Markov prior, which brings with it computationally efficient (i.e. linear-time) implementations of Gauss-Newton algorithms in the iterated Kalman smoother. This is important because non-probabilistic, state-of-the-art methods have linear complexity, but [10-12] are all based on full Gaussian process regression which has cubic complexity in the number of grid points. 

2) We describe a complete algorithm for solving BVPs, including initialisation of the optimiser, error estimation, and mesh refinement. These have not been part of [10-12]. (On top of this, the initialisation strategy is something that non-probabilistic solvers cannot do.) Calibration was treated in the aforementioned works, but arguably not to the extent of our paper.

We hope that both points are sufficiently well-described in the manuscript. We would be grateful for your further assessment regarding whether it needs a more prominent position in the exposition because we agree with you in that the differences should be made clear :)

Again, thank you for your evaluation. We hope we were able to resolve any misunderstandings. If not, please let us discuss further.

This paper extends filtering based probabilistic numerical methods for solving ODE initial value problems to the solution of boundary value ODE problems by introducing boundary value constraints through the likelihood.    The application is an important one, as pointed out in the introduction. However, the mathematical details provided in the paper seem insufficient to fully understand the proposed approach. 

Abstract

I could not verify the claim in the abstract that "In contrast to previous work, we introduce a Gauss–Markov prior and tailor it specifically to BVPs", because the prior (solution to equation 2) is not directly constrained at the boundaries. In fact, it seems to me that the BVP constraints are incorporated through the likelihood (equation 3) rather than directly through the prior. However, the prior is often referred to as a "bridge prior" in the paper, which makes me wonder if I have misunderstood something. 

Introduction 

The authors should discuss the difference between their approach and other existing PNN methods for solving BVPs. Two that I am aware of are Bayesian Solution Uncertainty Quantification for Differential Equations by Chkrebti et al. and Bayesian Probabilistic Numerical Methods by Cockayne et al.

Section 2

The prior should be defined outside of section 2.1, since "Generative model" typically refers to the likelihood rather than the prior.

The prior specification is incomplete. The prior is a diffusion process, specifically it is the solution to the SDE in equation (2). However, the form of the drift and diffusion terms are not specified. A question that naturally comes up is why a diffusion process is appropriate for modeling a smooth ODE solution. There is a vague suggestion that some of the components of the vector Y are actually integrated diffusion processes, which may at least guarantee that the prior over the solution is differentiable. The paper states that "Other [prior] choices are possible" without elaborating. It seems that other approaches in the literature may be more natural, for example jointly modeling the solution and derivatives through a Gaussian process prior allows better control of the smoothness. At a minimum I believe that a discussion of the prior and its features and drawbacks is required.  

Section 3 

Lack of clarity in section 2 makes the rest of the paper difficult to understand. The details of the filtering algorithm seem likewise difficult to understand for a general mathematical audience. 

In general, the paper is not self-contained. It references specific equations from various other papers that many readers will not have the time to track down. For example, the sentence starting on line 53 on page 2, "The entries in A ... are specified by the integrated Wiener process and imply that $Y_q$ models the $q$th derivative of the BVP solution" is incomplete. The reference to  "[6, Equation 2]", which presumably provides the form of A and an explanation of the derivative modeling, requires reading a different paper to be able to understand the proposed model. This happens several times throughout the paper.    Unsure

The paper proposes a probabilistic method to solve ode with boundary conditions. The. method is Bayesian, where the posterior distribution is computed with a Gaussian process as the prior, conditioning on satisfying the ode and boundary conditions. The posterior distribution is approximated with a Gaussian distribution where the mean and variance are obtained from implementing an extended Kalman smoother. The paper discusses at length how to select the algorithm hyper parameters, such as initialization and mesh and illustrates the algorithm on several numerical examples.            I am not sure about significant novelty introduced compared to prior work on probabilistic methods to solve BVPs that are referenced. For example, the claims in [12] are quite similar to the claims of this paper: a probabilistic method based on a Gaussian posterior that performs on par with deterministic methods.  Both methods seem to involve learning Gaussian processes. What are the significant and new ideas presented in the paper to distinguish it from prior work? From the way the paper is written, it seems like another probabilistic method way to  solve the BVP, with more discussion on choosing the hyperparameters.  

 Yes

This paper details a probabilistic method for solving nonlinear first order differential equation systems (ODEs) subject to given boundary conditions. The authors title this a boundary value problem (BVP). The paper builds on existing methods for solving first order ODEs in reference [6] in the paper, and represents the solution of the discretized ODE as the  MAP estimate of a Bayesian posterior. The primary contribution of the paper is a practical algorithm for computing the MAP estimate using the so-called iterated extended Kalman smoother. The primary contributions of the paper is to describe methods to initialize the IEKS, adaptively select the discretization grid and calibrate hyperparameters.  Overall, I found the paper to be a little narrow, and the algorithmic contributions to be somewhat narrow.
Originality: The paper hews very closely to existing works. The main results on initialization, error estimation, and calibration are rather boilerplate. The utility of the proposed algorithms seems limited beyond the specific setting of this paper.

Quality and Significance: The quality of the results and presentation are lacking. The results, as noted above, are pretty standard, and the proposed algorithm is really a mash-up of a many different algorithms. Overall I would say the results are not significant.

Clarity: I found the paper to be rather poorly written. In particular the description of the generative model leaves much to desire. I note that (2) is not the integrated Wiener process. Indeed, if one follows the description in [6], it is pretty clear that the multivariate process Y involves time integrals, and not SDEs. Indeed, I do not believe that the integrated Wiener process in [6] can be expressed as the solution of an SDE, like (2). Ultimately, however (2) doesn’t play a big role in the overall message of the paper, but it does show that the authors have a fundamental misunderstanding of the setting in [6]. Moving beyond this, the overall description of the posterior  is rather poor, and the reader is expected to interpolate the authors descriptions.  It would be hard to ascribe negative societal impact of solving first order ODEs. I have no feedback to offer on this front.

Meta-review: This paper was uneasy to review. The AC asked 10+ ppl if they could review it, but still could only find 2 reviewers. With the help of SAC, we found a 3rd reviewer. Unfortunately, some reviewer didn't fully understand the technicality of the paper, and the last reviewer who knows the subject better did not appreciate the significance of the paper's results. Based on all reviewers' feedback, we can do nothing but reject this paper.

The fact that we had a very hard time finding good-quality reviewers for this paper hints that: the paper's topic is far from mainstream machine learning (which isn't necessarily an issue), but the paper is also not presented in a way to appeal to the machine learning community (which is a real issue). We recommend that the reviewers either submit the paper to venues more focused on differential equations, or substantially revise the paper to suit a machine learning conference.

Decision: accept