Black Box Probabilistic Numerics

1 Introduction . Probabilistic numerics ( PN ) has attracted significant recent interest from researchers in machine learning , motivated by the possibility of incorporating probabilistic descriptions of numerical uncertainty into applications of probabilistic inference and decision support [ 1 , 2 ] . PN treats the intermediate calculations performed in running a traditional ( i.e . non-probabilistic ) numerical procedure as data , which can be used to constrain a random variable model for the quantity of interest [ 3 ] . Conjugate Gaussian inference has been widely exploited , with an arsenal of PN methods developed for linear algebra [ 4–11 ] , cubature [ 12–30 ] , optimisation [ 31–36 ] , and differential equations [ 37–55 ] . However , nonlinear tasks pose a major technical challenge to this approach , as well as to computational statistics in general , due to the absence of explicit conditioning formulae . Compared to traditional numerical methods , which have benefited from a century or more of sustained research effort , the current scope of PN is limited . The performance gap is broadly characterised by the absence of certain important functionalities—adaptivity , numerical well-conditioning , efficient use of computational resource—all of which contribute to limited applicability in real-world settings . This article proposes a pragmatic solution that enables state-of-the-art numerical algorithms to be immediately exploited in the context of PN . The idea , which we term black box probabilistic numerics ( BBPN ) , is a statistical perspective on Richardson ’ s deferred approach to the limit ( RDAL ) [ 56 ] . The starting point for BBPN is a sequence of increasingly accurate approximations produced by a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . traditional numerical method as its computational budget is increased . Extrapolation of this sequence ( to the unattainable limit of ‘ infinite computational budget ’ ) is formulated as a prediction task , to which statistical techniques can be applied . For concreteness , we perform this prediction using Gaussian processes ( GPs ) [ 57 ] , but other models could be used . Note that we do not aim to remove the numerical analyst from the loop ; the performance of BBPN is limited by that of the numerical method on which it is based . There are three main advantages of BBPN compared to existing methods in PN : ( 1 ) BBPN is applicable to any numerical task for which there exists a traditional numerical method ; ( 2 ) stateof-the-art performance and functionality are automatically inherited from the underlying numerical method ; ( 3 ) BBPN achieves a provably higher order of convergence relative to a single application of the numerical method on which it is based , in an analogous manner to RDAL . The main limitations of BBPN , compared to existing methods in PN , are : ( 1 ) multiple realisations of a traditional numerical method are required ( i.e . one datum is not sufficient in an extrapolation task ) , and ( 2 ) a joint statistical model has to be built for not just the quantity of interest ( as in standard PN ) , but also for the error associated with the output of a traditional numerical method . The capacity of generic statistical models , such as GPs , to learn salient aspects of this structure from data and to produce meaningful predictions over a range of real-world numerical problems , demands to be investigated . The article is organised as follows : In Section 2 we recall classical RDAL . In Section 3 we lift RDAL to the space of probability distributions , exploiting GPs to instantiate BBPN and providing a theoretical guarantee that higher order convergence is achieved by using GPs within BBPN . In Section 4 we present a detailed empirical investigation into BBPN , demonstrating its effectiveness on challenging tasks that go beyond the capability of current methods in PN , while also highlighting potential pitfalls . As part of this we perform a comparison of the uncertainty quantification properties of BBPN against earlier approaches . A closing discussion is contained in Section 5 . 2 Turning Lead into Gold . Our starting point is the celebrated observation of Richardson [ 56 ] , that multiple numerical approximations can be combined to produce an approximation more accurate than any of the individual approximations . To see this , consider an intractable scalar quantity of interest q⇤ 2 R , and suppose that q⇤ can be approximated by a numerical method q that depends on a parameter h > 0 , such that q ( h ) = q⇤ + Ch↵ +O ( h↵+1 ) ( 1 ) for some C 2 R ( which may be unknown ) and ↵ > 0 ( which is assumed known , and called the order of the method ) . Clearly q ( h ) converges to q⇤ as h ! 0 , but we also suppose that the cost of computing q ( h ) increases in the same limit , with exact evaluation of q ( 0 ) requiring a hypothetically infinite computational budget . Proposition 1 is the cornerstone of RDAL . It demonstrates that two evaluations of a numerical method of order ↵ can be combined to obtain a numerical method of order ↵+ 1 . An elementary proof of this foundational result is provided in Appendix A . Proposition 1 . Let q be a numerical method of order ↵ , as in ( 1 ) . Fix 2 ( 0 , 1 ) and let q ( h ) denote the height at which a straight line drawn through the points ( h↵ , q ( h ) ) and ( ( h ) ↵ , q ( h ) ) intersects the vertical axis in R2 . Then q is a numerical method of order ↵+ 1 . Now consider the natural generalisation of Proposition 1 , in which we compute approximations q ( hi ) along a decreasing sequence of values ( hi ) ni=1 . One can then fit a smooth interpolant to the points { ( h↵ i , q ( hi ) ) } ni=1 ( generalising the straight line through two points ) , then extrapolate this to h = 0 , to give an estimate for the quantity of interest . This simple idea is widely used in numerical analysis ; its potential to radically improve solution accuracy , given only a sequence of simple calculations as input , prompted Press et al . [ 58 , p. 922 ] to describe it as “ turning lead into gold ” . The practical success of RDAL depends on the choice of interpolant , with polynomial interpolation being most commonly used . Unqualified , RDAL is usually understood to refer to an order n 1 polynomial fitted to n points , which produces a numerical method of order ↵+n ; see Theorem 9.1 of [ 59 ] . Higher-order polynomial extrapolation is known to perform poorly unless the values ( hi ) ni=1 are able to be chosen specifically to mitigate Runge ’ s phenomenon [ 60 ] , motivating the Bulirsch–Stoer algorithm [ 61 ] , which instead fits a rational function interpolant . This allows both greater expressiveness and robustness than polynomial interpolation ( though not necessarily as efficiently [ 58 ] ) . These methods are all situated within the broad category of extrapolation methods in numerical analysis ; a comprehensive historical survey can be found in [ 62 ] . Figure 1 presents a simple visual demonstration of RDAL , applied to the method of Riemann sums for an oscillatory 1D integrand . While RDAL gives improved approximations , no quantification of estimation uncertainty is provided . The only attempt of which we are aware to provide uncertainty quantification for RDAL is due to [ 63 ] , who focused on the Navier–Stokes equation and a specific scalar quantity of interest . Here we go further , proposing the general framework of BBPN and introducing novel methodology that goes beyond scalar quantities of interest . The right-hand pane of Figure 1 displays the outcome of the method we are about to introduce , applied to the same task—observe that the true value of the integral falls within the ±2 credible set produced using BBPN . Details of the simulations in this figure are contained in Appendix C.1 . 3 Methodology . The core idea of BBPN is to model q ( h ) as a stochastic process Q ( h ) rather than fit a deterministic interpolant as in RDAL . The distribution of the marginal random variable Q ( 0 ) is then interpreted as a representation of the epistemic uncertainty in the quantity of interest q ( 0 ) . Conjugate Gaussian inference can be performed in BBPN , since one needs only to construct an interpolant . This means the challenge of nonlinear conditioning encountered in PN [ 4 ] is avoided , massively extending the applicability of PN . In addition to being able to leverage state-of-the-art numerical methods , the BBPN approach enjoys provably higher orders of convergence relative to a single application of the numerical method on which it is based ; see Section 3.3 . 3.1 Notation and Setup . Our starting point is to generalise ( 1 ) to encompass essentially all numerical tasks , following the abstract perspective of [ 64 ] . To do so , we observe that any quantity of interest q⇤ can be characterised by a sufficiently large collection of real values q⇤ ( t ) , with t ranging over an appropriate index set T . Definition 2 . A traditional numerical method is defined as a map q : [ 0 , h0 ) ⇥ T ! R , for some h0 > 0 such that , for all t 2 T , the function h 7 ! q ( h , t ) is continuous at 0 with limit q ( 0 , t ) = q⇤ ( t ) . For example , a ( univariate ) initial value problem ( IVP ) , in which t is interpreted as time , can be solved using a traditional numerical method q whose time step size h trades off approximation error against computational cost . The output q ( h , t ) of such a method represents an approximation to the true solution q⇤ ( t ) of the IVP , at each time t for which the solution is defined . In general , depending on the numerical task , the index t could be spatio-temporal , discrete , or even an unstructured set , while the meaning of the index h will depend on the numerical method . Definition 2 thus encompasses , among other things : ( 1 ) adaptive integrators for time-evolving partial differential equations ( PDEs ) , where h > 0 represents a user-specified error tolerance , and the spatio-temporal domain of the solution is indexed by T ; ( 2 ) iterative methods for approximating the singular values of a d ⇥ d matrix , where for example h : = w 1 with w the number of iterations performed , and the ordered singular values are indexed by T = { 1 , . . . , d } ; and ( 3 ) the simultaneous approximation of multiple related quantities of interest , where T indexes not only the domain ( s ) on which the individual quantities of interest are defined , but also the multiple quantities of interest themselves ( this situation arises , for example , in inverse problems that are PDE-constrained [ 65 ] ) . The perspective in Definition 2 is abstract but , as these examples make clear , it will typically only be possible to compute q at certain input values ( such as h = w 1 for w 2 N , or for just a finite collection of inputs t if the index set T is infinite ) , and furthermore each evaluation is likely to be associated with a computational cost . Thus complete information regarding the map q : [ 0 , h0 ) ⇥ T ! R will not be available in general , and there will therefore remain epistemic uncertainty in its complete description . Our aim in Section 3.2 , in line with the central philosophical perspective of PN , is to characterise this uncertainty using a statistical model .

Summary: This paper presents a probabilistic version of Richardson's "deferred approach to the limit" (RDAL) method.  RDAL is an established numerical analysis method (developed in 1927) with many applications. In RDAL several low-accuracy approximations are extrapolated to produce a more accurate approximation (that is costly to come with directly).  In the framework proposed in this draft, namely "Black box probabilistic numerics" (BBPN), the aforementioned extrapolation/curve fitting is carried out via Gaussian Processes and therefore comes with all the benefits for Bayesian approaches such as quantification of the  estimation uncertainty. This approach is simple and it can potentially be combined with Bayesian methods other than GPs as well.  In the experiment section the following applications are presented: nonlinear ODE/PDE i.e. ordinary/partial differential equations and eigenvalue problem.