fast algorithms for bayesian uncertainty quantification in large scale linear inverse problems based on low rank partial hessian approximations we consider the problem of estimating the uncertainty in large scale linear statistical inverse problems with high dimensional parameter spaces within the framework of bayesian inference when the noise and prior probability densities are gaussian the solution to the inverse problem is also gaussian and is thus characterized by the mean and covariance matrix of the posterior probability density unfortunately explicitly computing the posterior covariance matrix requires as many forward solutions as there are parameters and is thus prohibitive when the forward problem is expensive and the parameter dimension is large however for many ill posed inverse problems the hessian matrix of the data misfit term has a spectrum that collapses rapidly to zero we present a fast method for computation of an approximation to the posterior covariance that exploits the low rank structure of the preconditioned by the prior covariance hessian of the data misfit analysis of an infinite dimensional model convection diffusion problem and numerical experiments on large scale three dimensional convection diffusion inverse problems with up to 1 5 million parameters demonstrate that the number of forward pde solves required for an accurate low rank approximation is independent of the problem dimension this permits scalable estimation of the uncertainty in large scale ill posed linear inverse problems at a small multiple independent of the problem dimension of the cost of solving the forward problem

moments of infinite convolutions of symmetric bernoulli distributions we study the infinite convolution of symmetric bernoulli distributions associated to a parameter r we obtain an explicit formula for the moments as a function of bernoulli numbers and conditioned partitions applying this formula we obtain the moments as a quotient of polynomials in the parameter r the leading coefficient of the numerator is related to the asymptotic behavior of the moments and unexpectedly this coefficients are the absolute values of euler numbers

the space of cubic surfaces equipped with a line the cayley salmon theorem implies the existence of a 27 sheeted covering space of the parameter space of smooth cubic surfaces marking each of the 27 lines on each surface in this paper we compute the rational cohomology of the total space of this cover using the spectral sequence in the method of simplicial resolution developed by vassiliev the covering map is an isomorphism in cohomology in fact of mixed hodge structures and the cohomology ring is isomorphic to that of mathrm pgl 4 mathbb c we derive as a consequence of our theorem that over the finite field mathbb f q the average number of lines on a smooth cubic surface equals 1 away from finitely many characteristics this average is 1 o q 1 2 by a standard application of the weil conjectures

a stochastic newton mcmc method for large scale statistical inverse problems with application to seismic inversion we address the solution of large scale statistical inverse problems in the framework of bayesian inference the markov chain monte carlo mcmc method is the most popular approach for sampling the posterior probability distribution that describes the solution of the statistical inverse problem mcmc methods face two central difficulties when applied to large scale inverse problems first the forward models typically in the form of partial differential equations that map uncertain parameters to observable quantities make the evaluation of the probability density at any point in parameter space very expensive and second the high dimensional parameter spaces that arise upon discretization of infinite dimensional parameter fields make the exploration of the probability density function prohibitive the challenge for mcmc methods is to construct proposal functions that simultaneously provide a good approximation of the target density while being inexpensive to manipulate here we present a so called stoch

randomized algorithms for the low rank approximation of matrices we describe two recently proposed randomized algorithms for the construction of low rank approximations to matrices and demonstrate their application inter alia to the evaluation of the singular value decompositions of numerically low rank matrices being probabilistic the schemes described here have a finite probability of failure in most cases this probability is rather negligible 10 17 is a typical value in many situations the new procedures are considerably more efficient and reliable than the classical deterministic ones they also parallelize naturally we present several numerical examples to illustrate the performance of the schemes

a sequential particle filter method for static models summary particle filter methods are complex inference procedures which combine importance sampling and monte carlo schemes in order to explore consistently a sequence of multiple distributions of interest we show that such methods can also offer an efficient estimation

multilevel monte carlo path simulation we show that multigrid ideas can be used to reduce the computational complexity of estimating an expected value arising from a stochastic differential equation using monte carlo path simulations in the simplest case of a lipschitz payoff and a euler discretisation the computational cost to achieve an accuracy of o e is reduced from o e 3 to o e 2 log e 2 the analysis is supported by numerical results showing significant computational savings

the rational cohomology of overline mathcal m _4 we present two approaches to the study of the cohomology of moduli spaces of curves together they allow us to compute the rational cohomology of the moduli space m over bar 4 of stable comple

bayesian inverse problems with l_1 priors a randomize then optimize approach prior distributions for bayesian inference that rely on the l 1 norm of the parameters are of considerable interest in part because they promote parameter fields with less regularity than gaussian priors e g discontinuities and blockiness these l 1 type priors include the total variation tv prior and the besov space b 1 1 s prior and in general yield non gaussian posterior distributions sampling from these posteriors is challenging particularly in the inverse problem setting where the parameter space is high dimensional and the forward problem may be nonlinear this paper extends the randomize then optimize rto method an optimization based sampling algorithm developed for bayesian inverse problems with gaussian priors to inverse problems with l 1 type priors we use a variable transformation to convert an l 1 type prior to a standard gaussian prior such that the posterior distribution of the transformed parameters is amenable to metropolized sampling via rto we demonstrate thi

hessian based adaptive sparse quadrature for infinite dimensional bayesian inverse problems abstract in this work we propose and analyze a hessian based adaptive sparse quadrature to compute infinite dimensional integrals with respect to the posterior distribution in the context of bayesian inverse problems with gaussian prior due to the concentration of the posterior distribution in the domain of the prior distribution a prior based parametrization and sparse quadrature may fail to capture the posterior distribution and lead to erroneous evaluation results by using a parametrization based on the hessian of the negative log posterior the adaptive sparse quadrature can effectively allocate the quadrature points according to the posterior distribution a dimension independent convergence rate of the proposed method is established under certain assumptions on the gaussian prior and the integrands dimension independent and faster convergence than o n 1 2 is demonstrated for a linear as well as a nonlinear inverse problem whose posterior distribution can be effectively approximated by a gaussian distribution at the map point

finding structure with randomness probabilistic algorithms for constructing approximate matrix decompositions low rank matrix approximations such as the truncated singular value decomposition and the rank revealing qr decomposition play a central role in data analysis and scientific computing this work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low rank matrix approximation these techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets this paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions these methods use random sampling to identify a subspace that captures most of the action of a matrix the input matrix is then compressed either explicitly or implicitly to this subspace and the reduced matrix is manipulated deterministically to obtain the desired low rank factorization in many cases this approach beats its classical competitors in terms of accuracy robustness and or speed these claims are supported by extensive numerical experiments and a detailed error analysis the specific benefits of randomized techniques depend on the computational environment consider the model problem of finding the k dominant components of the singular value decomposition of an m times n matrix i for a dense input matrix randomized algorithms require bigo mn log k floating point operations flops in contrast to bigo mnk for classical algorithms ii for a sparse input matrix the flop count matches classical krylov subspace methods but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures iii for a matrix that is too large to fit in fast memory the randomized techniques require only a constant number of passes over the data as opposed to bigo k passes for classical algorithms in fact it is sometimes possible to perform matrix approximation with a single pass over the data

scalable posterior approximations for large scale bayesian inverse problems via likelihood informed parameter and state reduction two major bottlenecks to the solution of large scale bayesian inverse problems are the scaling of posterior sampling algorithms to high dimensional parameter spaces and the computational cost of forward model evaluations yet incomplete or noisy data the state variation and parameter dependence of the forward model and correlations in the prior collectively provide useful structure that can be exploited for dimension reduction in this setting both in the parameter space of the inverse problem and in the state space of the forward model to this end we show how to jointly construct low dimensional subspaces of the parameter space and the state space in order to accelerate the bayesian solution of the inverse problem as a byproduct of state dimension reduction we also show how to identify low dimensional subspaces of the data in problems with high dimensional observations these subspaces enable approximation of the posterior as a product of two factors i a projection of the posterior onto a low dimensional parameter subspace wherein the original likelihood is replaced by an approximation involving a reduced model and ii the marginal prior distribution on the high dimensional complement of the parameter subspace we present and compare several strategies for constructing these subspaces using only a limited number of forward and adjoint model simulations the resulting posterior approximations can rapidly be characterized using standard sampling techniques e g markov chain monte carlo two numerical examples demonstrate the accuracy and efficiency of our approach inversion of an integral equation in atmospheric remote sensing where the data dimension is very high and the inference of a heterogeneous transmissivity field in a groundwater system which involves a partial differential equation forward model with high dimensional state and parameters

particle markov chain monte carlo methods summary markov chain monte carlo and sequential monte carlo methods have emerged as the two main tools to sample from high dimensional probability distributions although asymptotic convergence of markov chain monte carlo algorithms is ensured under weak assumptions the performance of these algorithms is unreliable when the proposal distributions that are used to explore the space are poorly chosen and or if highly correlated variables are updated independently we show here how it is possible to build efficient high dimensional proposal distributions by using sequential monte carlo methods this allows us not only to improve over standard markov chain monte carlo schemes but also to make bayesian inference feasible for a large class of statistical models where this was not previously so we demonstrate these algorithms on a non linear state space model and a levy driven stochastic volatility model

dimension independent likelihood informed mcmc many bayesian inference problems require exploring the posterior distribution of high dimensional parameters that represent the discretization of an underlying function this work introduces a family of markov chain monte carlo mcmc samplers that can adapt to the particular structure of a posterior distribution over functions two distinct lines of research intersect in the methods developed here first we introduce a general class of operator weighted proposal distributions that are well defined on function space such that the performance of the resulting mcmc samplers is independent of the discretization of the function second by exploiting local hessian information and any associated low dimensional structure in the change from prior to posterior distributions we develop an inhomogeneous discretization scheme for the langevin stochastic differential equation that yields operator weighted proposals adapted to the non gaussian structure of the posterior the resulting dimension independent and likelihood informed dili mcmc samplers may be useful for a large class of high dimensional problems where the target probability measure has a density with respect to a gaussian reference measure two nonlinear inverse problems are used to demonstrate the efficiency of these dili samplers an elliptic pde coefficient inverse problem and path reconstruction in a conditioned diffusion

accelerating markov chain monte carlo with active subspaces the markov chain monte carlo mcmc method is the computational workhorse for bayesian inverse problems however mcmc struggles in high dimensional parameter spaces since its iterates must sequentially explore the high dimensional space this struggle is compounded in physical applications when the nonlinear forward model is computationally expensive one approach to accelerate mcmc is to reduce the dimension of the state space active subspaces are part of an emerging set of tools for subspace based dimension reduction an active subspace in a given inverse problem indicates a separation between a low dimensional subspace that is informed by the data and its orthogonal complement that is constrained by the prior with this information one can run the sequential mcmc on the active variables while sampling independently according to the prior on the inactive variables however this approach to increase efficiency may introduce bias we provide a bound on the hellinger distance between the true posterio

a computational framework for infinite dimensional bayesian inverse problems part ii stochastic newton mcmc with application to ice sheet flow inverse problems we address the numerical solution of infinite dimensional inverse problems in the framework of bayesian inference in part i of this paper t bui thanh o ghattas j martin and g stadler siam j sci comput 35 2013 pp a2494 a2523 we considered the linearized infinite dimensional inverse problem in part ii we relax the linearization assumption and consider the fully nonlinear infinite dimensional inverse problem using a markov chain monte carlo mcmc sampling method to address the challenges of sampling high dimensional probability density functions pdfs arising upon discretization of bayesian inverse problems governed by pdes we build upon the stochastic newton mcmc method this method exploits problem structure by taking as a proposal density a local gaussian approximation of the posterior pdf whose covariance operator is given by the inverse of the local hessian of the negative log posterior pdf the construction of the covariance is made tractable by invoking a low rank approximat

proposals which speed up function space mcmc abstract inverse problems lend themselves naturally to a bayesian formulation in which the quantity of interest is a posterior distribution of state and or parameters given some uncertain observations for the common case in which the forward operator is smoothing then the inverse problem is ill posed well posedness is imposed via regularization in the form of a prior which is often gaussian under quite general conditions it can be shown that the posterior is absolutely continuous with respect to the prior and it may be well defined on function space in terms of its density with respect to the prior in this case by constructing a proposal for which the prior is invariant one can define metropolis hastings schemes for mcmc which are well defined on function space stuart 2010 1 cotter et al 2 and hence do not degenerate as the dimension of the underlying quantity of interest increases to infinity e g under mesh refinement when approximating pde in finite dimensions however in practice despite the attractive theoretical properties of the currently available schemes they may still suffer from long correlation times particularly if the data is very informative about some of the unknown parameters in fact in this case it may be the directions of the posterior which coincide with the already known prior which decorrelate the slowest the information incorporated into the posterior through the data is often contained within some finite dimensional subspace in an appropriate basis perhaps even one defined by eigenfunctions of the prior we aim to exploit this fact and improve the mixing time of function space mcmc by careful rescaling of the proposal to this end we introduce two new basic methods of increasing complexity involving i characteristic function truncation of high frequencies and ii hessian information to interpolate between low and high frequencies the second more sophisticated version bears some similarities with recent methods which exploit local curvature information for example rmhmc girolami and calderhead 2011 3 and stochastic newton martin et al 2012 4 these ideas are illustrated with numerical experiments on the bayesian inversion of the heat equation and navier stokes equation given noisy observations

optimal scaling of discrete approximations to langevin diffusions we consider the optimal scaling problem for proposal distributions in hastings metropolis algorithms derived from langevin diffusions we prove an asymptotic diffusion limit theorem and show that the relative efficiency of the algorithm can be characterized by its overall acceptance rate independently of the target distribution the asymptotically optimal acceptance rate is 0 574 we show that as a function of dimension n the complexity of the algorithm is o n1 3 which compares favourably with the o n complexity of random walk metropolis algorithms we illustrate this comparison with some example simulations

a computational framework for infinite dimensional bayesian inverse problems part i the linearized case with application to global seismic inversion we present a computational framework for estimating the uncertainty in the numerical solution of linearized infinite dimensional statistical inverse problems we adopt the bayesian inference formulation given observational data and their uncertainty the governing forward problem and its uncertainty and a prior probability distribution describing uncertainty in the parameter field find the posterior probability distribution over the parameter field the prior must be chosen appropriately in order to guarantee well posedness of the infinite dimensional inverse problem and facilitate computation of the posterior furthermore straightforward discretizations may not lead to convergent approximations of the infinite dimensional problem and finally solution of the discretized inverse problem via explicit construction of the covariance matrix is prohibitive due to the need to solve the forward problem as many times as there are parameters our computational framework builds on the infinite dimensional form

sequential monte carlo samplers summary we propose a methodology to sample sequentially from a sequence of probability distributions that are defined on a common space each distribution being known up to a normalizing constant these probability distributions are approximated by a cloud of weighted random samples which are propagated over time by using sequential monte carlo methods this methodology allows us to derive simple algorithms to make parallel markov chain monte carlo algorithms interact to perform global optimization and sequential bayesian estimation and to compute ratios of normalizing constants we illustrate these algorithms for various integration tasks arising in the context of bayesian inference

bayesian inference of random fields represented with the karhunen loeve expansion abstract the integration of data into engineering models involving uncertain and spatially varying parameters is oftentimes key to obtaining accurate predictions bayesian inference is effective in achieving such an integration uncertainties related to spatially varying parameters are typically represented through random fields discretized into a finite number of random variables the prior correlation length and variance of the field as well as the number of terms in the random field discretization have a considerable impact on the outcome of the bayesian inference which has received little attention in the literature here we investigate the implications of different choices in the prior random field model on the solution of bayesian inverse problems we employ the karhunen loeve expansion for the representation of random fields we show that a higher order karhunen loeve discretization is required in bayesian inverse problems as compared to standard prior random field representations since the updated fields are non homogeneous furthermore the smoothing effect of the forward operator has a large influence on the posterior solution particularly if we are analyzing quantities of interest that are sensitive to local random fluctuations of the inverse quantity this is also reflected in posterior predictions such as the estimation of rare event probabilities we illustrate these effects analytically through a 1d cantilever beam with spatially varying flexibility and numerically using a 2d linear elasticity example where the young s modulus is spatially variable

a randomized maximum a posteriori method for posterior sampling of high dimensional nonlinear bayesian inverse problems we present a randomized maximum a posteriori rmap method for generating approximate samples of posteriors in high dimensional bayesian inverse problems governed by large scale forward problems we derive the rmap approach by 1 casting the problem of computing the map point as a stochastic optimization problem 2 interchanging optimization and expectation and 3 approximating the expectation with a monte carlo method for a specific randomized data and prior mean rmap reduces to the randomized maximum likelihood rml approach it can also be viewed as an iterative stochastic newton method an analysis of the convergence of the rmap samples is carried out for both linear and nonlinear inverse problems each rmap sample requires solution of a pde constrained optimization problem to solve these problems we employ a state of the art trust region inexact newton conjugate gradient method with sensitivity based warm starts an approximate metropolization approach is presented to reduce the bias in rma

a complex ball uniformization of the moduli space of cubic surfaces via periods of k3 surfaces in this paper we show that the moduli space of nodal cubic surfaces is iso morphic to a quotient of a 4 dimensional complex ball by an arithmetic subgroup of the unitary group this complex ball uniformization uses the periods of certain k3 surfaces which are naturally associated to cubic surfaces a similar uniformization is given for dif ferent covers of the moduli space corresponding to geometric markings of the picard group or a choice of a line on the surface we also give a detailed description of the boundary com ponents corresponding to singular surfaces

active subspace methods in theory and practice applications to kriging surfaces many multivariate functions in engineering models vary primarily along a few directions in the space of input parameters when these directions correspond to coordinate directions one may apply global sensitivity measures to determine the most influential parameters however these methods perform poorly when the directions of variability are not aligned with the natural coordinates of the input space we present a method to first detect the directions of the strongest variability using evaluations of the gradient and subsequently exploit these directions to construct a response surface on a low dimensional subspace i e the active subspace of the inputs we develop a theoretical framework with error bounds and we link the theoretical quantities to the parameters of a kriging response surface on the active subspace we apply the method to an elliptic pde model with coefficients parameterized by 100 gaussian random variables and compare it with a local sensitivity analysis method for dimension reduc

riemann manifold langevin and hamiltonian monte carlo methods the paper proposes metropolis adjusted langevin and hamiltonian monte carlo sampling methods defined on the riemann manifold to resolve the shortcomings of existing monte carlo algorithms when sampling from target densities that may be high dimensional and exhibit strong correlations the methods provide fully automated adaptation mechanisms that circumvent the costly pilot runs that are required to tune proposal densities for metropolis hastings or indeed hamiltonian monte carlo and metropolis adjusted langevin algorithms this allows for highly efficient sampling even in very high dimensions where different scalings may be required for the transient and stationary phases of the markov chain the methodology proposed exploits the riemann geometry of the parameter space of statistical models and thus automatically adapts to the local structure when simulating paths across this manifold providing highly efficient convergence and exploration of the target density the performance of these riemann manifold monte carlo methods is rigorously assessed by performing inference on logistic regression models log gaussian cox point processes stochastic volatility models and bayesian estimation of dynamic systems described by non linear differential equations substantial improvements in the time normalized effective sample size are reported when compared with alternative sampling approaches matlab code that is available from http www ucl ac uk statistics research rmhmc allows replication of all the results reported

geometric mcmc for infinite dimensional inverse problems bayesian inverse problems often involve sampling posterior distributions on infinite dimensional function spaces traditional markov chain monte carlo mcmc algorithms are characterized by deteriorating mixing times upon mesh refinement when the finite dimensional approximations become more accurate such methods are typically forced to reduce step sizes as the discretization gets finer and thus are expensive as a function of dimension recently a new class of mcmc methods with mesh independent convergence times has emerged however few of them take into account the geometry of the posterior informed by the data at the same time recently developed geometric mcmc algorithms have been found to be powerful in exploring complicated distributions that deviate significantly from elliptic gaussian laws but are in general computationally intractable for models defined in infinite dimensions in this work we combine geometric methods on a finite dimensional subspace with mesh independent infinite dimensional approaches our objective is to speed up mcmc mixing times without significantly increasing the computational cost per step for instance in comparison with the vanilla preconditioned crank nicolson pcn method this is achieved by using ideas from geometric mcmc to probe the complex structure of an intrinsic finite dimensional subspace where most data information concentrates while retaining robust mixing times as the dimension grows by using pcn like methods in the complementary subspace the resulting algorithms are demonstrated in the context of three challenging inverse problems arising in subsurface flow heat conduction and incompressible flow control the algorithms exhibit up to two orders of magnitude improvement in sampling efficiency when compared with the pcn method

auxiliary gradient based sampling algorithms summary we introduce a new family of markov chain monte carlo samplers that combine auxiliary variables gibbs sampling and taylor expansions of the target density our approach permits the marginalization over the auxiliary variables yielding marginal samplers or the augmentation of the auxiliary variables yielding auxiliary samplers the well known metropolis adjusted langevin algorithm mala and preconditioned crank nicolson langevin algorithm pcnl are shown to be special cases we prove that marginal samplers are superior in terms of asymptotic variance and demonstrate cases where they are slower in computing time compared with auxiliary samplers in the context of latent gaussian models we propose new auxiliary and marginal samplers whose implementation requires a single tuning parameter which can be found automatically during the transient phase extensive experimentation shows that the increase in efficiency measured as the effective sample size per unit of computing time relative to optimized implementations of pcnl elliptical slice sampling and mala ranges from tenfold in binary classification problems to 25 fold in log gaussian cox processes to 100 fold in gaussian process regression and it is on a par with riemann manifold hamiltonian monte carlo sampling in an example where that algorithm has the same complexity as the aforementioned algorithms we explain this remarkable improvement in terms of the way that alternative samplers try to approximate the eigenvalues of the target we introduce a novel markov chain monte carlo sampling scheme for hyperparameter learning that builds on the auxiliary samplers the matlab code for reproducing the experiments in the paper is publicly available and an on line supplement to this paper contains additional experiments and implementation details