Programming language semantics as a foundation for Bayesian inference

Bayesian modelling, in which our prior belief about the distribution on model parameters is updated by observed data, is a popular approach to statistical data analysis. However, writing specific inference algorithms for Bayesian models by hand is timeconsuming and requires significant machine learning expertise. Probabilistic programming promises to make Bayesian modelling easier and more accessible by letting the user express a generative model as a short computer program (with random variables), leaving inference to the generic algorithm provided by the compiler of the given language. However, it is not easy to design a probabilistic programming language correctly and define the meaning of programs expressible in it. Moreover, the inference algorithms used by probabilistic programming systems usually lack formal correctness proofs and bugs have been found in some of them, which limits the confidence one can have in the results they return. In this work, we apply ideas from the areas of programming language theory and statistics to show that probabilistic programming can be a reliable tool for Bayesian inference. The first part of this dissertation concerns the design, semantics and type system of a new, substantially enhanced version of the Tabular language. Tabular is a schema-based probabilistic language, which means that instead of writing a full program, the user only has to annotate the columns of a schema with expressions generating corresponding values. By adopting this paradigm, Tabular aims to be user-friendly, but this unusual design also makes it harder to define the syntax and semantics correctly and reason about the language. We define the syntax of a version of Tabular extended with user-defined functions and pseudo-deterministic queries, design a dependent type system for this language and endow it with a precise semantics. We also extend Tabular with a concise formula notation for hierarchical linear regressions, define the type system of this extended language and show how to reduce it to pure Tabular. In the second part of this dissertation, we present the first correctness proof for a Metropolis-Hastings sampling algorithm for a higher-order probabilistic language. We define a measure-theoretic semantics of the language by means of an operationallydefined density function on program traces (sequences of random variables) and a map from traces to program outputs. We then show that the distribution of samples returned by our algorithm (a variant of “Trace MCMC” used by the Church language) matches the program semantics in the limit. Lay Summary Bayesian probabilistic modelling, in which the user designs a model expressing how they believe some observable data is generated from some unknown parameters, is one of the most popular approaches to machine learning. However, implementing an efficient inference algorithm, calculating the expected values of unknown parameters, for a given probabilistic model, can be very difficult and time-consuming and require significant knowledge of machine learning and statistics. Meanwhile, there are many professionals whoe are not machine learning experts but would still like to apply probabilistic modelling to problems in their areas. Probabilistic programming aims to make Bayesian modelling more accessible by letting the user express the desired model as a program in a given probabilistic language— the expected values of unknown parameters are then computed automatically by the inference engine of the language. This dissertation aims to advance the state of probabilistic programming and consists of two parts. In the first part, we present a substantially extended version of a particular existing probabilistic language, called Tabular, which, instead of extending a general-purpose language with features for probabilistic programming, allows users to specify models as annotated database schemas. We extend this language with userdefined functions, which allow for some reusable model components to be defined just once and used in programs wherever needed. We show how to reduce models with functions to a simpler form on which we can run inference directly. Furthermore we define a dependent type system for Tabular, which catches common modelling errors and helps the user debug a model more quickly. We also endow the language with a semantics, which defines precisely the mathematical meanings of programs, and prove some properties of this extended language. We subsequently extend Tabular with a sub-language which allows expressing hierarchical linear models, a wide and commonly-used class of models, more concisely. The second part of this dissertation is concerned with correctness of inference algorithms for universal probabilistic languages, which can express a wide class of probabilistic models. These languages typically use inference algorithms such as Metropolis-Hastings Markov chain Monte Carlo, which generates a large number of samples of some unknown quantity to approximate its distribution. Such algorithms usually lack correctness proofs and bugs have been found in some of them. We present the first formal proof of correctness of a variant of Metropolis-Hastings for a functional probabilistic language, which shows that the distribution of samples actually approximates the true distribution of the quantity of interest.