Statistical inference in generative models using scoring rules
Keywords: generative neural networks
scoring rules
likelihood-free inference
summary statistics
generative models
deep learning
probabilistic forecasting
generalised Bayesian inference
simulation-based inference
Bayesian inference

Statistical models which allow generating simulations without providing access to the density of the distribution are called simulator models. They are commonly developed by scientists to represent natural phenomena and depend on physically meaningful parameters. Analogously, generative networks produce samples from a probability distribution by transforming draws from a noise (or latent) distribution via a neural network; as for simulator models, the density is unavailable. These two frameworks, developed independently from different communities, can be grouped into the class of generative models; compared to statistical models that explicitly specify the density, they are more powerful and flexible.
For generative networks, typically, a single point estimate for the parameters (or weights) is obtained by minimizing an objective function through gradient descent enabled by automatic differentiation. In contrast, for simulator models, samples from a probability distribution for the parameters are usually obtained via some statistical algorithm. Nevertheless, in both cases, the inference methods rely on common principles that exploit simulations. In this thesis, I follow the principle of assessing how a probabilistic model matches an observation by Scoring Rules. This generalises common statistical practices based on the density function and, with specific Scoring Rules, allows tackling generative models.
After a detailed introduction and literature review in Chapter 1, the first part of this thesis (Chapters 2 and 3) is concerned with methods to infer probability distributions for the parameters of simulator models. Specifically, Chapter 2 contributes to the traditional Bayesian Likelihood-Free Inference literature with a new way to learn summary statistics, defined as the sufficient statistics of the best exponential family approximation to the simulator model. In contrast, Chapter 3 departs from tradition by defining a new posterior distribution based on the generalised Bayesian inference framework, rather than motivated as an approximation to thestandard posterior. The posterior is defined through Scoring Rules computable for simulator models and is robust to outliers.
In the second part of the thesis (Chapters 4 and 5), I study Scoring Rule Minimization to determine the weights of generative networks; for specific choices of Scoring Rules, this approach better captures the variability of the data than popular alternatives. I apply generative networks trained in this way to uncertainty-sensitive tasks: in Chapter 4 I use them to provide a probability distribution over the parameters of simulator models, thus falling back to the theme of Chapters 2 and 3; instead, in Chapter 5, I consider probabilistic forecasting, also establishing consistency of the training objective with dependent training data.
Finally, I conclude in Chapter 6 with some final thoughts and directions for future work.