Bayesian Inference with Big Data : A Snapshot from a Workshop

Over the last half century, and particularly since the advent of Markov Chain Monte Carlo methods, Bayesian inference has enjoyed tremendous successes, so much so that some quarters have proclaimed the age-old philosophical arguments between Frequentism and Bayesianism resolved, and that the twenty-first century is the Bayesian century. Whether such proclamations are controversial or not, the twenty first century has brought a new challenge to Bayesian inference that may render moot such philosophical arguments without a serious discussion of computation. This is the challenge of Big Data. Indeed, the growth in volume and variety of data in the twenty first century has been staggering, and one could ask the question whether Bayesian inference is still a relevant statistical framework in this context. Does one still have to worry about model uncertainty and overfitting if there is so much data that uncertainties are negligible for all practical purposes? We believe there are a number of responses to this critique that rather point to the opposite direction. Firstly, big data often means large p with p ≫ N . For instance, modern MRI scanners record millions of voxels, possibly over time, for a handful of patients only. For another example, the cost for sequencing entire genomes is falling faster than Moore’s law, implying that within a few years a doctor could routinely sequence every patient’s genome: this is four billion basepairs per patient. Clearly, model uncertainty remains an important problem. Secondly, most of the data may not be relevant to certain model components. For instance, in recommendation systems with billions of user-product interactions, almost all of this data is irrelevant for any new user or product entering the system: this is the so-called cold start problem. Lastly, with the recent advances in deep learning, the number of parameters of modern neural network models have been growing at a faster pace than data volumes, with tens of billions parameters been reported by large internet firms such as Google, Yahoo! , Facebook and Baidu. Add to this the fact that the amount of “useful” or predictive information in data usually grows sublinearly in N . We thus observe that model capacity can in fact be growing faster than the amount of information we would like to store in it. That overfitting can still be a serious issue in this domain is evidenced by the fact that a new, rather extreme form of regularization (dropout) was invented to prevent it in neural networks [9]. One could argue that any model where regularization is not a consideration because the model’s capacity is small compared to the data volume is in fact underfitting and therefore not optimally predictive (unless the data was generated from the model, which is never the case in reality). What then are the challenges that a Bayesian approach faces in a Big Data context? Not surprisingly, a major challenge is computation. Traditionally two approaches exist to Bayesian inference: MCMC sampling and variational optimization. The former is asymptotically unbiased but often suffers from a significant variance in its estimates while the second is deterministic and has no variance but often exhibits rather strong biases. Until recently, standard coordinate ascent implementations of both methods were unsuited for very large datasets because every update (sampling or variational) depended on all data items.