Embarrassingly Parallel MCMC for Fast and Flexible Analysis of Spatiotemporal Data

Background. Large datasets are often collected or stored in a distributed fashion over a number of machines. We would like to develop scalable Bayesian inference algorithms for these datasets. Most existing algorithms require all data to be sent to a central node for processing, or regular communication between machines during parallel processing (both of which require a great deal of communication of information or parameters between machines). Additionally, we often want inference results on many different subsets of these datasets, for example, when analyzing local portions of large spatiotemporal data; most existing methods must rerun inference algorithms for each queried subset, which can take a great deal of time when the number of queries grows large. Aim. The aim of this project is to develop an “embarrassingly parallel” method of Bayesian inference, in which each machine performs inference on a subset of data without any communication to other machines. By doing this, we hope to be able to perform statistical learning of large, distributed datasets without needing the regular transfer of data or parameters. Data. We demonstrate the advantages of this method on multiple real datasets. We first show the ability of this method to increase the speed of inference in Bayesian regression and mixture models. We then show the ability to use this technique to analyze spatiotemporal data; in particular, our method is able to yield inference results on arbitrarily selected subsets of data without rerunning any of our initial inference algorithms. Here, we design a generative model for taxi trip data, and run our inference method on a set of 14 million taxi trips in New York City. Methods. We work in a Bayesian framework; hence the goal of the learning methods will be to compute a posterior distribution over model parameters. The strategy here is to perform Bayesian inference on each machine (on its subset of data) in parallel. After inference is complete, we transfer all inference results to a single machine and combine the local results to infer a global result (i.e. the posterior distribution given the entire, global dataset). More specifically, we perform Markov chain Monte Carlo (MCMC), a sampling algorithm, to conduct local inference. To combine the local results, we develop a sample combination algorithm that combines the local sample sets to produce samples from the full-data (global) posterior distribution. Results. We develop a method to perform inference in an embarrassingly parallel manner: on each node we conduct local inference on a subset of data, and afterwards, we combine the local results to yield a global inference result. We show that our combination procedure yields asymptotically exact global inference results (i.e. our results converge to the same value as if we had used the full data on a single machine), and we use our method to analyze several large real-world datasets. Conclusions. The goal of our project was to develop an embarrassingly parallel method of Bayesian inference, in which each machine performs inference on a subset of data without any communication to other machines. This method allows us to perform statistical learning on large, distributed datasets (e.g. spatiotemporal data, which may be split over time and space) without needing the regular transfer of data or parameters.