Bayesian Optimal Experimental Design of Streaming Data Incorporating Machine Learning Generated Synthetic Data

Authors: kentaro  hoffman
tyler h mccormick
Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-06 01:24:35

Abstract: This paper demonstrates two main innovations to aid in statistical inference using synthetic data in dynamic contexts. First, using a class of estimators which give valid statistical inference using synthetic and real data points, even when the operating characteristics of the synthetic data generation process are unknown, we illustrate how to incorporate our proposed estimators into dynamic linear models to analyze streaming data. Second, we combined our proposed estimators with Bayesian optimal experimental design to dynamically determine the optimal ratio of real and synthetic data to minimize model standard error.

Main paper: 
 introduction: 
 With the rapid advancement of Machine Learning (ML) and Artificial Intelligence (AI), data-based models are quickly becoming an important part of high-profile decision making. For example, ML and AI models are now being used to make medical diagnoses Ahsan et al. [2022] , rapidly screen documents Lee et al. [2023] , Gongane et al. [2022] , determine marketing strategies, Stone et al. [2020] , and aid in making parole decisions Wang et al. [2022] . However, in many ways, models are only as good as the data used to train them. Partially due to the success and ease of ML and AI models, there have been concerns that the proliferation of accurate-seeming but inaccurate data such as deepfakes or hallucinations could poison existing sources with lower quality data. The endpoint of this concerning direction is a phenomenon which has been referred to as "model-collapse " Shumailov et al. [2024] , Gibney [2024] , Gerstgrasser et al. [2024] . Model-collapse refers to a cascade where an AI generates enormous amounts of lower quality synthetic data, which is then used to train a new AI, which in turn generates even lower quality synthetic data due to being trained on lower quality data, and so on. With ML and AI models being more important parts of our decision making process than ever before, it is important to develop new methods to ensure we can get valid statistical inference even in the presence of synthetic data.To this end, this paper demonstrates two main innovations to aid in statistical inference using synthetic data in dynamic contexts. First, using a class of estimators which give valid statistical inference using synthetic and real data points, even when the operating characteristics of the synthetic data generation process are unknown, we illustrate how to incorporate our proposed estimators into dynamic linear models to analyze streaming data. Second, we combined our proposed estimators with Bayesian optimal experimental design to dynamically determine the optimal ratio of real and synthetic data to minimize model standard error. 
setup: 
 Borrowing the setup from Szpiro et al. [2010] , we posit that, at time t, our data is generated from a process is:EQUATIONwhere:EQUATIONHere B ϕ and B σ are vectors of B-splines. We shall say that at time point T -1 , a group of upstream researchers fit a deterministic ML function f (x) (which we shall refer to as the "upstream model") that approximates ϕ(x). These researchers are willing to disseminate f (x), but do not provide any information beyond this, including sample size or error rates. This mirrors the way modern AI models, such as ChatGPT, are shared.Later, a different group of researchers is interested in using a combination of real outcomes and upstream-model-generated synthetic outcomes for use in a downstream statistical model. As an example, this could occur in pragmatic clinical trails where one would like to learn the relationship between a clinical outcome, Y, and a biomedical covariate, X. However, if Y is expensive or difficult to collect, then one may use a previously trained prediction model to generate f (X) and regress that along with the collected Ys upon X [Gamerman et al., 2019 , Williams et al., 2015] .In our paper, our goal will be to describe the linear relationship β(t), between y and x at each time step t ∈ [0, ..., T ]. Formally, our goal is to estimate β t |Y t = {Y t real , Ŷ t syn } where Y t,real = {y 1,real ..., y t,real } and Ŷt,syn = { f (x 1,syn )..., f (x t,syn )}. A class of estimators which can give parameter estimates using a combination of real and black-box generated synthetic outcomes is known as Inference on Predicted Data (IPD) estimators Hoffman et al. [2024a] . Here, we will use a particular IPD estimator known as Prediction Powered Inference (PPI). In Angelopoulos et al. [2023] , the authors demonstrate that one can estimate a linear regression parameter β (as well as other convex estimation problems) via a PPI estimator, βP P I βP P I = βNaive + ∆where βNaive is the parameter from linearly regressing f (x syn ) on x syn and ∆ is the parameter from linearly regressing (y real -f (x real )) on x real . Angelopoulos et al. [2023] and Angelopoulos et al. [2024] show that these estimators not only give unbiased estimates but can lead to tighter standard errors compared to traditional estimators.Here we describe the update scheme for βP P I at time point t, which we denote βP P I,t . We shall assume here that Φ and Ψ are known (with full generality in the complete paper). Based on our data generating equation, we have:p(β t |Y t ) = p(β t |Y t ) (4) = p(β t |y real,t , y syn,t , Y t-1 ) (5) ∝ p(y real,t , y syn,t |β t )p(β t |, Y t-1 ) (6) = p(y real,t |β t )p(β t |, Y t-1 ) * p(y syn,t |β t )p(β t |, Y t-1).(7) p(y real,t |β t ) and p(y syn,t |β t ) are easy to sample from as we get them directly from the data generating equation. To estimate p(β t |Y t-1 ) in an online fashion using a PPI estimator, we employ the dynamic structure used in McCormick et al. [2011] and assume that:EQUATION∼ N ( βNaivet-1 + ∆t-1 , ΣNaive t-1+ Σ∆,t-1 ).(9) This yields a Kalman filter based prediction equation:EQUATION) where:R P P I t-1 = ΣNaive t-1 /λ N aive + Σ∆ t-1 /λ ∆ λ N aive and λ ∆ are fixed forgetting factors and are set to be less than 1. Estimation of the forgetting factors can be done via a model selection such as was done in McCormick et al. [2011] .To estimate βNaive,t and ∆t , we can use the fact that they are estimated on separate datasets which allows us to decompose the posterior into the product of two independent linear regression. Further improvements to estimating these terms can be done via the approach described in Hofer et al. [2024] .EQUATIONTo demonstrate this updating scheme, we simulated a single iteration of the updating scheme described above. At time t = 2, we generated 5000 datapoints from the data generating process with β = 1, x is a standard normal, and B ϕ is a B-spline for x with 3 degrees of freedom using the bs function from the splines package in R R Core Team [2021] and Φ = [2, 4, -10]. The upstream prediction model, f was set to be a simple linear regression with intercept. Note that this is purposefully a misspecified model and thus should not give unbiased estimates of β. At the next time step, t = 3, 20 real samples and 100 synthetic samples were generated from the data generating process and a forgetting factor of 0.0001 and 0.001 for λ N aive and λ ∆ respectively (the choice of very small values of forgetting factors is due to the large discrepancy between sample sizes and because in this simulation we are only doing one iteration. In a more realistic example with more steps, this discrepancy will be smaller and the forgetting factors will be much larger). Figure 1 illustrates 1000 posterior draws from ∆, βNaive and βP P I . Because f was trained in the past and is misspecified, the posterior of βNaive is very off, with a mean of 0.76 and standard error of 0.21. On the other hand, βP P I is much closer to the true value of 3 with a mean of 2.7 and a standard error of 0.39, which leads to an overall lower MSE of 2.25 versus 0.491. This underestimation bias in our final estimator is because our proposed estimator is combining information from the parameter estimate in the past and the present. In the case of this simulation, at t = 2, β was smaller.A comprehensive parameter tuning procedure to determine λ naive and λ ∆ has the potential to give even more accurate MSEs for β P P I .A unique aspect of doing dynamic regression using IPD-based estimators is that one is performing inference using a combination of synthetic and real data. However, as noted in Hoffman et al. [2024b] as one diverges (either in time or in predictive accuracy) from the original upstream model, the optimal ratio of real and synthetic data to minimize the standard error of a PPI estimator changes.To account for this, we propose at each time point to employ techniques from Bayesian optimal experimental design are used to determine the optimal ratio of real to synthetic data.Specifically, we shall assume that at each time point, the researchers are able to spend their limited budget C t in two ways. They can spend it on collecting just x values, each with a cost of c syn (which we shall refer to as "synthetic data") or spend c real and collect both (x, y) (which we shall refer to as "real data").To determine the best ratio of real to synthetic samples, we shall use techniques from Bayesian optimal experimental design Ryan et al. [2015] , Lindley [1956] . A common setup in Bayesian optimal experimental design is to choose a design that maximizes the expected information gain at each time point. In our case, the space of our decisions are parameterized by ζ t ∈ (0, 1) which is the fraction of budget, C t allocated to collecting real (each at cost c real,t with C t /c real,t samples if the entire budget was spent on real data) and synthetic samples (each at cost c syn,t with C t /c syn,t samples if the entire budget was spent on synthetic data). Following the setup, this yields a compact expression for estimating the Expected Information Gain (EIG) Rainforth et al. [2018] :EQUATIONAnd Foster et al. [2019] illustrated that this double integral is well approximated via the nested-Monte Carlo approach of which each piece is estimable based on the derivations above:EIG(ζ t ) ≈ 1 N N i=1 p(Y t n |β n,0 ) 1 M M i=1 p(Y t n |β n,m ) , β n, * ∼ p(β|Y t-1 ), Y t n ∼ p(Y t n |β n,0 )Choosing, at each t, the sample size ratio ζ t which maximizes our estimated EIG allows us to iteratively determine which type of sample-cheaper, but less reliable, synthetic data, or more expensive, but reliable, real data-would be most beneficial. An interesting point to note about this trade-off is that since, in the PPI estimator, the real samples help better estimate the bias of β while the synthetic samples help estimate the variance of β, this procedure to balance between real and synthetic samples using ζ t can also be thought of as an iterative bias-variance trade off. 
real data applications: 
 Finally, to demonstrate the utility of this approach, we will apply this approach to two different application domains. In the first, we consider the problem of ocean going vessel fuel efficiency estimation. A major problem for shipping companies is that while many accurate physics-based models to estimate fuel use exist, due to issues such as wear and tear and barnacle buildup, the predictions from such models decay in accuracy the longer the ship is out of port Fan et al. [2022] . Thus, we should expect that such models are most useful close to port and progressively less useful the longer the ship has been out. By treating physics-model predicted efficiency as our synthetic data and ship-board sensor data as our real data, we can determine the optimal balance of the two to get the most accurate estimates of total fuel efficiency.In the second context, we take a sociological problem of public opinion surveying. When designing such a survey, one must typically make a decision as to whether one would prefer the more expensive, but more accurate in-person surveys, or the cheaper but more prone to bias internet surveys Wu et al. [2022] . Inspired by Egami et al. [2023] , we will demonstrate the optimal budget allocation between online and in-person polling to track public opinion on large studies with online and in-person components such as the American Community survey Bureau [2020] .Workshop on Bayesian Decision-making and Uncertainty, 38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix:  


Decision: Scores are mixed, one positive and one negative. The negative review is complaining about lack of comprehensiveness, but at the workshop stage I think this is a lesser criticism compared to the positive review, which comes from a seasoned expert. I therefore recommend acceptance.

Reviews: review: title: title: Review of Bayesian Optimal Experimental Design of Streaming Data Incorporating Machine Learning Generated Synthetic Data
main_review: main_review: The paper extends existing methods to perform inference incorporating synthetic data to the context of dynamic models. By leveraging prediction-powered inference estimators, the authors extend this approach to dynamic linear models and use optimal experimental design to iteratively determine the optimal amount of synthetic data to be incorporated into the model based on a budget constraint.

I found the paper interesting, and I believe that the topic is important.
title: title: Review of the online inference of model parameters and Bayesian optimal experiment design
main_review: main_review: This work addresses a very helpful direction of performing online statistical inference with streaming data and optimal ratio of real and synthetic data. It also discusses about several real-data applications. It adapts the methods so that it can perform Kalman filtering-based online inference.

However, I think there are a lack of enough data simulation and also missing real-data analysis results in the paper. For simulations, it only includes a set of simulation. For example, it didn't conduct enough simulation analysis including varying model parameters, sample sizes or forgetting factors. 

Also for the part of optimal sample size ratio of real and synthetic data, it proposes the method but didn't have corresponding simulation analysis for the validation of the optimal size ratio decision. 

It discusses about some real-data application examples, but the paper didn't conduct real-data analysis and demonstrates the performance of the proposed method.

Besides, I think a potential improvement may be to add priors for the forgetting rates and get the posterior inference of the rates.