## 3 Machine Learning Models: Results and Interpretation
### Discussion of Research Contributions  
Our novel contributions in this paper include (1) integrating multiple data sources, (2) dividing time into multiple regimes, and (3) highly accurate models. This research highlights the benefits of recognizing distinct regimes in order to obtain high predictive accuracy. A regime is a fundamental \"sea change\", which can be difficult to recognize while it is happening. The announcement of a state-wide lockdown, however, is a clear sign that current models need to be set aside and new models developed. Moreover, it may not be necessary to capture the exact moment of transition. We tried several models at slightly different date boundaries between the regimes; it did not make a material difference. The two regimes highlight the importance of implementing strict lockdown policies to discourage movement outside the home and across state lines, which has implications from one regime to the next.  
Our models were Gradient Boosted Machines in both regimes, having 95.2% and 99.2% variance explained in the lockdown and reopen regimes, respectively. The models are independent in that the model for the reopen regime did not rely on information from the lockdown regime in the form of time-lagged variables. Although the importance of the variables shifts substantially from regime to regime, they are modeled with extremely high accuracy. This suggests that although human population behavior does change, the Gradient Boosted Machine algorithm if given high quality relevant data, can adapt. These models constitute a novel and substantial contribution to the COVID-19 literature. These models are well-known machine learning methods [53; 54], and the results show the power of Gradient Boosted Machines models, a particular type of ensemble model [53; 54; 55]. They reduce errors by combining uncorrelated submodels. In so doing, they produce a consensus from them, which tends to outperform any single model. This is the \"wisdom of the crowd\" [56; 57; 58].  
Ensemble models are often considered opaque, black-box models, which are difficult to interpret and understand by human analysts but highly accurate [59]. The computational complexity enables them to obtain extremely high accuracy, but practitioners may nevertheless be reluctant to use them. Practitioners want to understand the model and be able to explain, justify, and communicate it to others. They may reason that unless the black box is at least X% more accurate than a clear-box model, where X is quite large, they cannot justify it. They would rather keep it simple, even if it is far from optimal. In addition, there may be legal, auditing, or regulatory requirements concerning transparency. If the algorithm is audited and it cannot be explained, there may be serious legal or financial consequences [60]. In order to peer inside the black-box, rendering it a grey-box so that we can understand it better, we used variable importance plots and partial dependency plots [55; 60]. By having grey-box explanations we need not blindly trust the ensemble models, because we have maintained a human-in-the-loop to determine their face validity [61; 62].  
Another contribution of this paper is the analysis of a variety of public datasets and the vital importance of GPS data, easily obtained from de-identified data sources, to track the movement of individuals. Such tracking enables the examination of how well populations are following lockdown orders. It also enables the tracking of individuals who may be infected and the notification of nearby individuals. Google and Apple have created such a real-time, privacy-preserving contact tracing technology [63].