## 3 Machine Learning Models: Results and Interpretation
(R package neuralnet)  
Gradient Boosted Machine: an ensemble of sequential trees to focus on errors of previous trees; able to find interaction effects; uses gradient descent search to rapidly minimize error via arbitrary, differentiable loss function; many trees help to ensure that the local minimum found is the global minimum. (R package LightGBM)  
Random Forest: an ensemble technique to fit a large number of bootstrap-sampled aggregation (bagging) of trees along with a restricted random selection of variables at each tree split. It tends to perform very well but be difficult to interpret. (R package RandomForest)  
We used three standard measures of predictive accuracy in addition to the variance explained (adjusted R-squared) of the training data: Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Symmetric Mean Absolute Percentage Error (SMAPE) [49; 50]. For adjusted R-squared, the range is 0 for no variance explained to 1 for all variance explained. For RMSE, MAE, and SMAPE, smaller is better. Table 3 shows the accuracy generally held when going from training to test partition, regardless of the method used, in the Lockdown Regime. The different accuracy measures are in general agreement, with Gradient Boosted Machine nearly always having the lowest error, indicated in gray. We choose Gradient Boosted Machine as the best method for the Lockdown Regime and show its lift and residual plots in Figure 2. The residuals are small for the entire range of COVID-19 cases (log). In Figure 3, we see the nine variables ranked by relative importance, along with partial dependence plots for the top four: Non-home dwell time, LEX-Out-state, Population density, and LEX-Total. Variable importance is determined by the loss of predictive power if the values of the variable are permuted [51,52]. We focus on the top four variables by illustrating their partial dependence plots, starting with Non-Home Dwell Time. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Rank**} & \multirow{2}{*}{**Method**} & \multicolumn{4}{c}{**Train (80\%)**} & \multicolumn{4}{c}{**Test (20\%)**} \\ \cline{2-9}  & & **adj. R\({}^{2}\)** & **RMSE** & **MAE** & **SMAE** & **RMSE** & **MAE** & **SMAE** \\ \hline
1 & Gradient Boosted Machine & **0.952** & **0.480** & **0.349** & **8.410** & **0.491** & **0.353** & 8.169 \\ \hline
2 & Neural Network & 0.945 & 0.514 & 0.377 & 8.845 & 0.518 & 0.371 & **7.968** \\ \hline
3 & Generalized Additive Model & 0.920 & 0.618 & 0.472 & 10.997 & 0.565 & 0.426 & 9.222 \\ \hline
4 & Ridge Regression & 0.914 & 0.640 & 0.485 & 10.814 & 0.584 & 0.449 & 9.729 \\ \hline
5 & Multiple Regression & 0.900 & 0.690 & 0.531 & 12.042 & 0.643 & 0.499 & 11.067 \\ \hline
6 & Random Forest & 0.865 & 0.804 & 0.612 & 12.491 & 0.794 & 0.606 & 11.972 \\ \hline
7 & Regression Tree & 0.835 & 0.891 & 0.626 & 13.841 & 0.923 & 0.685 & 14.452 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Accuracy Metrics for All Methods in Lockdown Regime. Figure 2: Lift and Residual Plots for Gradient Boosted Machine, Lockdown Regime. In Figure 3, the partial dependence plot (Figure 3a) shows the average amount of time people spent outside their homes. That metric reflects states that were not under a strict lockdown order. Even with closed schools and businesses, most people could leave their homes. As can be seen from the plot, the higher the number of cases, the less time people dwelled outside their homes. This variable tracks the predicted and actual values best in the middle, slightly under-contributing on the low end and over-contributing on the high end of the variable range. As people perceived that it was safer to leave their residence, they were more likely to do so, including the crossing of state borders. The second-ranked variable is LEX-Out-State travel, as seen in the partial dependence plot (Figure 3b). The plot has a shape similar to that of Non-Home Dwelling Time. The smaller the number of cases, the more time spent outside one's state.