## 2 Eruption Forecasting Algorithm
### Cross Validation  
Binary classification problems attempt to categorize the outcome of an event into one of two categories, either true \((1)\) or false \((0)\). This process can result in one of four possible outcomes that are defined as follows:  
* _True Positive (TP)_: Predicted and actual results are 1 (Valid Detection)
* _False Positive (FP)_: Predicted result is 1, but actual result is 0 (False Alarm)
* _False Negative (FN)_: Predicted result is 0, but actual result is 1 (Missed Detection)
* _True Negative (TN)_: Predicted and actual results are 0 (Valid Non-detection)  
The quality of a binary classifier is assessed through a receiver operating characteristic (ROC) analysis. A ROC curve is generated by plotting the prediction algorithm's true positive rate (TPR or sensitivity) versus its false positive rate (FPR or 1-specificity). These parameters are defined as  
\[TPR(t)=\frac{TP(t)}{TP(t)+FN(t)}=Sensitivity(t) \tag{22}\]  
\[FPR(t)=\frac{FP(t)}{TN(t)+FP(t)}=1-Specificity(t) \tag{23}\]  
and are both a function of the decision (detection) threshold, \(t\). The algorithms predictive capability is quantified by computing the \"area under the ROC\" (AUROC) curve [56,66]. Its predictive power increases as the AUROC approaches 1.0 and decreases as the AUROC approaches 0.0. An AUROC of 0.5 is equivalent to randomly selecting an outcome. The objective of any classification algorithm is to maximize the AUROC. The accuracy and precision of the algorithm represent a measure of its ability to consistently estimate the true outcome of an event. These metrics are estimated using  
\[Accuracy(t)=\frac{TP(t)+TN(t)}{TP(t)+TN(t)+FP(t)+FN(t)} \tag{24}\]  
\[Precision(t)=\frac{TP(t)}{TP(t)+FP(t)} \tag{25}\]where both expressions are a function of \(t\).  
The prediction power of the forecasting algorithm is characterized using a bootstrapped, leave-one-out (LOO), cross validation methodology. This process requires the removal of one sample from the training data, regeneration of the statistical model using the remaining data, and prediction of the outcome of the removed sample via the new model. Cross validation for each forecasting stage was conducted using 50,000 bootstrapped datasets. This is repeated for each sample in the training set for a collection of detection thresholds that range between 0.0 and 1.0. If the resulting probability is greater than or equal to the threshold, the outcome of the event is declared to be true. If the probability is less than the threshold, the outcome of the event is declared to be false. Since the outcome of each of the training events is known, the number of TP, TN, FP, and FN detections can be determined as a function of the detection threshold and plotted in ROC space.  
ROC curves for the intrusion, eruption, and intensity event tree nodes are shown in Figures 5-7. In each plot, the blue line represents the nodes forecasting capability and the black line is the ROC resulting from randomly guessing an outcome. Optimum detection thresholds for each stage of the algorithm are determined from the point on the ROC curve that is closest to the point of perfect classification (0,1), which represents the best trade-off between the TPR and FPR. AUROC estimates for the intrusion, eruption, and intensity nodes are 0.78, 0.81, and 0.80, respectively. These results suggest the algorithm has good forecasting capabilities.  
Figure 6: Receiver Operating Characteristics for the eruption event tree node. The AUROC value of approximately 0.81 suggests this node will have fair to good predictive capabilities. TPR, FPR, accuracy, and precision estimates of 75%, 21%, 78%, and 74% are obtained at the optimum detection threshold, (0.47).  
Figure 7: Receiver Operating Characteristics for the intensity event tree node. The AUROC value of approximately 0.80 suggests this node will have fair to good predictive capabilities. TPR, FPR, accuracy, and precision estimates of 73%, 19%, 78%, and 70% are obtained at the optimum detection threshold, (0.21).