## 5 Application: IoT Temperature Time Series
### Intervention Impact Analysis and Interpretation  
For each alert event in this experiment, we are given a label indicating whether a panel of domain experts believed the customer took corrective action based on visual inspection of time series plots similar to Figure 1. If a majority of experts thought that action had been taken, an alert was labeled as \"Action\", indicating that appreciable customer action likely occurred; otherwise, it was labeled as \"No Action\". In the absence of observed labels, this is the closest approximation to the ground truth that we have and constitutes the benchmark we will compare against.  
Due to this inherently biased labeling scheme, the labels we have are more aptly described as \"pseudo-labels\" in that they do not represent an objective truth. For alerts that experts labeled as \"No Action\", we find that there is a high degree of correspondence between the model and expert labels, as indicated in Table 4. This result is unsurprising, as its quite obvious to both experts and the models when no action has been taken because the observed internal temperature will remain flat or even decrease after the alert. Conversely, for instances labeled \"Action\" by experts, every model is likely to disagree, as shown in Table 4. This is attributable to the fact that whenever post-alert temperatures experience a sharp positive increase, human labelers are biased towards labeling it an action, regardless of the historical time series behavior or its correlation with the exogenous predictor. For example, Figure 4 shows an alert event labeled as \"Action\" by domain experts; however, there is clearly a strong, positive correlation between the internal and external temperatures that appears to insigate the increase in post-alert internal temperature. Furthermore, the post-alert increase in the response variable is quite modest and is congruent with pre-intervention temperature levels. Unsurprisingly, every possible combination of model and kernel tested returns a decision of \"No Action\" for this alert.  
Indeed, the results of the impact analysis are congruent with our goals in that the models are far more conservative in assessing customer intervention. Expert opinion is that customers typically do not take action, so it is desirable to have models that require a large shift in post-alert behavior in order to declare an alert event as having been addressed. To  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **RMSE** & **sMAPE** & **CRPS** & **CPU Time** \\ \hline GP-NARX RBF & 13.456 & 0.046 & 10.705 & 33.69 \\ GP-NARX Matern 3/2 & 13.605 & 0.046 & 10.882 & 34.88 \\ GP-NARX Matern 5/2 & 13.669 & 0.047 & 10.875 & 34.63 \\ GP-NARX OU & 13.385 & 0.046 & 10.717 & 32.51 \\ \hline GP-RLARX RBF & 15.748 & 0.051 & 12.253 & 741.50 \\ GP-RLARX Matern 3/2 & 15.610 & 0.051 & 12.429 & 889.78 \\ GP-RLARX Matern 5/2 & 15.453 & 0.051 & 12.309 & 954.87 \\ GP-RLARX OU & 14.831 & 0.049 & 11.798 & 907.22 \\ \hline TP-NARX RBF & 13.110 & 0.044 & 10.340 & 31.95 \\ TP-NARX Matern 3/2 & 13.234 & 0.046 & 10.361 & 30.72 \\ TP-NARX Matern 5/2 & 13.073 & 0.046 & 10.361 & 31.27 \\ TP-NARX OU & 13.728 & 0.047 & 10.707 & 28.82 \\ \hline TP-RLARX RBF & 13.666 & 0.046 & 10.886 & 967.20 \\ TP-RLARX Matern 3/2 & 13.149 & 0.046 & 10.481 & 1131.45 \\ TP-RLARX Matern 5/2 & 13.312 & 0.046 & 10.574 & 1129.89 \\ TP-RLARX OU & 13.003 & 0.045 & 10.394 & 1104.85 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Forecast metrics and CPU times.  
that end, the RLARX models yield the best results in that they are both highly unwilling to assume an intervention has been successful without significant evidence.