## 2 Materials and Methods
### Case Study
#### 2.3.5 Evaluation  
To evaluate the explanatory power of TIs, benchmark and TI model performances are compared. In order to assess performance, various metrics measuring the discrepancies and similarities between a model's targets and predictions are computed. Firstly, the root mean squared error (RMSE) and the mean absolute error (MAE) are calculated for benchmark and TI models. By analysing the percentage change in these discrepancy metrics, %RMSE and %MAE, overall accuracy improvements are determinable. The %RMSE and %MAE are calculated according to Equation (15). \[\%\text{ME}=100\times\Big{(}\frac{\text{ME}^{Bench}-\text{ME}^{ TI}}{\text{ME}^{Bench}}\Big{)}, \tag{15}\]  
where ME\({}^{Bench}\) is the mean benchmark model error and ME\({}^{TI}\) is the mean TI model error. A positive %RMSE and %MAE indicate that the inclusion of TI features, overall, reduces forecast errors. Secondly, the Pearson correlation coefficient (PCC) is calculated for benchmark and TI models. The PCC is a similarity metric, whose square is equal to the coefficient of determination. By analysing the percentage change in this metric, %PCC, overall improvements in a model's goodness of fit are ascertainable. The %PCC is calculated according to Equation (16). \[\%\text{PCC}=100\times\Big{(}\frac{\text{PCC}^{ TI}-\text{PCC}^{Bench}}{\text{PCC}^{Bench}}\Big{)}. \tag{16}\]  
A positive %PCC indicates that the inclusion of TI features, overall, increases a model's goodness of fit. Thirdly, interquartile and tail squared residuals are computed for benchmark and TI models. Interquartile squared residuals, \(\{(e_{t})^{2}\}_{t=1}^{N_{\text{IQR}}}\), are the forecast errors of interquartile targets, \(\{\{y_{t}\}_{t=1}^{N_{\text{IQR}}}\,|\,Q1\leq y_{t}\leq Q3\}_{t=1}^{T}\), where \(T\) is the test size, and \(Q1\) and \(Q3\) are the lower and upper quartiles respectively. Similarly, tail squared residuals, \(\{(e_{t})^{2}\}_{t=1}^{N_{\text{Tail}}}\), are the forecast errors of tail targets, \(\{\{y_{t}\}_{t=1}^{N_{\text{Tail}}}\,|\,y_{t}<Q1\cup y_{t}>Q3\}_{t=1}^{T}\). Four custom measures of forecasting accuracy, %W, %L, %W,  
Figure 2: Diagrams displaying TI model inputs—lagged prices concatenated with a lagged TI—and outputs. Per Section 2.3.2, \(d\) to \(d-5\) represent the 6-day look-back period, while \(avg\) stands for the averaged 8-week look-back. \(\mathbf{M}\) is a black box representation of ML models. (**a**) Linear and ensemble models; (**b**) Deep models. and \(\bar{\rm L}\) are calculated from both the interquartile and tail errors. %W and %L measure the percentage of squared TI residuals smaller/larger than squared benchmark residuals. \(\bar{\rm W}\) and \(\bar{\rm L}\) measure the mean squared error difference between TI and benchmark model forecasts. By analysing these four metrics, improvements in interquartile and tail accuracies are determinable. Formally, with either the interquartile or tail sets, %W, %L, \(\bar{\rm W}\) and \(\bar{\rm L}\) are calculated according to Equations (17) and (18). \[\%{\rm W}=100\times\frac{n_{w}}{N},\ \ \bar{\rm W}= \frac{1}{n_{w}}\times\sum_{t=1}^{n_{w}}d_{t},\ \ \ \ \forall d_{t}\in\{\{d_{t}\}_{t=1}^{n_{w}}|d_{t}<0\}_{t=1}^{N}, \tag{17}\] \[\%{\rm L}=100\times\frac{n_{l}}{N},\ \ \ \ \bar{\rm L}= \frac{1}{n_{l}}\times\sum_{t=1}^{n_{l}}d_{t},\ \ \ \ \forall d_{t}\in\{\{d_{t}\}_{t=1}^{n_{l}}|d_{t}\geqslant 0\}_{t=1}^{N}, \tag{18}\]  
where \(d_{t}=(e_{t}^{TI})^{2}-(e_{t}^{Bench})^{2}\), and \(N\) is either \(N_{IQR}\) or \(N_{Tail}\). A %W higher than 50% and a negative \(\bar{\rm W}+\bar{\rm L}\) are indicative of more accurate TI model forecasts. Finally, to determine the statistical significance of any performance improvements, one tailed Diebold-Mariano (DM) tests [38] with a mean squared error loss function are conducted. The DM test statistically evaluates differences between two models' forecasts by comparing their residuals, in our case \(\{e_{t}^{Bench}\}_{t=1}^{T}\) and \(\{e_{t}^{TI}\}_{t=1}^{T}\).