## 4 Experiments
### Prediction Performance  
We measure accuracy in various ways:  
1. _Forecast period average accuracy (FPAA)_: In this measure we calculate the ratio of the number of days per forecast period that our model predicts the index movement correctly (i.e., accurately predicts that the index went up when it actually went up and vice versa) to the total number of forecast period observations. Next, we take the ratio of the number of periods that our model was over 50% accurate to the total number of forecast periods. For example, suppose over three forecast periods our model was accurate 60%, 60% and 30% of the time. Then, the FPAA would be 66.66%, as our model predicted over 50% accuracy twice in the three forecast periods. This measure gives us a point in time measure of how our model is performing and is labeled as average accuracy in each of the runs discussed below. The measure indicates whether identical trades over the forecast period \(F\) will lead to a gain on average. The FPAA measure is a new measure that we developed for this paper.
2. _Overall accuracy (OA)_: In this measure we focus on the daily measures of accuracy. We take individual day level predictions and compare it to the actuals to generate a confusion matrix for each experiment, from which we calculate accuracy. This measure of accuracy gives us a rolling historical measure of the performance of our model. This measure is the number of correctly predicted days across all days in all experiments.
3. _Precision, Recall, F1, AUC_: These are standard measures for prediction algorithms based on a binary confusion matrix. We recap these here, where \(TP\) is the number of true positives; \(FP\) is false positives; \(TN\) is true negatives; and \(FN\) is false negatives. Precision is \(TP/(TP+FP)\); Recall is \(TP/(TP+FN)\); and \(F1\) is the harmonic mean of precision and recall, i.e., \(2/(1/Precision+1/Recall)\). The area under the ROC curve (AUC) measures the tradeoff between the false positive rate (\(FPR=FP/(FP+TN)\)) and the true positive rate (\(TPR=Recall\)). AUC is one of the most comprehensive measures of prediction performance.  
Figure 8 shows the results for the case when the look back period is 30 days and the forecast period is either 10 or 30 days. In the former we obtain many more non-overlapping rolling periods. This results in 348 non-overlapping forecast periods for \(F=10\), and 115 non-overlapping periods for \(F=30\). Forecast period average accuracy is 64% for \(F=10\) days and 77% for \(F=30\) days. Even when we check how many blocks we had accuracy greater than 52.7%, we arrive at the same forecast period average accuracy of 77%. both well over 50% as can be seen from the number of points in Figure 8 that lie above the 0.5 level on the y-axis. The OA is slightly over 50%, i.e., 53.9% for \(F=10\) days and 52.7% for \(F=30\) days. Given that the percentage of days that the market went up in the sample is 52.7%, this suggests very weak predictability. Hence, using short-term forecasts, we see that markets are efficient when conditioned on the information in large datasets and modern deep learning tools. Even if the forecast period accuracy were exploitable in some way, the levels of accuracy are average but may not be enough (especially after transactions costs) to suggest that market efficiency may not be supported when a larger universe of information is considered in modified weak-form efficiency tests.