## 5 Empirical Results
The case of \(F=1000\) days look-ahead quickly dissipates to the baseline level expected, as the forecast period is very large, and is to be expected if the market is efficient, which there is evidence of (in [14]). Therefore, it is more interesting to examine how the OA declines as the forecast period is extended from 10 days to 30 days. Figure 9: Histograms of the results from all experiments, where the test data was in-sample (IS). The four plots are, reading left to right, top to bottom, the cases when \(F=10,\ 30,\ 1000,\ 5000\) days, respectively. The red line is at the 52.7% accuracy cut off. Figure 11: Histograms of the results from all experiments, where the test data was for the Nonstationary out-of-sample experiment (case NS). The three plots are the cases when \(F=10\), \(30\), \(1000\) days, respectively. The red line is at the 52.7% accuracy cut off. Figure 10: Histograms of the results from all experiments, where the test data was for the Stationary out-of-sample experiment (case OS). The three plots are the cases when \(F=10\), \(30\), \(1000\) days, respectively. The red line is at the 52.7% accuracy cut off. First, in all three experimental conditions, OA declines as we extend the forecast period from 10 to 30 days. This suggests that recent history does matter in prediction, and is also intuitive, as the joint statistical distribution of market stocks is likely to be stationary for small look-ahead windows. We examine this phenomenon for the OS case with additional granularity, taking the forecast period to be in the \(\{5,10,15,20,25,30\}\) days, results shown in Table 2. The forecast accuracy declines (albeit slightly) as we increase the forecast period, confirming that a small amount of the loss of predictability may be attributed to nonstationarity. Second, we see that the difference in prediction accuracy (\(OA\approx 0.58\)) between the IS case, stationary case, and nonstationary case is small for the case when \(F=10\) days. The suggests that the lack of predictability here (\(1-OA=0.4\)) comes from randomness and not nonstationarity. Third, for the \(F=30\) days case, the difference between the three cases is also small, but the difference between the IS case and the two OS cases is bigger than for the \(F=10\) case. We may attribute this to the effect of nonstationarity. Fourth, we see from the \(F=1000\) days case, that the OA has dropped back to the baseline of 0.527, as the full effect of nonstationarity and randomness erase all predictability. Fifth, in both, the \(F=10,30\) cases, the accuracy level is higher than the baseline percentage number of days for which the market goes up (52.7%). Yet, it is hard to argue that the predictability is sufficient to generate low-risk profit after transactions costs. In fact, even ignoring transactions costs, note that in no case was the OA different from the baseline expected percentage of \"up\" days of 0.527 by more than 0.5 standard deviations, see Table 1. Clearly, there is no statistically significant evidence of superior prediction ability. Hence, these experiments confirm, with large-scale testing, that markets are efficient for an investigation of predictability of the S&P 500 index. Sixth, the verdict that markets are efficient (for a test of broad index prediction as undertaken here) is further supported by the poor results on other metrics such as precision, recall, AUC, and F1 scores. We see an imbalance where precision is average, but recall is very high. Therefore, the algorithm misses more down ticks in the market but captures most of the upticks. With such an algorithm, one would act aggressively on up signals and ignore most down signals. The low average AUC scores further confirm that markets are efficient. In order to assess whether simpler techniques might in fact yield better results, we fit a logistic regression model to the dataset as well.