## 3 Data Engineering
### Feature Engineering  
We face two issues with the data matrix. One, there are missing values. Since over the past 54 years many stocks have entered and exited the S&P 500 index, we had to account for the fact that a security would have influenced the movement of the index only during a certain period. Also, even for just the \(L\)-day history in the feature set, some columns may have missing data as stocks may have left the index and others may have entered it. Two, if we use all 500 stocks' returns for a day and we have a look back period of \(L=30\) days, then our feature set will have 15,000 variables and the training sample will only be of size \(N=5000\) or \(N=10,000\) (the two training sample lengths we experimented with). Both these issues are resolved by reducing the dimension of the feature set. Our remediation entails using only a few return percentiles each day, which results in a drastic reduction in dimension of the feature set. We convert the daily returns of the 500 underlying component stocks to \(C=19\) percentiles. The percentiles considered are at the 1, 2, 3, 5, 10, 15, 20, 30, 40, 50, 60, 70, 80, 85, 90, 95, 97, 98, and 99 percentile levels. This results in a dataset that has the sign of the daily return (direction) of the S&P as the (binary) dependent variable, and 19 percentiles as the explanatory variables per day, thereby reducing daily data dimension from 500 to 19. These percentiles allow us to capture the entire distribution through a discrete approximation, and places each day's data into a standard data structure for the probability distribution of returns. This reorganization of the data  
Figure 2: Depiction of the source data, prices and returns. The top panel shows the sparse data on prices and the lower panel shows the data converted to returns, using the formula: \(R_{t}=(S_{t}-S_{t-1})/S_{t}\), where \(R,S\) stand for returns and stock prices, respectively. One row of data is lost when converting prices to returns. Note that for the index, we convert the data into sign of movement, where the value is 1 if the index moved up, else it is 0. The “NA” values show days for which there is no data observation. allows us to abstract from the fact that the underlying stocks in the index are changing, yet considers the influence of their returns in predicting the forward movement of the S&P. It allows modeling the return of the index based on all its 500 component stocks at any point in time, reduced to a summarized set of \(C=19\) values. It also results in a tidier dataset for the prediction exercise that is the focus of our analysis. In addition, there are no missing values in the transformed data matrix. With a look back period of 30 days, for example, each row of the dataset now has size 570 (\(C\times L=19\times 30\)) values of stock return percentiles. The dataset is no longer sparse and is shown in Figure 3.  
More formally, as shown in Figure 4, this feature creation essentially lines up \(L\) days of stock return history from day \((t-1)\) back to day \((t-L)\), placed side by side in the same row for a given date. Therefore, there will be a block of columns for \((t-1)\), another block for day \((t-2)\), and so on, until a final block of columns for day \((t-L)\), each column representing one of \(C\) stock return percentiles for one of the \(L\) preceding days. Therefore, the size of the data input for training the algorithm is \(N\) rows of \(C\times L\) features. The corresponding training labels will be a single column of values, which are \(1\) if the index moved up on date \(t\), else zero. Figure 4: Data for prediction analysis after rearrangement. The label is the sign of the S&P movement, shown in the last column. The feature set comprises \(L\) sets of \(C=19\) percentiles for \(L\) lagged days. Figure 3: Depiction of the returns for each day in percentiles, where 500 stocks returns are reduced to 19 values for the following percentiles: 1, 2, 3, 5, 10, 15, 20, 30, 40, 50, 60, 70, 80, 85, 90, 95, 97, 98, and 99.