# Feature Learning for Stock Price Prediction Shows a Significant Role of Analyst Rating
## 3 Experiment Design
### Approach for Evaluation  
To do a comprehensive evaluation of the performance of the various models and optimise the utility of the model in a real-world scenario, different metrics were chosen for the individual signals and the overall performance of the model as depicted by Table 4.  
For the buy signals, it is important to have a higher precision against recall as we want to make sure if we look at a buy signal produced by our program is truly a buy signal even if we miss out on a few opportunities (i.e., recall, capturing all the signals) to invest here and there. For the sell signals, the opposite might be true as we would want to know about all the times when we should be selling (i.e., recall, capturing all the signals) because if we do not, then we might be heading towards an impending doom without knowing and can lose all profits there. Thus, it does not hurt to sell a bit early (i.e., precision, if captured should be correct) rather than waiting longer and hoping to generate more profits. As for the hold signal, a low precision would mean one becomes vulnerable in situations when they were supposed to either buy or sell but the classifier tells them to hold making them rigid in terms of their portfolio position--consequently leading to missed investment opportunities or even extreme losses. Similarly, for the recall value, one would want to know about all the hold values because if they do not and are recommended to either buy or sell, then in the best case they become extremely sensitive to the slightest of market changes or worse off compromise their entire portfolio due to an incorrect buy or sell. So, for hold signals, it makes sense to pay attention to both these values and the F-1 score would a great metric for such a situation, accounting for both precision and recall. \begin{table}
\begin{tabular}{c c} \hline \hline
**Entity** & **Metric** \\ \hline Buy Signals & Precision \\ Sell Signals & Recall \\ Hold Signals & F1 Score (Micro-Weighted) \\ Overall Model & F1 Score (Micro-Weighted) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Metrics for different signals. \begin{table}
\begin{tabular}{c c} \hline \hline
**Model Used** & **Parameters** \\ \hline Support Vector Machines & Linear Kernel \\ Naive Baes & Gaussian \\ Multomomial \\ Random Forests & No. of trees = 10 \\ \multirow{2}{*}{Decision Trees} & Gini Index \\ \multirow{2}{*}{K Nearest Neighbours} & Entropy Information gain \\ \multirow{2}{*}{Neural Networks} & 5 Neighbours \\ \multirow{2}{*}{Neural Networks} & 7 Neighbours \\ \multirow{2}{*}{Neural Networks} & 9 Neighbours \\ \multirow{2}{*}{Neural Networks} & 2 hidden layers containing 5 and 2 hidden units respectively \\ \hline \hline \end{tabular}
\end{table} 
Table 3: List of models used for experimentation. Being a system that needs to be applied in real life, merely calculating theoretical accuracy of its various signals would not mean anything if it is unable to successfully navigate the real world and generate profit. Thus, the test dataset, generated to evaluate the effectiveness of the classifier in the earlier steps of the evaluation stage, was passed into a custom backtesting library that would feed each days data one at a time to the trained classifier and then make trades based on the recommendations issued by it--thus, mimicking a real-world scenario of consuming data and making trades. As the purpose of the backtesting library was merely to evaluate the effectiveness of the classifier and the process of building an effective classifier was the actual focus of this study, the library and its function were kept fairly straightforward where testing was carried out only for one equity at a time and only one share could be held or traded at any given time. Though the library enabled both buying and shorting of equities and even broker fees were accounted for where it was set to a standard cost of $0.01 per transaction.