# Text Mining of Stocktwits Data for Predicting Stock Prices
## 5 Results
The highest score achieved was with the percentage change method two labels. Percentage change with two labels has consistently performed better than the percentage change three labels. The binary classification has persistently generated similar results to percentage change two labels. BERT outperforms in 1-year data, whereas Naive Bayes has achieved the highest F1 score in 2 years FAANG data. Traditional machine learning models perform average with varying methods of labelling. Naive Bayes, Logistic Regression and Random forest were trained using cross-validation (k-fold). Tables 6 and 7 below show the results for four different machine learning traditional methods. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Model**} & **Binary Classification** & \
\begin{tabular}{c} **Percentage Change-2** \\ **Labels** \\ \end{tabular} &
\begin{tabular}{c} **Percentage Change-3** \\ **Labels** \\ \end{tabular} \\ \cline{2-4}  & \multicolumn{3}{c}{**1 Year**} \\ \cline{2-4}  & **F1 Score-Macro** & **F1 Score-Macro** & **F1 Score-Macro** \\ \hline
**Naive Bayes** & 0.57 & **0.58** & 0.41 \\ \hline Logistic Regression & 0.51 & 0.5 & 0.31 \\ \hline Random Forest & 0.54 & 0.55 & 0.34 \\ \hline Gradient Boosting & 0.34 & 0.38 & 0.11 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Traditional machine learning model results with K-fold (1 year). \begin{table}
\begin{tabular}{c} **F1** \\ **Score-Macro** \\ \end{tabular} & **Accuracy** \\ \hline
**Naive Bayes** & **0.41** & **49\%** & **0.41** & **49\%** \\ \hline
**Gradient Boosting** & 0.11 & 20\% & 0.1 & 19\% \\ \hline
**Logistic regression** & 0.3 & 48\% & 0.32 & 47\% \\ \hline
**Random Forest** & 0.33 & 33\% & 0.34 & 34\% \\ \hline
**BERT** & 0.33 & 48\% & 0.37 & 50\% \\ \hline
**FinBERT** & 0.32 & 33\% & 0.36 & 36\% \\ \hline
**FinALBERT** & 0.29 & 47\% & 0.32 & 46\% \\ \hline
**CNN with Word2 Vec** & 0.26 & 42\% & 0.26 & 34\% \\ \hline
**CNN with BERT** & 0.17 & 23\% & 0.2 & 44\% \\
**embeddings** & 0.21 & 46\% & 0.11 & 20\% \\ \hline
**Word2 Vec with BiLSTM and Attention** & 0.21 & 47\% & 0.2 & 44\% \\ \hline
**Fasttext with BiLSTM** & 0.21 & 46\% & 0.21 & 44\% \\ \hline
**FinALBERT with BiLSTM and Attention** & 0.21 & 46\% & 0.21 & 44\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Results for Percentage change (Positive, Neutral, and Negative). On comparison of the results, it was evident that cross-validation did not aid in increasing the F1 score for any of the significant model. There was a minimal increase in the F1 scores. Naive Bayes continuously achieved the highest score among the other models. On the other hand, Gradient boosting regressor model performance kept degrading and with Percentage change three labels it had given the lowest scores. The FinALBERT model was also fine-tuned with FinancialPhraseBank dataset. This dataset was also used in FinBERT for testing the performance of their model. We have experimented on sentences with 100% agreement. It consists of 2264 English sentences selected from financial news. These sentences are manually annotated by 16 professionals from the business and finance world. Two configurations were used to fine-tune the model. Two labels (Positive and Negative) and three labels (Positive, Neutral, and Negative) were used. Table 8 depicts the results achieved. Both the label types attained similar F1 macro average score. Unlike our labelling methods, the labels in FinancialPhraseBank are based on sentiment due to which the difference in result is definitive. To complete the study, the last step was to make a prediction based on proposed FinALBERT model. As the FAANG dataset for one year is fine-tuned with FinALBERT, the dataset was divided in 90:10 ratio for training and testing. For prediction, the two weeks of data for each company is chosen. Predictions are made solely on percentage change labelling method with two labels, i.e., Positive and Negative.