# Text Mining of Stocktwits Data for Predicting Stock Prices
## 2 Literature Review  
In recent times, there is an increase in the use of deep learning and machine learning approaches on tex mining and financial domain [4, 5, 6]. In this section we review papers and experiments in the field of finance [7, 8]. We also review labelling techniques natural language processing, machine learning and data extracted from a platform such as Twitter, and multiple ways of predicting stock performances based on it [8]. The most relevant reviews and results observed from the literature review are stated below. [PERSON] et al. [9] and few others [8] explain binary classification, which is one of our labelling techniques categorises words as negative and positive, where positive terms were set to 1, and negative terms were set to 0. Percentage change as explained by [PERSON] et al. [10], using a simple statistical function to estimate missing stock prices that were absent for weekends and other holidays when the market is closed using tweets from StockTwits and historical stock prices from Yahoo finance. It assumed that for all previously known value of stock price, \"sprevious\" and next known value to be \"xnext\", the value of missing value \"y\" is calculated as (xprevious+xnext)/2. Based on this percentage change classification on tweets was imposed by labelling tweets of the current day with change close price of the current day and previous day greater than +0.5 as positive, change greater than \(-0.5\) as negative and change between [\(-0.5\), \(+0.5\)] are labelled as neutral. It was necessary to decide on the right set of models for our experiment with financial stock data, so we researched various models that already exist to understand the implementation, and the performance is compared. To understand further about transformer models, we looked into BERT: Bidirectional Encoder Representations from Transformers initiated by [PERSON] et al. [11], which talks about refining fine-tuning-based methods that have proved to obtain new state-of-the-art results on about 11 natural language processing tasks including GLUE (with 7.7% improvement) and SQuAD v2.0 (with 5.1 point improvement). The [PERSON] et al. [12] study on BERT pre-training which was known to be expensive, significantly undertrained and used various datasets of various sizes and introduce RoBERTa to measure the significance of key hyperparameter and training data size that achieved state-of-the-art results on GLUE, RACE and SQuAD. The usage of financial stock Twitter data trained on a BERT model was conducted by [PERSON] et al. [13] where FinBERT was introduced, which is a language model built out of BERT by pre-training it on a financial dataset and fine-tuning the same for sentimental analysis. [PERSON] et al. [13] was able to achieve state-of-the-art on FiQA sentiment scoring and Financial PhraseBank, and the results were compared against ULMFit and ELMo for financial sentiment analysis. GPU/TPU memory limitations are usually observed when there is a surge in model size and slower training period. To solve this problem, [PERSON] et al. [14] introduced two-parameter reduction techniques to the original pre-trained BERT model, naming it to be ALBERT which uses fewer parameters only. The first technique uses the breakdown of large vocabulary embedding matrix into smaller ones to separate the size of hidden layers from that of vocabulary embedding, which will help in rising hidden size without increasing parameter size. The second technique explains a cross-layer parameter sharing to avoid the growth of parameter with the network's depth. As compared to BERT-large, the configuration of ALBERT-xxlarge achieves 30% parameter reduction, considerably improving the performance on SQuAD2 (+3.1%) and SST-2(+2.2%). [PERSON] et al. [10] introduced a system which collects previous financial stock connected tweets and used machine learning algorithms such as Support Vector Machine, achieving an accuracy of 0.8, and Naive Bayes Bernoulli achieving an accuracy of 0.79 after categorising negative and positive sentiments.