LSTM vs BERT — a step-by-step guide for tweet sentiment analysis | by Yuki Takahashi | Towards Data Science

Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
After seeing the competitive result of BERT in the sentiment analysis on financial text, I performed another preliminary study on more informal text as the ultimate goal is to analyse traders’ voice over the phones and chat in addition to the news sentiment. In this post, I let LSTM and BERT analyse a number of tweets from Stocktwit.
Unlike formal financial text, traders voice and chat contains by far informal languages. Traditional rule based models or simple vectorisation techniques such as BoW, Tfidf, word2vec saw poor performance in my past research because of the fact that
a word is often used for different meanings in each context
spelling and sentence is far from grammatically correct language
the balance of grouping words by stemming, lemmatizing, stop words removal, etc. and keeping the original forms is difficult
a dictionary has to be built specifically on this language for a rule based approach, where I failed to generalise enough
LSTM, which has been one of the most famous RNN based model in NLP, performed well. It is largely thanks to the fact that
sentence structure is fairly straightforward — simply process from left to right will suffice
each input text is short and good size of history to remember
the task is rather simple classification
On the other hand, BERT I used here was pretrained on wikipedia, where the language is pretty different. Training BERT from scratch was not option due to the limitation of resources. In this situation, would it still be worth trying BERT for better performance than LSTM?
The input text here is taken from Stocktwits as the similar language of traders’ voice here. There are around one million tweets that have been hand-labelled with 0 (negative) to 4 (positive), which have been loaded as messages and sentiments list respectively. Note that actual environments usually need a lot more work to do on preparing the inputs such as the sound recognition, data clean-up, streaming infra. This post skip those steps and start from the point where the data have been loaded.
> ##### Sample input messages ######> print(messages)["$AMZN sick! they’re running a prime flash sale on shares too!", "$AAPL has a good Piotroski-F score of 7.00. This indicates a good health and profitability. https://www.chartmill.com/analyze.php?utm_source=stocktwits&amp;utm_medium=FA&amp;utm_content=PROFITABILITY&amp;utm_campaign=social_tracking#/AAPL?r=fa&amp;key=bb853040-a4ac-41c6-b549-d218d2f21b32", "$FB got rid of this trash today, i admit that bears were right", ...]> print(sentiments)[4, 2, 0, ...]
Before the training, the input texts need preprocessing such as removing URL, ticker symbols, @mentions, symbols etc. Here I simply remove them as it will not be available in voices, but there’re also interesting researches on how to utilise such information such as emoticons and hashtags rather than removing them if the final goal is to analyse the tweet text.
Now the input has been cleaned up as follows.
> ###### Input messages after preprocessing ######> print(preprocessed)["sick they re running a prime flash sale on shares too", "has a good piotroski f score of this indicates a good health and profitability", "got rid of this trash today i admit that bears were right", ...]
The next step is to tokenize the text. Here, I use python NLTK library but also give options to use different ways to see what works best for the input. After a few experiments, I decided to use nltk.word_tokenize() without the lemmatization and stopword removal.
Once the input text is tokenized, we can create a corpus and vocab in the following manner. Word Cloud or bar chart is a good way to quickly view the frequent words in the input. The distribution shows the label is imbalanced having more neutral than other sentiments. It is possible to balance the data by resampling (under- or over-) but here take as they are because this ratio would represent actual occurrence of the sentiment in the tweet stream.
Now that the input data are ready, create the neural network based model and tokenizer for the model.
Use pytorch to create a LSTM based model. The class extends torch.nn.Module and define the layers as embedding → lstm → dropout → dense (fully-connected) → output (softmax). The tokenizer for LSTM is to pad the input to the right or to the left up to the specified maximum length and truncate if the input exceeds the maximum length, designed to be used during the training for each batch instead of preprocess all inputs.
Here I use the Hugging Face implementation of BERT. Simply use their transformers and pre-trained model and tokenizer.
Create a dataset class and data loader for batching. There are many different ways to define them and this is just a very simple solution to be used with the defined tokenizer which returns torch.tensor.
The performance was measured in terms of accuracy, f1 and training time for different input sizes. Use Stratified Shuffle Split from scikit-learn that can perform under-sampling by preserving the label distrubtion, based on given train and test size. In each loop, measure the duration by perf_counter() for completing the training.
Also, define a simple function to return Accuracy and F1 score.
Note that model parameters are defined here when instantiating the model classes, which can be updated according to the input data.
Define the training process as follows:
Loop the training batchesRun the evaluation at every 1/5 of the entire dataOnce an epoch is complete, show the resultEnd if the score is not improved beyond the patience, or start the next epoch
Loop the training batches
Run the evaluation at every 1/5 of the entire data
Once an epoch is complete, show the result
End if the score is not improved beyond the patience, or start the next epoch
AdamW Optimizer and Linear Schedule with Warmup for the learning rate are used but these can be swapped with other options as needed.
In each training batch cycle, tokenize the input messages and move to torch.tensor, perform the feedforward prediction, calculate the loss and back propagate to update the weights. Clipping to avoid exploding gradient problem before the next batch step.
At the end of each epoch shows Confusion Matrix and scores/loss for both training and validation data.
Finally :)
After the first epoch of the smallest samples (n=1,000), the model simply classify all data as ‘Neutral’, which is reasonable as the ‘Neutral’ is the majority class.
Completing the five epochs, it looks to have classified the data almost at random. It starts overfitting to the training data from the third epoch (= validation cycle = 10).
The largest dataset (n=500,000) contains 500 times more data than the first cycle. It performed much better and now classify most of labels correctly. Overfitting from the fourth cycle, so three epochs will be the best.
Three epochs of training the pretrained BERT took almost the same time as the five epochs of the above LSTM model.
BERT shows the similar result but it starts overfitting in third epoch for the largest dataset (n = 500,000).
As shown below, it naturally performed better as the number of input data increases and reach 75%+ score at around 100k data. BERT performed a little better than LSTM but no significant difference when the models are trained for the same amount of time.
In this post, tweets from stockswits are cleaned, tokenized and analyzed to predict the sentiment by a LSTM model as well as a pretrained BERT model.
Given the same resource and time, the pretrained BERT perfomed slightly better than LSTM but no significant difference.
Potentially, training the BERT model from scratch on similar tweets could produce much better result, while the required resources and cost is beyond this study.
Here is a piece of code which handles tweet stream as input and output the sentiment with confidence level, using the trained model above.
Sample output shown as follows.