NLP in the Financial Market — Sentiment Analysis | by Yuki Takahashi | Towards Data Science

Deep learning in Computer Vision has been successfully adopted in a variety of applications since a pioneer CNN called AlexNet on ImageNet in 2012. On the contrary, NLP has been behind in terms of the deep neural network utilisation. A lot of applications which claim the use of AI often use some sort of rule-based algorithm and traditional machine learning rather than deep neural networks. In 2018 saw a state-of-the-art (STOA) model called BERT outperformed human scores in some NLP tasks. Here, I apply several models for a sentiment analysis task to see how useful they are in the financial market where I’m from. The code is in jupyter notebook and available in git repo.
NLP tasks can be broadly categorised as follows.
Text Classification — filtering spam emails, categorising documentsWord Sequence — word translation, POS tags, named entity recognitionText Meaning — topic modelling, search, question answeringSequence to Sequence —machine translation, text summarisation, Q&ADialog Systems
Text Classification — filtering spam emails, categorising documents
Word Sequence — word translation, POS tags, named entity recognition
Text Meaning — topic modelling, search, question answering
Sequence to Sequence —machine translation, text summarisation, Q&A
Dialog Systems
Different approaches will be required for different tasks and in most case are the combination of multiple NLP techniques. When developing a bot, the backend logic is often a rule based search engine and ranking algorithms to form a natural communication.
There is a good reason for this. The language has grammar and word orders, which could be better handled by a rule-based approach, while machine learning approach could learn collocations and word similarities better. Vectorisation techniques such as word2vec, bag-of-word help the model to express a text in mathematical way. The most famous examples are:
King - Man + Woman = QueenParis - France + UK = London
The first example describes the gender relationship and the second captures the concept of capital city. In these approaches, however, the context are not captured as the same word is always represented by the same vector in any text, which is not true in many cases. Recurrent Neural Network (RNN) architecture, which uses the previous information from input sequence and handles time series data, performed well in capturing and remembering the context. One of the typical architecture is Long short-term memory (LSTM), which is composed of input gate, output gate and forget gate to overcome the varnishing gradient problem of RNN. The are many improved models based on LSTM, such as bidirectional LSTM to capture the context not only from preceding words but also from backwards. These were good for some specific tasks but not quite in practical applications.
In 2017 saw a new approach to tackle the problem without recurrent nor convolution architecture. Attention is All You Need proposes a transformer architecture, which is encoder-decoder stack based on attention mechanism. Bidirectional Encoder Representations from Transformers (BERT), is a masked language model with multiple encoder stack by Google in 2018, which achieved STOA in GLUE, SQuAD and SWAG benchmarks with a big improvement. There are a number of articles and blogs explaining the architecture, such as the one by Jay Alammar.
Working in the financial industry, I struggled to see the sufficient robust performance in our past R&D of machine learning models on NLP for the production use in trading systems over the past few years. Now that BERT based models are getting matured and easy to use thanks to Huggingface implementation and many pre-trained models have been made public. My goal is to see if this latest development in NLP reaches a good level to use in my domain. In this post, I compare different models on a rather simple task of the sentiment analysis on financial texts as a baseline to judge if it’s worth trying another R&D in a real solution.
Models compared here are:
Rule based approach using LexiconTraditional Machine Learning approach using TfidfLSTM as a Recurrent Neural Network architectureBERT (and ALBERT)
Rule based approach using Lexicon
Traditional Machine Learning approach using Tfidf
LSTM as a Recurrent Neural Network architecture
BERT (and ALBERT)
I took the following two inputs to represent different languages in the industry for the sentiment analysis task.
Financial Headlines as more formal styleTweets from Stocktwits as trader informal talk
Financial Headlines as more formal style
Tweets from Stocktwits as trader informal talk
I will write another post for the latter, so focus on the former data here. It is an example of texts containing more formal financial domain specific languages and I used FinancialPhraseBank by Malo et al. (2014), which consist of 4845 hand labeled headline texts by 16 persons and provided with agree level. I used 75% agreed labels with 3448 texts as training data.
## Input text samplespositive "Finnish steel maker Rautaruukki Oyj ( Ruukki ) said on July 7 , 2008 that it won a 9.0 mln euro ( $ 14.1 mln ) contract to supply and install steel superstructures for Partihallsforbindelsen bridge project in Gothenburg , western Sweden."neutral "In 2008 , the steel industry accounted for 64 percent of the cargo volumes transported , whereas the energy industry accounted for 28 percent and other industries for 8 percent."negative "The period-end cash and cash equivalents totaled EUR6 .5 m , compared to EUR10 .5 m in the previous year."
Please be noted that all the data belong to the sources and people must honour their copyright and the licence terms.
Here are the four models I compared the performance.
Creating domain specific dictionaries is a traditional approach and simple yet strong in some cases where the source is from a particular person or media. Loughran and McDonald Sentiment Word Lists. This list contains more than 4k words which appears on financial statements with sentiment labels. Note: This data requires the license to use for commercial application. Please check their website before use.
## Samplenegative: ABANDONnegative: ABANDONEDconstraining: STRICTLY
I used negative 2355 words and positive 354 words. It includes the word form, so do not perform stemming and lemmatisation on the input. It is important to consider negation for this kind of approach. Words such as not, no, don’t, etc. change the meaning of negative words to positive and here I simply flip the sentiment if one of negate words occurring within three words preceding a positive words.
Then, the tone score is defined as follows and fed into classifiers along the positive/negative counts.
tone_score = 100 * (pos_count — neg_count) / word_count
14 different classifier were trained with default parameters, then used grid search cross validation for hyper-parameter tuning of Random Forest.
The inputs were tokenized by NLTK word_tokenize(), then lemmatised and stop words were removed. Then fed into TfidfVectorizer and classified by Logistic Regression and Random Forest Classifier.
As LSTM is designed to remember long-term memory which expresses the context, used a custom tokenizer to extract alphabetical letters as they are without lemmatisation or stop word removal. Then the inputs were fed into an embedding layer, then two lstm layer. To avoid overfitting, apply dropout as is often the case, then fully-connected layers and finally take the log softmax.
As alternative, also tried GloVe word embedding from Stanford, which is an unsupervised learning algorithm for obtaining vector representations for words. Here, took the pre-trained on Wikipedia and Gigawords with 6B tokens, 400k vocab size and 300 dimentional vectors. Around 90% of words in our vocab were found in this GloVe vocab and the rest are initialised randomly.
I used pytorch implementation of BERT model by transformers from Huggingface. Now (v3) they provide tokenizer as well as encoder which generate text ids, pad masks and segment ids that can be directly used in their BertModel and no custom implementation was necessary for a standard training process.
Similarly to the LSTM model, the output from BERT is then passed to drop-out, fully-connected layters and then apply log softmax. Training the model from scratch is not an option without enough budget for computation resource and also no enough data, so I used pre-trained models and fine tuned. The pre-trained models used as as follows:
BERT: bert-base-uncased
ALBERT: albert-base-v2
Training process with the pre-trained bert looks like this.
First, input data are split to train set and test set at 8:2. The test set is kept untouched until all parameters are fixed and used only once for each model. As the dataset is not large, cross validation is used to evaluate the parameter sets. In addition, to overcome the issue of imbalanced and smaller dataset, Stratified K-Fold Cross Validation is applied for hyper-parameter tuning.
Evaluation is based F1 score as the input data are imbalanced, while the accuracy is also referred.
Grid Search Cross Validation is used for Model A and B, whereas custom Cross Validation is performed for Deep Neural Network models of C and D.
Fine tuned BERT based models clearly outperform the other models after spending more or less similar time in hyper-parameter tuning.
Model A didn’t perform well because the input was too simplified as tone score, which is a single value judging the sentiment, and the random forest model ended up labelling the majority class of neutral to most of data. Simple linear model performed better by just applying threshold to tone score but still quite low in both accuracy and f1 score.
We didn’t balance the input data using methods like under/over-sampling or SMOTE because it can rectify this issue but would deviate from the actual situation where the imbalance exists. Potential improvement to this model is to build a custom lexicon instead of L-M dictionary if the cost can be justified to build a lexicon per problem to solve. More complex negation could also improve the accuracy of the prediction.
Model B was muchbetter than the previous model, but it overfits to the training set with almost 100% of accuracy and f1 score and failed to be generalised. I tried to reduce the model complexity to avoid overfitting but it ended up lower score in validation set. The balancing data could help solve this issue or collect much more data.
Model C produced a similar result to the previous model but not improved much. In fact, the number of training data were insufficient to train the neural network from scratch and needed to a number of epochs, which tends to overfit. The pre-trained GloVe embedding does not improve the result. One possible improvement on this latter model is to train the GloVe using a bunch of text from similar domain such as 10K, 10Q financial statements instead of using pre-trained model from wikipedia.
Model D performed quite well with more than 90% in both accuracy and f1 score in cross validation as well as final test. It correctly classify negative texts at 84% whereas positive texts at 94%, which could be due to the number of inputs but better to look closely to improve the performance further. This indicates the fine tuning of the pre-trained model performs well on this small data set thanks to transfer learning and the language model.
This experiment shows the potential of BERT based model application to my domain where previous models have failed to produce sufficient performance. The result is, however, not deterministic and based on a rather simple manual hyperparameter tuning on free-tier GPU and it could be different depending on the input data and tuning approaches.
It is also worth noting that in practical application, getting the proper input data is also quite important. The model cannot be trained well without data with good quality as often referred to as “garbage in, garbage out”.
I will cover these points next time. All the code used here are available in git repo.