FinBERT: Financial Sentiment Analysis with BERT
By: Zulkuf Genc
Tags: Sentiment Analysis, Finance, Bert, Naturallanguageprocessing, Artificial Intelligence

Doğu Tan Aracı, Zulkuf Genc

Shares of food delivery companies surged despite the catastrophic impact of coronavirus on global markets.

The statement above carries a clear positive message about food delivery companies next to a negative one about global markets. It is not an easy task for computers to extract this positive message because it requires understanding of what is positive and negative in the financial context without getting confused with the rest of the sentence.

In this blog post, we will share a simplified story of how we solved this financial sentiment classification problem with Bidirectional Encoder Representations from Transformers (BERT) and improved the state-of-the-art performance by 15 percentage points. In addition, we will also share our production settings, code, data and guidelines for you to train and use our financial sentiment classifier.

Why did we do this?

Prosus is one of the largest technology investors in the world. On a daily basis, we need to sift through an overwhelming amount of information, largely in form of unstructured text data, about the sectors and companies of our interest. Financial sentiment analysis is one of the essential components in navigating the attention of our analysts over such continuous flow of data.

We quickly noticed that naive approaches such as bag-of-words would simply not cut it, as they generally disregard essential context information in the text. We needed more advanced techniques to crack the financial context and understand what is positive and what is negative from the financial point of view.

The current state-of-the-art approach to natural language understanding is using pre-trained language models by fine-tuning them for specific (downstream) tasks such as question answering or sentiment analysis.We followed that recipe and developed FinBERT as a BERT-based language model with a deeper understanding of financial language and fine-tuned it for sentiment classification.

A New Era in NLP

If we were trying to tackle this problem in let’s say 2017, we would have had to come up with a custom model and train it from scratch. In that case we would have needed very large amounts of labeled data for a somehow acceptable performance. Labeled data with good quality isn’t very easy to acquire, especially in a niche domain like finance, where expertise is required and definitely does not come cheap.

Luckily, NLP’s ImageNet moment happened in 2018. Starting with ULMFit, researchers cracked how to do transfer learning efficiently for NLP problems. The idea is simple: First, get textual data that is available in abundance, like Wikipedia. Then, train a language model with that data, which is basically predicting the next word in a sentence. Finally, fine-tune the language model for your task with one or several task-specific layers. The advantage of this approach is that you don’t need a huge dataset for fine-tuning, because the model learns about language during the original language model training. The heavy lifting is already done!

Many other language models followed ULMFit path but used different training schemas, most significant being BERT. BERT was the language model that made the whole concept of pre-training and fine-tuning flow very popular. BERT brought two core innovations to language modelling: (1) It borrowed the transformer (T of BERT) architecture from machine translation, which does a better job of modelling long-term dependencies than RNN-based ones (excellent overview here). (2) It introduced the Masked Language Modelling (MLM) task, where a random 15% of all tokens are masked and the model predicts them, enabling true bi-directionality (B of BERT). Intuitive explanations of how transformers and BERT can be found here and here.

BERT achieved state-of-the-art performance in almost all of the down-stream tasks such as text classification and question answering it was applied to. But more importantly, it was a huge step towards democratisation of NLP. Now that the compute-heavy (Google used 16 TPU’s for 4 days to pre-train BERT) language model training is already done, anyone with a decent computing power could train a very accurate NLP model for their niche task based on a pre-trained language model.

FinBERT

BERT was perfect for our task of financial sentiment analysis. Even with a very small dataset, it was now possible to take advantage of state-of-the-art NLP models. But since our domain — finance is very different from the general purpose corpus BERT was trained on, we wanted to add one more step before going for sentiment analysis. Pre-trained BERT knew how to talk, but now it was time to teach it how to talk like a trader. We took the pre-trained BERT and then further trained it on a purely financial corpus. That corpus is Reuters TRC2, and available upon request here. Our purpose was to achieve better domain adaptation by exposing the model to financial jargon, before fine-tuning for the final task. We further pre-trained BERT using Hugging Face’s excellent library transformers (back then it was pytorch-pretrained-bert) and scripts they provided.

The training steps in FinBERT

Once we had the pre-trained and domain-adapted language model, the next step was to fine-tune it with labeled data for financial sentiment classification. We used the dataset of Financial Phrasebank. It is a very well thought-out and carefully labeled albeit a small dataset. Researchers extracted 4500 sentences from various news articles, which include financial terms. Then 16 experts and master students with finance backgrounds labeled them. They didn’t only report labels but also inter-annotator agreement level for each sentence, which means how many experts labelled as positive, neutral and negative.

Some examples from Financial PhraseBank

Fine-tuning a transformer-based language model for classification is a straight-forward process. A classification layer is added after BERT’s special token [CLS], which is used for sequential tasks like sentence classification or textual entailment. Then the whole model is fine-tuned with classification losses. A visual representation of this structure is on the picture below.