FINBERT:  FINANCIAL SENTIMENT ANALYSIS   WITH PRE-TRAINED LANGUAGE MODELS

1 INTRODUCTION . With unprecedented amount of textual data being created every day , analyzing large bodies of text from distinct domains like medicine or finance is of the utmost importance . Yet it is more difficult to apply supervised NLP methods , like text classification , in these domains than it is for more general language . The difficulty comes from two factors : 1 ) The most sophisticated classification methods that make use of neural nets require vast amounts of labeled data and labeling domain-specific text snippets requires costly expertise . 2 ) The NLP models trained on general corpora are not well-suited to supervised tasks since domain-specific texts have a specialized language with unique vocabulary and expressions . NLP transfer learning methods look like a promising solution to both of the challenges mentioned above , and are the focus of this paper . The core idea behind these models is first training a language model on a very large corpus and then initializing down-stream models with the weights learned from the language modeling task . The initialized layers can range from the single word embedding layer Peters et al . ( 2018 ) to the whole model Howard & Ruder ( 2018 ) . This approach should reduce the size of the required labeled data since language models learn the language syntax and semantic in an unsupervised way on a very large unlabeled corpora by predicting the next word . By further pretraining a language model on a domain specific unlabeled corpus , the model can learn the semantic relations in the text of the target domain , which is likely to have a different distribution than a general corpus . In this paper , we explore the effectiveness of using and fine-tuning a pre-trained language model , BERT Devlin et al . ( 2018 ) , in financial sentiment classification using the Financial PhraseBank created by Malo et al . ( 2014 ) and FiQA Task-1 sentiment scoring dataset in Maia et al . ( 2018b ) . The main contributions of this paper are the following : • We introduce FinBERT , which is a language model based on BERT for financial NLP tasks . We evaluate FinBERT on two financial sentiment analysis datasets , where we achieve the state-of-the-art on FiQA sentiment scoring and Financial PhraseBank . • We implement two other pre-trained language models , ULMFit and ELMo for financial sentiment analysis and compare these with FinBERT . • We conduct experiments to investigate several aspects of the model , including : effects of further pre-training on financial corpus , training strategies to prevent catastrophic forgetting and fine-tuning only a small subset of model layers for decreasing training time without a significant drop in performance . 2 FINBERT . BERT ( Devlin et al. , 2018 ) is a language model that consists of a set of transformer encoders ( Vaswani et al. , 2017 ) stacked on top of each other . It defines the language modeling in a novel way . Instead of predicting the next word given previous ones , BERT ” masks ” a randomly selected 15 % of all tokens . With a softmax layer over vocabulary on top of the last encoder layer the masked tokens are predicted . A second task BERT is trained on is ” next sentence prediction ” . Given two sentences , the model predicts whether or not these two actually follow each other . Following the previous work ( Howard & Ruder , 2018 ) on the effectiveness of further pre-training a language model on a target domain , we experimented with two approaches : The first is pre-training the model on a relatively large corpus from the target domain . For that , we further pre-train a BERT language model on a financial corpus . The second approach is pre-training the model only on the sentences from the training classification dataset . Although the second corpus is much smaller , using data from the direct target might provide better target domain adaptation . Sentiment classification is conducted by adding a dense layer after the last hidden state of the [ CLS ] token . This is the recommended practice for using BERT for any classification task ( Devlin et al. , 2018 ) . Then , the classifier network is trained on the labeled sentiment dataset . An overview of all the steps involved in the procedure is presented on figure 2 . While the focus of this paper is classification , we also implement regression with almost the same architecture on a different dataset with continuous targets . The only difference is that the loss function being used is mean squared error instead of the cross entropy loss . As it was pointed out by Howard & Ruder ( 2018 ) , catastrophic forgetting is a significant danger with this fine-tuning approach . Because the fine-tuning procedure can quickly cause model to ” forget ” the information from language modeling task as it tries to adapt to the new task . In order to deal with this phenomenon , we apply three techniques as it was proposed by Howard & Ruder ( 2018 ) : slanted triangular learning rates , discriminative fine-tuning and gradual unfreezing . Slanted triangular learning rate applies a learning rate schedule in the shape of a slanted triangular , that is , learning rate first linearly increases up to some point and after that point linearly decreases . Discriminative fine-tuning is using lower learning rates for lower layers on the network . Assume our learning rate at layer l is α . Then for discrimination rate of θ we calculate the learning rate for layer l− 1 as αl−1 = θαl . The assumption behind this method is that the lower layers represent the deep-level language information , while the upper ones include information for actual classification task . Therefore we fine-tune them differently . With gradual freezing , we start training with all layers but the classifier layer as frozen . During training we gradually unfreeze all of the layers starting from the highest one , so that the lower level features become the least fine-tuned ones . Hence , during the initial stages of training it is prevented for model to ” forget ” low-level language information that it learned from pre-training . 3 EXPERIMENTAL SETUP . 3.1 DATASETS . In order to further pre-train BERT , we use a financial corpus we call TRC2-financial . It is a subset of Reuters ’ TRC21 , which consists of 1.8M news articles that were published by Reuters between 2008 and 2010 . We filter for some financial keywords in order to make corpus more relevant and in limits with the compute power available . The resulting corpus , TRC2-financial , includes 46,143 documents with more than 29M words and nearly 400K sentences . The main sentiment analysis dataset used in this paper is Financial PhraseBank2 from Malo et al . 2014 Malo et al . ( 2014 ) . Financial Phrasebank consists of 4845 english sentences selected randomly from financial news found on LexisNexis database . These sentences then were annotated by 16 people with background in finance and business . The annotators were asked to give labels according to how they think the information in the sentence might affect the mentioned company stock price . The dataset also includes information regarding the agreement levels on sentences among annotators . The distribution of agreement levels and sentiment labels can be seen on table 1 . We set aside 20 % of all sentences as test and 20 % of the remaining as validation set . In the end , our train set includes 3101 examples . For some of the experiments , we also make use of 10-fold cross validation . FiQA Maia et al . ( 2018b ) is a dataset that was created for WWW ’ 18 conference financial opinion mining and question answering challenge3 . We use the data for Task 1 , which includes 1,174 financial news headlines and tweets with their corresponding sentiment score . Unlike Financial Phrasebank , the targets for this datasets are continuous ranging between [ −1 , 1 ] with 1 being the most positive . Each example also has information regarding which financial entity is targeted in the sentence . We do 10-fold cross validation for evaluation of the model for this dataset . 3.2 BASELINE METHODS . For contrastive experiments , we consider baselines with three different methods : LSTM classifier with GLoVe embeddings , LSTM classifier with ELMo embeddings and ULMFit classifier . It should be noted that these baseline methods are not experimented with as thoroughly as we did with BERT . Therefore the results should not be interpreted as definitive conclusions of one method being better . We implement two classifiers using bidirectional LSTM models . In both of them , a hidden size of 128 is used , with the last hidden state size being 256 due to bidirectionality . A fully connected feed-forward layer maps the last hidden state to a vector of three , representing likelihood of three 1The corpus can be obtained for research purposes by applying here : https : //trec.nist.gov/data/reuters/reuters.html 2The dataset can be found here : https : //www.researchgate.net/publication/251231364 FinancialPhraseBank-v10 3Data can be found here : https : //sites.google.com/view/fiqa/home LPS ( Malo et al. , 2014 ) , HSC ( Krishnamoorthy , 2018 ) and FinSSLX ( Maia et al. , 2018b ) results are taken from their respective papers . For LPS and HSC , overall accuracy is not reported on the papers . We calculated them using recall scores reported for different classes . For the models implemented by us , we report 10-fold cross validation results . labels . The difference between two models is that one uses GLoVe embeddings , while the other uses ELMo embeddings . A dropout probability of 0.3 and a learning rate of 3e-5 is used in both models . We train them until there is no improvement in validation loss for 10 epochs . Classification with ULMFit consists of three steps . The first step of pre-training a language model is already done and the pre-trained weights are released by Howard and Ruder ( 2018 ) . We first further pre-train AWD-LSTM language model on TRC2-financial corpus for 3 epochs . After that , we finetune the model for classification on Financial PhraseBank dataset , by adding a fully-connected layer to the output of pre-trained language model . 3.3 IMPLEMENTATION DETAILS . For our implementation BERT , we use a dropout probability of p = 0.1 , warm-up proportion of 0.2 , maximum sequence length of 64 tokens , a learning rate of 2e − 5 and a mini-batch size of 64 . We train the model for 6 epochs , evaluate on the validation set and choose the best one . For discriminative fine-tuning we set the discrimination rate as 0.85 . We start training with only the classification layer unfrozen , after each third of a training epoch we unfreeze the next layer . An Amazon p2.xlarge EC2 instance with one NVIDIA K80 GPU , 4 vCPUs and 64 GiB of host memory is used to train the models . 4 EXPERIMENTAL RESULTS . The results of FinBERT , the baseline methods and state-of-the-art on Financial PhraseBank dataset classification task can be seen on table 2 . We present the result on both the whole dataset and subset with 100 % annotator agreement . For all of the measured metrics , FinBERT performs clearly the best among both the methods we implemented ourselves ( LSTM and ULMFit ) and the models reported by other papers ( LPS Malo et al . ( 2014 ) , HSC Krishnamoorthy ( 2018 ) , FinSSLX Maia et al . ( 2018a ) ) . LSTM classifier with no language model information performs the worst . In terms of accuracy , it is close to LPS and HSC , ( even better than LPS for examples with full agreement ) , however it produces a low F1-score . That is due to it performing much better in neutral class . LSTM classifier with ELMo embeddings improves upon LSTM with static embeddings in all of the measured metrics . It still suffers from low average F1-score due to poor performance in less represented labels . But it ’ s performance is comparable with LPS and HSC , besting them in accuracy . So contextualized word embeddings produce close performance to machine learning based methods for dataset of this size . ULMFit significantly improves on all of the metrics and it doesn ’ t suffer from model performing much better in some classes than the others . It also handily beats the machine learning based models LPS and HSC . This shows the effectiveness of language model pre-training . AWD-LSTM is a very Yang et al . ( 2018 ) and Piao & Breslin ( 2018 ) report results on the official test set . Since we don ’ t have access to that set our MSE , and R2 are calculated with 10-Fold cross validation . large model and it would be expected to suffer from over-fitting with this small of a dataset . But due to language model pre-training and effective training strategies , it is able to overcome small data problem . ULMFit also outperforms FinSSLX , which has a text simplification step as well as pre-training of word embeddings on a large financial corpus with sentiment labels . FinBERT outperforms ULMFit , and consequently all of the other methods in all metrics . In order to measure the performance of the models on different sizes of labeled training datasets , we ran LSTM classifiers , ULMFit and FinBERT on 5 different configurations . The result can be seen on figure 2 , where the cross entropy losses on test set for each model are drawn . 100 training examples is too low for all of the models . However , once the training size becomes 250 , ULMFit and FinBERT starts to successfully differentiate between labels , with an accuracy as high as 80 % for FinBERT . All of the methods consistently get better with more data , but ULMFit and FinBERT does better with 250 examples than LSTM classifiers do with the whole dataset . This shows the effectiveness of language model pre-training . The results for FiQA sentiment dataset , are presented on table 3 . Our model outperforms state-ofthe-art models for both MSE and R2 . It should be noted that the test set these two papers Yang et al . ( 2018 ) Piao & Breslin ( 2018 ) use is the official FiQA Task 1 test set . Since we don ’ t have access to that we report the results on 10-Fold cross validation . There is no indication on Maia et al . ( 2018b ) that the train and test sets they publish come from different distributions and our model can be interpreted to be at disadvantage since we need to set aside a subset of training set as test set , while state-of-the-art papers can use the complete training set .

Summary: This paper described the application of BERT in the field of financial sentiment analysis. Authors find that when fine-tuned with in-domain data, BERT outperforms the state-of-the-art, demonstrating that language model pre-training can transfer knowledge learned from unsupervised large corpus to new domain with minimum effort. Experiments are conducted to explore 1) the utility of different in-domain dataset for further pre-training; 2) strategies to avoid catastrophic forgetting, and 3) effectiveness of fine-tuning a subset of the full model.