FINBERT:  FINANCIAL SENTIMENT ANALYSIS   WITH PRE-TRAINED LANGUAGE MODELS

1 INTRODUCTION . With unprecedented amount of textual data being created every day , analyzing large bodies of text from distinct domains like medicine or finance is of the utmost importance . Yet it is more difficult to apply supervised NLP methods , like text classification , in these domains than it is for more general language . The difficulty comes from two factors : 1 ) The most sophisticated classification methods that make use of neural nets require vast amounts of labeled data and labeling domain-specific text snippets requires costly expertise . 2 ) The NLP models trained on general corpora are not well-suited to supervised tasks since domain-specific texts have a specialized language with unique vocabulary and expressions . NLP transfer learning methods look like a promising solution to both of the challenges mentioned above , and are the focus of this paper . The core idea behind these models is first training a language model on a very large corpus and then initializing down-stream models with the weights learned from the language modeling task . The initialized layers can range from the single word embedding layer Peters et al . ( 2018 ) to the whole model Howard & Ruder ( 2018 ) . This approach should reduce the size of the required labeled data since language models learn the language syntax and semantic in an unsupervised way on a very large unlabeled corpora by predicting the next word . By further pretraining a language model on a domain specific unlabeled corpus , the model can learn the semantic relations in the text of the target domain , which is likely to have a different distribution than a general corpus . In this paper , we explore the effectiveness of using and fine-tuning a pre-trained language model , BERT Devlin et al . ( 2018 ) , in financial sentiment classification using the Financial PhraseBank created by Malo et al . ( 2014 ) and FiQA Task-1 sentiment scoring dataset in Maia et al . ( 2018b ) . The main contributions of this paper are the following : • We introduce FinBERT , which is a language model based on BERT for financial NLP tasks . We evaluate FinBERT on two financial sentiment analysis datasets , where we achieve the state-of-the-art on FiQA sentiment scoring and Financial PhraseBank . • We implement two other pre-trained language models , ULMFit and ELMo for financial sentiment analysis and compare these with FinBERT . • We conduct experiments to investigate several aspects of the model , including : effects of further pre-training on financial corpus , training strategies to prevent catastrophic forgetting and fine-tuning only a small subset of model layers for decreasing training time without a significant drop in performance . 2 FINBERT . BERT ( Devlin et al. , 2018 ) is a language model that consists of a set of transformer encoders ( Vaswani et al. , 2017 ) stacked on top of each other . It defines the language modeling in a novel way . Instead of predicting the next word given previous ones , BERT ” masks ” a randomly selected 15 % of all tokens . With a softmax layer over vocabulary on top of the last encoder layer the masked tokens are predicted . A second task BERT is trained on is ” next sentence prediction ” . Given two sentences , the model predicts whether or not these two actually follow each other . Following the previous work ( Howard & Ruder , 2018 ) on the effectiveness of further pre-training a language model on a target domain , we experimented with two approaches : The first is pre-training the model on a relatively large corpus from the target domain . For that , we further pre-train a BERT language model on a financial corpus . The second approach is pre-training the model only on the sentences from the training classification dataset . Although the second corpus is much smaller , using data from the direct target might provide better target domain adaptation . Sentiment classification is conducted by adding a dense layer after the last hidden state of the [ CLS ] token . This is the recommended practice for using BERT for any classification task ( Devlin et al. , 2018 ) . Then , the classifier network is trained on the labeled sentiment dataset . An overview of all the steps involved in the procedure is presented on figure 2 . While the focus of this paper is classification , we also implement regression with almost the same architecture on a different dataset with continuous targets . The only difference is that the loss function being used is mean squared error instead of the cross entropy loss . As it was pointed out by Howard & Ruder ( 2018 ) , catastrophic forgetting is a significant danger with this fine-tuning approach . Because the fine-tuning procedure can quickly cause model to ” forget ” the information from language modeling task as it tries to adapt to the new task . In order to deal with this phenomenon , we apply three techniques as it was proposed by Howard & Ruder ( 2018 ) : slanted triangular learning rates , discriminative fine-tuning and gradual unfreezing . Slanted triangular learning rate applies a learning rate schedule in the shape of a slanted triangular , that is , learning rate first linearly increases up to some point and after that point linearly decreases . Discriminative fine-tuning is using lower learning rates for lower layers on the network . Assume our learning rate at layer l is α . Then for discrimination rate of θ we calculate the learning rate for layer l− 1 as αl−1 = θαl . The assumption behind this method is that the lower layers represent the deep-level language information , while the upper ones include information for actual classification task . Therefore we fine-tune them differently . With gradual freezing , we start training with all layers but the classifier layer as frozen . During training we gradually unfreeze all of the layers starting from the highest one , so that the lower level features become the least fine-tuned ones . Hence , during the initial stages of training it is prevented for model to ” forget ” low-level language information that it learned from pre-training . 3 EXPERIMENTAL SETUP . 3.1 DATASETS . In order to further pre-train BERT , we use a financial corpus we call TRC2-financial . It is a subset of Reuters ’ TRC21 , which consists of 1.8M news articles that were published by Reuters between 2008 and 2010 . We filter for some financial keywords in order to make corpus more relevant and in limits with the compute power available . The resulting corpus , TRC2-financial , includes 46,143 documents with more than 29M words and nearly 400K sentences . The main sentiment analysis dataset used in this paper is Financial PhraseBank2 from Malo et al . 2014 Malo et al . ( 2014 ) . Financial Phrasebank consists of 4845 english sentences selected randomly from financial news found on LexisNexis database . These sentences then were annotated by 16 people with background in finance and business . The annotators were asked to give labels according to how they think the information in the sentence might affect the mentioned company stock price . The dataset also includes information regarding the agreement levels on sentences among annotators . The distribution of agreement levels and sentiment labels can be seen on table 1 . We set aside 20 % of all sentences as test and 20 % of the remaining as validation set . In the end , our train set includes 3101 examples . For some of the experiments , we also make use of 10-fold cross validation . FiQA Maia et al . ( 2018b ) is a dataset that was created for WWW ’ 18 conference financial opinion mining and question answering challenge3 . We use the data for Task 1 , which includes 1,174 financial news headlines and tweets with their corresponding sentiment score . Unlike Financial Phrasebank , the targets for this datasets are continuous ranging between [ −1 , 1 ] with 1 being the most positive . Each example also has information regarding which financial entity is targeted in the sentence . We do 10-fold cross validation for evaluation of the model for this dataset . 3.2 BASELINE METHODS . For contrastive experiments , we consider baselines with three different methods : LSTM classifier with GLoVe embeddings , LSTM classifier with ELMo embeddings and ULMFit classifier . It should be noted that these baseline methods are not experimented with as thoroughly as we did with BERT . Therefore the results should not be interpreted as definitive conclusions of one method being better . We implement two classifiers using bidirectional LSTM models . In both of them , a hidden size of 128 is used , with the last hidden state size being 256 due to bidirectionality . A fully connected feed-forward layer maps the last hidden state to a vector of three , representing likelihood of three 1The corpus can be obtained for research purposes by applying here : https : //trec.nist.gov/data/reuters/reuters.html 2The dataset can be found here : https : //www.researchgate.net/publication/251231364 FinancialPhraseBank-v10 3Data can be found here : https : //sites.google.com/view/fiqa/home LPS ( Malo et al. , 2014 ) , HSC ( Krishnamoorthy , 2018 ) and FinSSLX ( Maia et al. , 2018b ) results are taken from their respective papers . For LPS and HSC , overall accuracy is not reported on the papers . We calculated them using recall scores reported for different classes . For the models implemented by us , we report 10-fold cross validation results . labels . The difference between two models is that one uses GLoVe embeddings , while the other uses ELMo embeddings . A dropout probability of 0.3 and a learning rate of 3e-5 is used in both models . We train them until there is no improvement in validation loss for 10 epochs . Classification with ULMFit consists of three steps . The first step of pre-training a language model is already done and the pre-trained weights are released by Howard and Ruder ( 2018 ) . We first further pre-train AWD-LSTM language model on TRC2-financial corpus for 3 epochs . After that , we finetune the model for classification on Financial PhraseBank dataset , by adding a fully-connected layer to the output of pre-trained language model . 3.3 IMPLEMENTATION DETAILS . For our implementation BERT , we use a dropout probability of p = 0.1 , warm-up proportion of 0.2 , maximum sequence length of 64 tokens , a learning rate of 2e − 5 and a mini-batch size of 64 . We train the model for 6 epochs , evaluate on the validation set and choose the best one . For discriminative fine-tuning we set the discrimination rate as 0.85 . We start training with only the classification layer unfrozen , after each third of a training epoch we unfreeze the next layer . An Amazon p2.xlarge EC2 instance with one NVIDIA K80 GPU , 4 vCPUs and 64 GiB of host memory is used to train the models . 4 EXPERIMENTAL RESULTS . The results of FinBERT , the baseline methods and state-of-the-art on Financial PhraseBank dataset classification task can be seen on table 2 . We present the result on both the whole dataset and subset with 100 % annotator agreement . For all of the measured metrics , FinBERT performs clearly the best among both the methods we implemented ourselves ( LSTM and ULMFit ) and the models reported by other papers ( LPS Malo et al . ( 2014 ) , HSC Krishnamoorthy ( 2018 ) , FinSSLX Maia et al . ( 2018a ) ) . LSTM classifier with no language model information performs the worst . In terms of accuracy , it is close to LPS and HSC , ( even better than LPS for examples with full agreement ) , however it produces a low F1-score . That is due to it performing much better in neutral class . LSTM classifier with ELMo embeddings improves upon LSTM with static embeddings in all of the measured metrics . It still suffers from low average F1-score due to poor performance in less represented labels . But it ’ s performance is comparable with LPS and HSC , besting them in accuracy . So contextualized word embeddings produce close performance to machine learning based methods for dataset of this size . ULMFit significantly improves on all of the metrics and it doesn ’ t suffer from model performing much better in some classes than the others . It also handily beats the machine learning based models LPS and HSC . This shows the effectiveness of language model pre-training . AWD-LSTM is a very Yang et al . ( 2018 ) and Piao & Breslin ( 2018 ) report results on the official test set . Since we don ’ t have access to that set our MSE , and R2 are calculated with 10-Fold cross validation . large model and it would be expected to suffer from over-fitting with this small of a dataset . But due to language model pre-training and effective training strategies , it is able to overcome small data problem . ULMFit also outperforms FinSSLX , which has a text simplification step as well as pre-training of word embeddings on a large financial corpus with sentiment labels . FinBERT outperforms ULMFit , and consequently all of the other methods in all metrics . In order to measure the performance of the models on different sizes of labeled training datasets , we ran LSTM classifiers , ULMFit and FinBERT on 5 different configurations . The result can be seen on figure 2 , where the cross entropy losses on test set for each model are drawn . 100 training examples is too low for all of the models . However , once the training size becomes 250 , ULMFit and FinBERT starts to successfully differentiate between labels , with an accuracy as high as 80 % for FinBERT . All of the methods consistently get better with more data , but ULMFit and FinBERT does better with 250 examples than LSTM classifiers do with the whole dataset . This shows the effectiveness of language model pre-training . The results for FiQA sentiment dataset , are presented on table 3 . Our model outperforms state-ofthe-art models for both MSE and R2 . It should be noted that the test set these two papers Yang et al . ( 2018 ) Piao & Breslin ( 2018 ) use is the official FiQA Task 1 test set . Since we don ’ t have access to that we report the results on 10-Fold cross validation . There is no indication on Maia et al . ( 2018b ) that the train and test sets they publish come from different distributions and our model can be interpreted to be at disadvantage since we need to set aside a subset of training set as test set , while state-of-the-art papers can use the complete training set .

Summary: This paper proposes a domain adaptation type of task via proposing fine-tuning of pre-trained models such as BERT on data from financial domains. The paper starts off with a good motivation about requiring some kind of domain adaptation particularly when performing tasks such as sentiment analysis on data sets from the financial domain. However, there is not much novelty in this paper.