FINBERT:  FINANCIAL SENTIMENT ANALYSIS   WITH PRE-TRAINED LANGUAGE MODELS
Authors: Dogu Araci
Zulkuf Genc
Venue:  (2020)
Keywords: Financial sentiment analysis
financial text classification
transfer learning
pre-trained language models
BERT
NLP

Abstract: While many sentiment classification solutions report  high accuracy scores in product or movie review datasets, the performance of the methods in niche domains such as finance still largely falls behind. The reason of this gap is the domain-specific language, which decreases the applicability of existing models, and lack of quality labeled data to learn the new context of positive and negative in the specific domain. Transfer learning has been shown to be successful in adapting to new domains without large training data sets. In this paper, we explore the effectiveness of NLP transfer learning in financial sentiment classification. We introduce FinBERT, a language model based on BERT, which improved the state-of-the-art performance by 14 percentage points for a financial sentiment classification task in FinancialPhrasebank dataset.

Under review as a conference paper at ICLR 2020
FINBERT: FINANCIAL SENTIMENT ANALYSIS WITH PRE-TRAINES LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
While many sentiment classification solutions report high accuracy scores in prod- uct movie review datasets, the performance of the methods in niche domains such as finance still largely falls behind. The reason of this gap is the domain- specific language, which decreases the applicability of existing models, and lack of quality labeled data to learn the new context of positive and negative in the spe- cific domain. Transfer learning has been shown to successful in adapting to new domains without large training data sets. In this paper, we explore the effective- ness of NLP transfer learning in financial sentiment classification. We introduce FinBERT, language model based on BERT, which improved the state-of-the-art performance by 14 percentage points for a financial sentiment classification task in FinancialPhrasebank dataset.
INTRODUCTION
With unprecedented amount of textual data being created every day, analyzing large bodies of text from distinct domains like medicine finance is of the utmost importance. Yet it is more difficult to apply supervised NI.P methods, like text classification, these domains than it is for more general language. The difficulty comes from two factors: 1) The most sophisticated classification methods that make use of neural nets require vast amounts of labeled data and labeling domain-spectte snippets requires costly expertise. 2) The NLP models trained on general corpora are not well-suited to supervised tasks since domain-specific texts have a specialized language with unique vocabulary and expressions.
text
NLP transfer learning methods look like promising solution to both of the challenges mentioned above, and are the focus of this paper. The core idea behind these models is first training language model on very large corpus and then initializing down-stream models with the weights learned from the language modeling task. The initialized layers can range from the single word embedding et al. (2018) to the whole model model Howard & Ruder (0188). This approach should reduce the size of the required labeled data since language models learn the language syntax and semantic in an unsupervised way very large unlabeled corpora by predicting the next word. By further pre- training a language model on a domain specific unlabeled corpus, the model can learn the semantic relations in the text of the target domain, which is likely to have different distribution than general corpus.
In this paper, explore the effectiveness of using and fine-tuning a pre-trained language model, BERT Devlin et al. (2018), in financial sentiment classification using the Financial PhraseBank created by Malo al. (2014) and FiQA Task-1 sentiment scoring dataset in Maia et al. (2018b).
The main contributions of this paper are the following:
We introduce FinBERT, which is a language model based on BERT for financial NLP tasks.
We evaluate FinBERT on two financial sentiment analysis datasets, where we achieve the state-of-the-art on FiQA sentiment scoring and Financial PhraseBank.
We implement two other pre-trained language models, ULMFit and ELMo for financial sentiment analysis and compare these with FinBERT.
We conduct experiments to investigate several aspects of the model, including: effects of further pre-training on financial corpus, training strategies prevent catastrophic forgetting

---

Under review as a conference paper at ICLR 2020
and fine-tumie tinettunigg only a small subset of model layers for decreasing training time without significant drop in performance.
2 FINBERT
BERT (Devlin et al. 2018 is language model that consists of set of transformer encoders (Vawwani et al. 2017) stacked on top of each other. It defines the language modeling in a novel way. Instead of predicting the next word given previous ones, BERT "masks" randomly selected 15% of all tokens. With softmax layer over vocabulary on top of the last encoder layer the masked tokens are predicted. A second task BERT is trained on is "next sentence prediction". Given two sentences, the model predicts whether or not these two actually follow each other.
Following the previous work (Howard & Ruder 2018) on the effectiveness of further pre-training language model on target domain, we experimented with two approaches: The first is pre-training the model on a relatively large corpus from the target domain. For that, we further pre-train a BERT language model on a financial corpus. The second approach is pre-training the model only on the sentences from the training classification dataset. Although the second corpus is much smaller, using data from the direct target might provide better target domain adaptation.
Sentiment classification is conducted by adding dense layer after the last hidden state of the [CLS] token. This is the recommended practice for using BERT for any classification task (Devlin et al., 2018). Then, the classifier network is trained on the labeled sentiment dataset. An overview of all the steps involved in the procedure is presented on figure 2
sattrraa pediciin
sodd sesci prodect
Masaii LM procisiion
Seecinart anaddtr
joun
2rosder2
(S2P)
youS
200n
Jouu
Crowdinge
Tolan
[UADIN
your8
Irteen
yourg
Prren
Languaga cooe
Figure 1: Sample figure caption.
While the focus of this paper is classification, we also implement regression with almost the same ar- chitecture on differrntt dataset with continuous targets. The only difference is that the loss function being used is mean squared error instead of the cross entropy loss.
As it was pointed out by Howard & Ruder (2018), catastrophic forgetting is a significant danger with this fine-tuning approach. Because the fine-tuning procedure can quickly cause model to "forget" the information from language modeling task as it tries to adapt to the new task. In order to deal with this phenomenon, we apply three techniques it was proposed by Howard & Ruder (2018): slanted triangular learning rates, discriminative fine-tuming and gradual unfreezing.
Slanted triangular learning rate applies a learning rate schedule in the shape of slanted triangular, that is, learning rate first linearly increases up to some point and after that point linearly decreases.
Discriminative fine-tuning is using lower learning rates for lower layers the network. Assume our learning rate at layer I is a. Then for discrimination rate of 8 we calculate the learning rate for layer 1 - 1 as a:-1 -ay. The assumption behind this method is that the lower layers represent the deep-level language information, while the upper ones include information for actual classification task. Therefore we fine-tune them differently.
2

---

Under review as a conference paper at ICLR 2020
Table 1: Distribtution of sentiment labels and agreement levels in Financial PhraseBank
Agreement level
Positive
Negative
Neutral
Count
100%
25.2%
13.4%
61.45 11.4%
2262
75% 99%
26.6%
9.8%
63.6%
1191
66% 74%
36.7%
12.3%
50.9%
765
50% 65%
31.1%
14.4%
54.5%
627
All
28.1% 1%
12.4%
59.4%
4845
With gradual freezing, we start training with all layers but the classifier layer as frozen. During training we gradually unfreeze all of the layers starting from the highest one, that the lower level features become the least fine-tuned ones. Hence, during the initial stages of training it is prevented for model to "forget" low-level language information that it learned from pre-trainnt pre-training.
EXPERIMENTAL SETUP
DATASETS
In order to further pre-train BERT, of Reuters' TRC21 which consists of 1.8M news articles that were published by Reuters between 2008 and 2010. We filter for some financial keywords in order to make corpus more relevant and in limits with the compute power available. The resulting corpus, TRC2-financial, includes 46,143 documents with more than 29M words and nearly 400K sentences.
financial corpus we call TRC2-finanial. It is subset
The main sentiment analysis dataset used in this paper is Financial PhraseBank from Malo et al. Malo al. 2014). Financial Phrasebank consists of 4845 english sentences selected randomly from financial news found on LexisNexis database. These sentences then were annotated by 16 people with background in finance and business. The annotators were asked to give labels according to how they think the information in the sentence might affect the mentioned company stock price. The dataset also includes information regarding the agreement levels on sentences among annotators. The distribution of agreement levels and sentiment labels can be seen table We set aside 20% of sentences as test and 20% the remaining as validation set. In the end, our train set includes 3101 examples. For some of the experiments,
also make of 10-fold cross validation.
FiQA Maia et al. (2018b) a dataset that was created for WWW '18 conference financial opin- ion mining and question answering challengd" We use the data for Task I, which includes 1,174 financial news headlines and tweets with their corresponding sentiment score. Unlike Financial Phrasebank, the targets for this datasets continuous ranging between [-1, 1] with being the most positive. Each example also has information regarding which financial entity is targeted in the sentence. We do 10-fold cross validation for evaluation of the model for this dataset.
3.2 BASELINE METHODS
For contrastive experiments, consider baselines with three different methods: LSTM classifier with GLoVe embeddings, LSTM classifier with ELMo embeddings and ULMFit classifier. It should be noted that these baseline methods are not experimented with thoroughly as we did with BERT. Therefore the results should not be interpreted as definitive conclusions of one method being better.
We implement two classifiers using bidirectional LSTM models. In both of them, hidden size of 128 is used, with the last hidden state size being 256 due to bidirectionality. A fully connected feed-forward layer maps the last hidden state to vector of three, representing likelihood of three
'The corpus can be obtained for research
purposes
by applying here:
"The dataset
can found here:
http://www..eearrhhgtgmmm/miiiih
FinancialPhrasBBankllll
Data can be found here: https://sieesgooollcco///v/wveee

---

Under review as a conference paper at ICLR 2020
Table 2: Experimental Results the Financial PhraseBank dataset
All data
Data with 100% agreement
Model
Loss
Accuracy
F1 Score
Loss
Accuracy
F1 Score
LSTM
0.81
0.71
0.64
0.57
0.81
0.74
LSTM with ELMo
0.72
0.75
0.7
0.50
0.84
0.77
ULMFit
0.41
0.83
0.79
0.20
0.93
0.91
LPS
0.71
0.71
0.79
0.80
HSC
0.71
0.76
0.83
0.86
FinSSLX
0.91
0.88
FinBERT
0.37
0.86
0.84
0.13
0.97
0.95
LPS (alo al. 2014), HSC (Krishnamoorthy 2018) and FinSSLX (Maia et al. 2018h) results taken from their respective papers, For LPS and HSC, overall accuracy is not reported on the papers. We calculated them using recall scores reported for different classes. For the models implemented by us, we report 10-fold cross validation results.
labels. The difference between two models is that one uses GLoVe embeddings, while the other uses ELMo embeddings. A dropout probability of 0.3 and learning rate of 3e-5 is used in both models. We train them until there is no improvement in validation loss for 10 epochs.
Classification with ULMFit consists of three steps. The first step of pre-training a language model is already done and the pre-trained weights are released by Howard and Ruder (2018). We first further pre-train AWD-LSTM language model on tune the model for classification Financial PhraseBank dataset, by adding fully-connected layer to the output of pre-trained language model.
corpus for epochs. After that, we fine-
3.3 IMPLEMENTATION DETAILS
For our implementation BERT, 0.2, maximum sequence length of 64 tokens, a learning rate of 2e - 5 and a mini-batch size of 64. We train the model for 6 epochs, evaluate on the validation set and choose the best one. For discriminative fine-tuning we set the discrimination rate as 0.85. We start training with only the classification layer unfrozen, after each third of a training epoch we unfreeze the next layer. An Amazon p2.xlarge EC2 instance with one NVIDIA K80 GPU, vCPUs and 64 GiB of host memory used to train the models.
dropout probability of p 0.1, warm-up proportion of
EXPERIMENTAL RESULTS
The results of FinBERT, the baseline methods and state-of-the-art Financial PhraseBank dataset classification task can be seen on table 2 with 100% annotator agreement.
We present the result on both the whole dataset and subset
For all of the measured metrics, FinBERT performs clearly the best among both the methods we implemented ourselves (LSTM and ULMFit) and the models reported by other papers (LPS Malo et al. (2014), Krishnaaaoyr (2018), FinSSLX(Man et al. (2018))) LSTM classifier with no language model information performs the worst. In terms accuracy. close to LPS and HSC, (even better than LPS for examples with full agreement), however it produces a low F1-score. That is due to it performing much better in neutral class. L.TTM classifier with EL embeddings improves upon LSTM with static embeddings in all of the measured metrics. It still suffers from low average Fl-score due to poor performance in less represented labels. But it's performance is comparable with LPS and HSC, besting them in accuracy. So contextualized word embeddings produce close performance to machine learning based methods for dataset this size.
ULMFit significantly improves all of the metrics and it doesn't suffer from model performing much better in some classes than the others. It also handily beats the machine learning based models LPS and HSC. This shows the effectiveness of language model pre-training. AWD-LSTM very
4

---

Under review as a conference paper at ICLR 2020
Table 3: Experimental Results on FiQA Sentiment Dataset
Model
MSE
R²
Yang et. al. (2018)
0,08
0,40
Piao and Breslin (2018)
0.09
0.41
FinBERT
0.07
0.55
Yang et al. (2018) and Piao Breslin 2018 report results on the official test set. Since we don't have access that set our MSE, and R are calculated with 10-Fold cross validation.
large model and would be expected to suffer from over-fitting with this small dataset. But due to language model pre-training and effective training strategies, it is able to overcome small data problem. ULMFit also outperforms FinSSLX, which has a text simplification step as well as pre-training of word embeddings on large financial corpus with sentiment labels.
LSTM
1.0
LLMPh
with ELMo
Ph:BoRT
08
Test Test loss
0.8
0.7
0.5
100
250
500
1000
3000
Training set size
Figure 2: Test loss different training set sizes
FinBERT outperforms ULMFit, and consequently all of the other methods all metrics. In order to measure the performance of the models on different sizes of labeled training datasets, we ran LSTM classifiers, ULMFit and FinBERT on 5 different configurations. The result can be seen on figure 2, where the cross entropy losses test set for each model drawn. 100 training examples is too low for all of the models. However, once the training size becomes 250, ULMFit and FinBERT starts successfully differentiate between labels, with accuracy as high 80% for FinBERT. All of the methods consistently get better with more data, but ULMFit and FinBERT does better with 250 examples than LSTTM classifiers do with the whole dataset. This shows the effectiveness of language model pre-training.
The results for FIQA sentiment dataset, are presented table 3 the-art models for both MSE and R² It should be noted that the test set these two papers Yang al. (2018) Piao & Breslin 2018) use the official FiQA Task 1 test Since we don't have access to that we report the results on 10-Fold cross validation. There is no indication on Maia et al. (2018)) that the train and test sets they publish come from different distributions and our model can be interpreted to be at disadvantage since we need to set aside subset of training set as test set, while stae-of-thheaae papers can use the complete training set.
Our model outperforms state-of-
5 EXPERIMENTAL ANALYSIS
5.1 EFFECTS OF FURTHER PRE-TRAINING
We first measure the effect of further pre-training on the performance of the classifier. We compare three models: No further pre-training (denoted by Vanilla BERT), Further pre-training
5

---

Under review as a conference paper at ICLR 2020
Table 4: Performance with different pre-training strategies
Model
Loss
Accuracy
Score
Vanilla BERT
0.38
0.85
0.84
FinBERT-task
0.39
0.86
0.85
FinBERT-domain
0.37
0.86
0.84
Results are reported on 10-fold cross validation.
classification training (denoted by FinBERT-task), 3) Further pre-training on domain corpus, TRC22financial (denoted by FinBERT-domain) Models are evaluated with loss, accuracy and macro average Fl scores on the test dataset. The results can be seen on table
The classifier that were further pre-trained on financial domain corpus performs best among the three, though the difference is not very high. There might be four reasons behind this result: 1) The corpus might have a different distribution than the task set, 2) BERT classifiers might not improve significantly with further pre-training, 3) Short sentence classification might not benefit significantly from further pre-training, 4) Performance is already good, that there is not much room for im- provement. We think that the last explanation is the likeliest, because for the subset of Financial Phrasebank that all of the annotators agree the result, accuracy of Vanilla BERT is already 0.96. The performance on the other agreement levels should be lower, as even the humans can't agree fully on them. More experiments with another financial labeled dataset is necessary conclude that effect of further pre-training on domain corpus is not significant.
5.2 CATASTROPHIC FORGETTING
For measuring the performance of the techniques against catastrophic forgetting, we try four differ- settings: No adjustment (NA), only with slanted triangular learning rate (STL), slanted triangu- lar learning rate and gradual unfreezing (STL+GU) and the techniques in the previous one, together with discriminative fine-tuning. We report the performance of these four settings with loss on test function and trajectory of validation loss over training epochs. The results can be seen on table 5 and figure 3
1.0
All lhree
STL DFT
STL
None
STL GU
Validation loss
0.0
0.7
0.
04
Epoch
Figure Validation loss trajectories with different training strategies
Applying all three of the strategies produce the best performance in terms of test loss and accuracy. Gradual unfreezing and discriminative fine-tuning have the same reasoning behind them: higher level features should be fine-tuned more than the lower level ones, since information learned from language modeling are mostly present in the lower levels. We see from table that using only dis- criminative tne-tuning with slanted triangular learning rates performs triangular learning rates alone. This shows that gradual unfreezing the most important technique for our case.
than using the slanted

---

Under review as a conference paper at ICLR 2020
Table 5: Performance with different fine-tuning strategies
Strategy
Loss
Accuracy
Fl Score
None
0.48
0.83
0.83
STL
0.40
0.81
0.82
STL + GU
0.40
0.86
0.86
STL DFT
0.42
0.79
0.79
All three
0.37
0.86
0.84
Results are reported on 10-fold cross validation. STL: slanted triangular learning rates, GU: gradual unfreezing, DFT: discriminative fine-tuning.
Table Performance on starting training from different layers
First layer unfreezed
Loss
Accuracy
Training time
Embeddings layer
0.37
0.86
332s
Layer-1
0.39
0.83
302s
Layer-2
0.39
0.83
291s
Layer-3
0.38
0.83
272s
Layer-4
0,38
0.82
250s
Layer-5
0,40
0.83
240s
Layer-6
0,40
0.81
220s
Layer-7
0.39
0.82
205s
Layer-8
0.39
0.84
188s
Layer-9
0.39
0.84
172s
Layer-10
0.41
0.84
158s
Layer-1l
0.45
0,82
144s
Layer-12
0.47
0.81
133s
Classification layer
1.04
0.52
119s
One way that catastrophic forgetting show itself is the sudden increase in validation loss after several epochs. As model is trained, it quickly starts to overfit when no measure is taken accordingly. As it can be seen the figure B that is the case when none of the aforementioned techniques applied. The model achieves the best performance on validation set after the first epoch and then starts overfit. While with all three techniques applied, model is much more stable. The other combinations lie between these twO cases.
5.3 TRAINING ONLY A SUBSET OF THE LAYERS
BERT is a very large model. Even on small datasets, fine-tuning the whole model requires significant time and computing power. Therefore if slightly lower performance can be achieved with fine- tuning only a subset of all parameters, it might be preferable some contexts. Especially if training set is very large, this change might make BERT more convenient to Here experiment with fine-tuning only the last many encoder layers.
The results presented table 6 Fine-tuning only the classification layer does achieve close performance to fine-tuning other layers. However fine-tuning only the last layer handily outperforms the state-of-the-art machine learning methods like HSC. After Layer-9, the performance becomes virtually the same, only to be outperformed by fine-tuning the whole model. This result shows that in order utilize BERT, an expensive training of the whole model is not mandatory. A fair trade-off can be made for much less training time with a small decrease in model performance.
5.4
WHERE DOES THE MODEL FAIL??
With 97% accuracy on the subset of Financial PhraseBank with 100% annotator agreement, we think it might be interesting exercise examine cases where the model failed to predict the true

---

Under review as a conference paper at ICLR 2020
label. Therefore in this section will present several examples where model makes the wrong prediction. Also in Malo et al. (2014), it is indicated that most of the anter-annotator disagreements are between positive and neutral labels (agreement for separating positive-negative negative- and positive-neutral are 98.7%, 94.2% and 75.2% respectively). Authors attribute that the difficulty of distinguishing "commonly used company glitter and actual positive statements". We will present the confusion matrix in order to observe whether this is the case for FinBERT as well.
Example 1:
loss totaled euro 0.3 million compared to loss of euro 2.2 million in the first quarter of 2005
True value: Positive Predicted: Negative
This implementation is very important to the operator since it is about to launch its Fixed to Mobile convergence service in Brazil
Example 2:
True value: Neutral Predicted: Positive
Example 3:
The situation of coated magazine printing paper will
continue to be weak True value: Negative Predicted: Neutral
The first example actually the most common type of failure. The model fails to do the math in which figure is higher, and in the absence of words indicative of direction like "increased", might make the prediction neutral. However, there are many similar cases where it does make the true prediction too. Examples 2 and are different versions of the same type of failure. The model fails to distinguish a neutral statement about a given situation from a statement that indicated polarity about the company. In the third example, information about the company's 's business would probably help.
73% of the failures happen between labels positive and negative, while number is 5% for neg- ative and positive. That is consistent with both the inter-annotator agreement numbers and common sense. It is easier to differentiate between positive and negative. But it might be more challenging to decide whether a statement indicates positive outlook or merely an objective observation.
CONCLUSION AND FUTURE WORK
In this paper, we implemented BERT for the financial domain by further pre-training it on a financial corpus and fine- fine-tuning it for sentiment analysis (FinBERT). This work is the first application of BERT for finance to the best of our knowledge and one of the few that experimented with further pre-training on a domain-specific corpus. On both of the datasets used, we achieved state-of-the- art results by significant margin. For the classification task, we increased the state-of-the art by 15% in accuracy.
In addition to BERT, we also implemented other pre-training language models like ELMo and ULM- Fit for comparison purposes. ULMFit, further pre-trained a financial corpus, beat the previous state-of-the art for the classification task, only to a smaller degree than BERT. These results show the effectiveness of pre-trained language models for down-stream task such as sentiment analysis especially with a small labeled dataset. The complete dataset included more than 3000 examples, but FinBERT was able surpass the previous state-of-the art even with a training set as small as 500 ex- amples. This is important result, since deep learning techniques for NLP have been traditionally labeled as too "data-hungry", which is apparently no longer the case.
We conducted extensive experiments with BERT, investigating the effects of further pre-training and several training strategies. We couldn't 't conclude that further pre-training on a Nomain-specifii corpus significantly better than not doing for case. Our speculation is that BERT already performs good enough with our dataset that there is not much room for improvement that further pre- training provide. We also found that learning rate regimes that fine-tune the higher layers more
8

---

Under review as a conference paper at ICLR 2020
aggressively than the lower ones perform better and more effective in preventing catastrophic forgetting. Another conclusion from our experiments was that, comparable performance can be achieved with much less training time by fine-tuning only the last 2 layers of BERT.
REFERENCES
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for I Language Understanding. 2018. ISSN 0140-525X. doi: arXiv: 1811.03600v.. URL
http://tww.....ee.e.....///ee//a arxiv. mexiv.org/abe abs/ 1810 04805
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classifica- tion, jan 2018. URL /ttp::::/aaaiv..r////aaaaaa
06146
Srikumar Krishnamoorthy. Sentiment analysis of financial news articles using performance in- dicators. Knowledge and Information Systems, 56(2)373-3944 aug 2018. ISSN 02193116. doi: 10.1007/11011--07-1113411 CC1150017-113--
URL
cwo.cocmmmmmmmm.m 1007/
Macedo Maia, Andr Freitas, and Siegfried Handschuh. FinSSLx: A Sentiment Analysis Model for the Financial Domain Using Text Simplification. In 2018 IEEE 12th International Confer- ence on Semantic Computing (ICSC), pp. 318-319. IEEE, jan 2018a. ISBN 978-1 178-1-5386-4408 ieee org/ document.
9. doi: (0.1109/ICSC.201 20.00006.... URL
8334488/
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross Medermott, Manel Zarrouk, Alexandra Balahur, and Ross Mc-Dermott. Companion of the The Web Conference 2018 on The Web Conference 2018, (WWW) 2018, Lyon. France, April 23-27, 2018. ACM, 2018b. ISBN 9781450356404. doi: 10. 144431845555 URL fEtps:///oo: www.oog////. 1115//111111511815555
http://doi.aco wtt.:/aaaa..a......m/ 1tt///11.115/118455335
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Informa- tion Science and Technology, 65(4):782-796, 2014. ISSN 23301643. doi: 10.1002//si.23062.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. 2018, ISSN 9781941643377. doi: 10.18653/v1N118-12000 URL // http:////a..aampoooooo/e/o.o//// Sbs/1802.05366
Guangyuan Piao and John G Breslin. Financial Aspect and Sentiment Predictions with Deep Neural Networks. pp. 1973-19711 2018. ISBN 9781450356404. doi: 10.11453184558.111182.. URL pietps://dos wrg. 1444145330 9155/3311155888811822
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. (Nips), 2017. URL http:
// /17006/ 03762
Steve Yang, Jason Rosenfeld, and Jacques Makutonin. Financial Aspect-Based Sentiment Analysis using Deep Representations. 2018. URL
pdfhttp:/ arxiv. wrw. abs 1808 07931


Reviews:
[{"review_id": "r1gkpwkhcH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper described the application of BERT in the field of financial sentiment analysis. Authors find that when fine-tuned with in-domain data, BERT outperforms the state-of-the-art, demonstrating that language model pre-training can transfer knowledge learned from unsupervised large corpus to new domain with minimum effort. Experiments are conducted to explore 1) the utility of different in-domain dataset for further pre-training; 2) strategies to avoid catastrophic forgetting, and 3) effectiveness of fine-tuning a subset of the full model. \n\nI am in favor of rejecting this paper and my reasons are as follows:\n\nFirst, this paper may lack deeper innovation, although it demonstrates a good application of the BERT models in financial domain. For example, the framework of general-domain LM pretraining, to in-domain LM pretraining and finally in-domain classifier fine-tuning, as well as techniques of catastrophic forgetting were already proposed in Howard & Ruder 2018. Therefore, I think this paper may be more suitable for other (finance) application-oriented venues.\n\nSecond, the dataset used in evaluation is of small size (for example, Financial PhraseBank test set has one 1K). Thus, even though the paper is about transfer learning to domains without large data, I find it might be more convincing to draw a solid conclusion with a larger test set.\n\nThis paper is well organized and easy to follow. It may be beneficial to clarify in a few places (if space permits):\n1) Some description or statistics of the data may be helpful (e.g., average sentence length or some examples);\n2) Citations to Elmo and ULMFit can be made more explicit. Authors did cite Peters 2018 and Howard 2018 at the beginning of the paper, but may want to explicitly associate them with ‘Elmo’ and ‘ULMFit’ when these two terms first occur respectively;\n3) For table 2, does the ‘all data’ or ‘data with 100% agreement’ include training data (80%) or just the test data (20%)?\nThe difference between FinBERT(-domain) and ULMFit can be explicitly contrasted in the paper. Is the former initialized with BERT while latter with ULMFit?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1g2yTe2Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a domain adaptation type of task via proposing fine-tuning of pre-trained models such as BERT on data from financial domains. The paper starts off with a good motivation about requiring some kind of domain adaptation particularly when performing tasks such as sentiment analysis on data sets from the financial domain. However, there is not much novelty in this paper.\n\n1)The authors do not propose any new model architectures. Even if we were to argue the novelty is in terms of their empirical work, there are some flaws/missing details in the experiments.\n2)In table 1 authors present agreement amongst annotators, it would be nice if in addition to mentioning the source of the data, the authors included what metric was used to attain agreement. I had to read the original paper releasing the data set to figure this out.\n3)Table 4 presents results that do not seem significant. It is hard to conclude if a certain pre-training strategy worked for sure.\n\nOn the whole I am very lukewarm on this paper. I find this paper lacking in novelty. Seems like an ambitious class project turned into an ICLR submission.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkeWvWtCtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an analysis of the BERT language model on financial text. FinBERT is evaluated on two datasets from the financial domain: a sentiment prediction dataset (classification with 3 different classes) and a sentiment score prediction (the score is a float number between -1 and 1). \n\nI find the phrasing \"FinBERT is a language model based on BERT\" misleading; I think FinBERT is BERT trained on financial text. There is no modification that is done to the original BERT model.\n\nThe paper presents several experiments using BERT as the language model and fine-tuning for the financial tasks. FinBERT is compared to a few baselines such as LSTMs with ElMO embeddings and ULMfit. I find interesting that the model performs better on the subset of the dataset for which there is perfect agreement between the annotators.\n\nI also find the results on training on financial data interesting. The results seem to indicate that further training on financial text does not seem to result in additional improvement when compared to original BERT.\n\nWhile I find the analysis and the experiments presented in the paper interesting, the novelty of the paper is rather low. There is no new idea introduced in this paper, it contains a series of experiments with BERT on financial text and tasks.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xnFZHVqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for financial sentiment analysis based on the texts obtained from news. The method is based on an existing method BERT (Devlin et al. 2018).  The authors have performed thorough experimental studies of the BERT method on an existing dataset TRC2-financial, a subset of TRC2 consisting of 1.8M news articles. Although the results may be of interest to communities working in this area, there are no or little novel contributions. By reading section 2 (only one page), which describes the method used, I have the impression that the authors took the method BERT and then applied this to the TRC2-financial dataset and then reported the results and also discussed some parameter choices in the BERT method. Therefore, the only value about this paper is the experimental results. Apart from this, there are no other contributions or insights to the methods/problems. In addition, section 2 is over-brief and very unclear, and it only contains a brief summary of the BERT method. For these reasons, I think the paper should be rejected for lacking novelty and writing quality. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents FinBERT, a BERT-based model that is further trained on a financial corpus and evaluated on Financial PhraseBank and Financial QA. The authors show that FinBERT slightly outperforms baseline methods on both tasks.

The reviewers agree that the novelty is limited and this seems to be an application of BERT to financial dataset. There are many cases when it is okay to not present something entirely novel in terms of model as long as a paper still provides new insights on other things. Unfortunately, the new experiments in this paper are also not convincing. The improvements are very minor on small evaluation datasets, which makes the main contributions of the paper not enough for a venue such as ICLR.

The authors did not respond to any of the reviewers' concerns. I recommend rejecting this paper.
Decision: Reject