FinEntity: Entity-level Sentiment Classification for Financial Texts
Authors: Yixuan Tang
Yi Yang
Allen H Huang
Andy Tam
Justin Z. Tang
Venue: EMNLP 2023 Main (2023)
Keywords: Named Entity Recognition
Sentiment Analysis
Financial NLP

Abstract: In the financial domain, conducting entity-level sentiment analysis is crucial for accurately assessing the sentiment directed toward a specific financial entity. To our knowledge, no publicly available dataset currently exists for this purpose. In this work, we introduce an entity-level sentiment classification dataset, called FinEntity, that annotates financial entity spans and their sentiment (positive, neutral, and negative) in financial news. We document the dataset construction process in the paper. Additionally, we benchmark several pre-trained models (BERT, FinBERT, etc.) and ChatGPT on entity-level sentiment classification. In a case study, we demonstrate the practical utility of using FinEntity in monitoring cryptocurrency markets. The data and code of FinEntity is available at https://github.com/yixuantt/FinEntity.

FinEntity: Entity-level Sentiment Classification for Financial Texts"
Yixuan Tang', Yi Yang Allen H Huang Andy Tam Justin Z Tang
Hong Kong University of Science and Technology*
Hong Kong Monetary Authority i
{yixuantang, imyiyang, acahuang]@ust. hk {akftam, jztaag) gov. hk
Abstract
timent datasets are unsuitable for the entity-level sentiment classification task.
In the financial domain, conducting entity-level sentiment analysis is crucial for accurately as- sessing the sentiment directed toward a specific financial entity. To knowledge, no publicly available dataset currently exists for this pur- pose. this work, we introduce an entity-level sentiment classification dataset, called FinEn- tity, that annotates financial entity spans and their sentiment (positive, neutral, and negative) in financial news. We document the dataset construction process in the paper. Addition- ally, we benchmark several pre-trained mod- els (BERT, FinBERT, etc.) and ChatGPT on entity-level sentiment classification. In case study, demonstrate the practical utility of using FinEntity in monitoring cryptocurreney markets. The data and code of FinEntity is available https: / htt///gtthtuuu com/yi xuantt/ FinEntity.
Developing natural language processing (NLP) system for entity-level sentiment classification ne- cessitates the availability of dataset with entity tagging and sentiment annotation. To our knowl- edge, no such public dataset currently exists. In this paper, we this gap by constructing a new dataset that annotates both the financial entity spans and their associated sentiments within a text sequence. We outline the development of a high-quality entity- level sentiment classification dataset for the finan- cial domain, called FinEntity.
Subsequently, we benchmark several pre- pre-trained language models (PLMs) and a zero-shot Chat- GPT model (gpt-3.5-turbe on entity-level senti- ment classification tasks. The results demonstrate that fine-tuning PLMs FinEntity outperforms the zero-shot GPT model. This finding suggests that manually collecting a high-quality domain-specine dataset and fine-tuning PLMs is more suitable than relying on the zero-shot GPT model.
Introduction
"We see ChatGPT's prowess and traction with con- sumers as a near-term threat to Alphabet's 's mul- tiple and boost for Microsoft and Nvidia."! "1 In this Wall Street Journal article, multiple financial entities are mentioned, but their sentiments are con- trasting (Positive for Microsoft and Nvidia. and Negative for Alphabet ). In fact, a consid- erable portion of real-world financial text (such as news articles, analyst reports, and social media data) contains multiple entities with varying senti- ment (Malo et al., 2014; Huang et al., 2023; Sinha and Khandait, 2021; Shah et al., 2023a). Neverthe- less, most existing sentiment classification corpora in the financial domain are sequence-level, i.e., the sentiment label is associated with the entire text sequence. Consequently, these sequence-leve sen-
We further demonstrate the practical utility of FinEntity in investment and regulatory applica- tions, extending the work of Ashwin al. (2021); Wong et al. (2022). Collaborating with a regula- tory agency, we apply the fine-tuned PLMs to a unique cryptocurrency news dataset. Experimen- tal results indicate that the individual cryptocur- rency sentiment, inferred using the FinEntity fine- tuned PLM, exhibits stronger correlation with cryptocurrency prices than traditional sequence- level sentiment classification models. Furthermore. the inferred individual cryptocurrency sentiment can better forecast future cryptocurrency prices leading to enhanced risk monitoring for regulatory agencies and investors.
The views analysis expressed this paper are not necessarily represent the views HKMA. emors are the authors' own.
We make the Finlintity dataset publicly available and hope it will be a valuable resource for financial researchers and practitioners in developing more accurate financial sentiment analysis systems.
those the authors
http:://www. wsj.ccom www./aricce//micooooftaaaaae.eee and-goog-------------- how--bothhhave-o-heeee bcar-ais--ossss 106000022

---

2 Related Work
Recognition model and a sequence-level senti- ment classification model to infer the number of ORG entities and sequence sentiment. Subse- quently, we compile a dataset with a balanced dis- tribution of posttve//ngaaiivennuural articles, en- suring that 80% of the sequences contain more than one entity, Following prescreening, we ob- tain dataset comprising 4,000 financial news quences..
Financial Sentiment Classification. NLP tech- niques have gained widespread adoption in the finance domain (Huang et al., 2023; Yang et al., 2023). One of the essential applications is finan- cial sentiment classification (Kazemian et al., 2016; Yang et al., 2022; Frankel et al., 2022; Chuang and Yang, 2022). However, prior literature on financial sentiment classification focuses on the entire text sequence (Kazemian et al., 2016; Yang et al., 2022: Frankel et al., 2022). If a text paragraph contains multiple financial entities with opposing sentiment (as common in financial news or analyst reports), sentiment analysis for the entire text sequence may no longer be accurate. Consequently, more fine- grained sentiment analysis approach is needed, one that is specific to individual financial entities.
Label. Entity-level sentiment classification is a sequence labeling task. As such, we employ the BILOU annotation scheme. Specifically, each token in an input sequence is assigned one of the BILOU labels, indicating the beginning, in- side, last, outside, and unit of an entity span in the sequence. Additionally, each annotated BILU entity is tagged with sentiment label (posi- tive/neutrall/negative), while the O entity does not receive a sentiment label. Consequently, each token in the input sequence assigned to one of thirteen labels BBILU-postivveheuttalnnggggg and one O label).
Financial Entity Tagging and Sentiment Dataset. Existing financial sentiment classification datasets, such Financial Phrase Bank (Malo et al., 2014), SemEval- (Cortis al., 2017), AnalystTone Dataset (Huang et al., 2023), Headline News Dataset (Sinha and Khandait, 2021) and Trillion Dollar Words (Shah et al., 2023a), are based on entire text sequence (sentence or article). FiQA, an open challenge dataset 2, features aspect-level sentiment; however, it does not include entity an- notations. SEndFN (Sinha et al., 2022) is dataset for financial entity analysis in short news headlines. However, this dataset uses pre-defined entity list match entities and does not have entity tagging, so it is still not applicable to recognize financial entities from text. Morcover, most of the headlines contain only one entity, which makes it close to a sequence-level sentiment dataset. For financial entity tagging dataset, FiNER (Shah et al., 2023b) and FNXL (Sharma et al., 2023) created for financial entity recognition and numeral span tag- ging respectively, but both lacks sentiment anno- tation. Therefore, we aims to bridge this gap by constructing a high-quality, entity-level sentiment classification dataset, which not only label the fi- nancial entity spans in sentences, but also annotate their associated sentiments.
Annotators. A total of 12 annotators are recruited, all of whom are senior-year undergraduate students majoring in Finance Business at an English- speaking university. Three annotators label the same example to ensure data quality and perform cross-check validation. Therefore, each annota- tor is assigned 1,000 examples. We employ the LightTag platform 6 for the annotation job. Anno- tators are instructed to tag all named entities in the sequence and the sentiment associated with each entity. Focusing the financial domain, we limit the named entity to companies (such as Apple Inc.), organizations (such as The Fed), and asset classes (such as equity, REIT, and Crude Oil). Annotators are advised not to tag persons, locations, events, or other named entities. A screenshot of the annota- tion interface is shown in Appendix A. Annotation Consistency. A total of 28,631 enti- ties are annotated by the annotators, and we con- duet cross-checks in order to ensure data quality. Initially, we employ the Jaccard similarity coeffi- cient to measure entity-level consistency between pairs of annotators. The overall Jaccard similar- ity of the dataset is 0.754. Furthermore, the num- ber of examples with a Jaccard similarity equal to 1.0 is 44.35%, indicating that 44.35% of examples
3 Dataset Construction
Initial Dataset. We obtain a financial news dataset from Refinitiv Reuters Database. In the prescreen- ing step, we utilize a pre-trained Named Entity
https://uuggingface.ooo http://wwwaaaecool//eeeet/eee/ee www.ssimm/bbettteeeeeeeoooo baser-bbaeNNRR "hups://huggingface httpaaaww..oyccyoeaaeeeae soryiyanghkusviiibertooo
'A sequence consists of multiple sentences. "htps://www.lightage

---

Positive
Negative
Neutral
Total
BERT
BERT CRP
RaBBRk
FinBURT
ChaEPP
ChaRIPT
CRR
(few-
Number Percentage 23.60%
503
498
1,130
2.131
shoo
23.37%
53.03%
100%
Negsuve
0.75
0.82
0.53
0.35
0.38
DUG2
Postive
0.51
0.81
0,81
0.84
0.39
0.75
Neutral
0.62
0.81
0.84
0.82
0.71
061
Table I: Sentiment Label Distribution of Entities
Aug
0.39
Marro Ang
0.80
0.81
0.83
0.85
0.56
0.65
Weighned Aog
0.60
0.81
0.83
0.54
0.59
0.66
Single Entity
Multiple Entity
Total
Table 3: Entity-level Sentiment Classification Results,
Number 390
589
979
Percentage
39.83%
60.16%
100%
prompt designed to elicit structured responses. The detailed prompt for zero-shot and few-shot learning is shown in Appendix C.
Table 2: Singlc/Multiple Entity Distribution
in the dataset have exactly the same [begin, end] span by all three annotators. We filter this subset for further sentiment consistency checks. Subse- quently, we utilize Fleiss' Kappa to measure each example's sentiment annotation consistency (Gwet, 2014). We select examples with a Fleiss' Kappa higher than 0,8 and then use majority voting to obtain the entity's ' sentiment, ensuring high consis- tency in sentiment annotations.
Evaluation. We randomly partition the dataset into 80% training dataset and 20% testing dataset. We employ Seqeval (Nakayama, 2018) for evaluation, reporting Fl-scores, which include the negative, positive, and neutral classifications, respectively. It is important to note that for entity-level sentiment classification, testing example is considered cor- rectly classified only if all the named entities in the example are correctly tagged and their associated sentiments are correctly classified. This implies that this task is much more challenging than tradi- tional sequence-level sentiment classification tasks. Results. We present the benchmarking results in Table 3. These results demonstrate that fine- tuning PLMs exceeds the performance of ChatGPT model, in line with (Rogers et al., 2023) which suggests that zero-shot ChatGPT is not strong baseline for many NL.P tasks. The results provide important implications that for domain-specific, customized NLP tasks, manually collecting a high- quality dataset, through more labor-intensive, in- deed achieves better performance than the current state-of-the-art generative LLM.
Final Dataset: FinEntity. The final FinEntity dataset contains 979 example paragraphs featuring 503 entities classified as Positive, 498 entities clas- sified as Negative, and 1,130 entities classified as Neutral, resulting in a total of 2,131 entities. Table and Table 2 are detailed distrutions of FinEntity. The sentiment label distribution of entities is fairly balanced. Moreover, About 60% of the financial text in the dataset contains multiple entities. A sample of FinEntity is shown in Appendix B. Our ensuing analysis is based on the FinEntity dataset.
4 Benchmarking Entity-level Sentiment Classification
PLMs. We benchmark several PLMs for entity- level sentiment classification. We consider fine- tuning BERT (bert-base-casede (Devlin et al,, 2018) and a finance-domain specific FinBERT (Yang et al., 2020) by incorporating linear layer on top of each token's 's hidden output. In order ac- count for token label dependency, we replace the linear layer with a conditional random field (CRF) layer, yielding BERT-CRF and FinBERT-CRF re- spectively. We implement those PLMs using the transformers library (Wolf et al., 2019).
5 Case Study: Cryptocurrency Market
In this section, we demonstrate the practical utility of the FinEntity dataset for investment and regu- latory applications. As case study, focus on the cryptocurrency market, which is notoriously volatile and has been plagued by instances of ma- nipulation and fraud, as evidenced by the recent FTX crash. As such, is crucial for regulators and investors to closely monitor the market sentiment toward different cryptocurrencies.
ChatGPT. For comparison with ofate-of-the-art generative LLMs, we examine the zero-shot and few-shot in-context learning performance of ChatGPT by querying OpenAl's 3pt-3.5-turbe t-turbo model with a 0.0 temperature value. Following pre- vious literature (Shah et al., 2023a), we construct a
In this case study, collaborate with regula- tory agency responsible for overseeing the mone- tary and financial systems in both local and global markets. The team at the regulatory agency shares a cryptocurrency news dataset ooen-overlapping with our Reuters data) that they have internally col-

---

lected. The dataset comprises 15.290 articles span- ning from May 20, 2022 to February 2023. Each article is associated with timestamp, enabling time-series analysis. We select four cryptocurren- cies (Bitcoin, Ethereum, Dogecoin, Ripple) as the target entities for analysis, owing to their substan- tial market dominance and the attention they re- ceive from investors.
find that 37.53% of the examples contain more than one entity. Among these examples, 12.50% in- clude entities that have opposing sentiments. Thus, the entity-level sentiment model can more accu- rately infer the sentiment focal eryptocurrency compared to traditional sequence-level models.
5.1 Sentiment Classification
pkostandrret
Traditional approach: sequence-level. To fa- cilitate comparison, utilize sequence-level pre-trained financial sentiment classification model (Huang ct al., 2023). For each focal cryptocurrency, such as Bitcoin, we extract sentences containing the target word (e.g., Bitcoin, BTCC) from articles on the same date and feed them into the model to obtain sentiment labels (pssitive/ngaaiveeeuttall) Our approach: entity-level. We employ the FinBERT-CRE model, which is fine-tuned on the FinEntity dataset, to extract daily entity-level senti- ment. Specifically, we feed an article into FinBERT- CRF and obtain a set of named entities along with their associated sentiments. Subsequently, we group entities and aggregate sentiments to derive daily sentiment labels.
22
3222 06
2022-7.0 04
202100 23
2022000
282 10
202-11
202101
201307
Figure I: Daily Bitcoin prices and the sentiments in- ferred from FinBERT-CRF model.
5.3 Prediction Experiment
We also conduct a forecasting task where we pre- dict the next day's Bitcoin price using its price time series and the inferred sentiment. For the price fea- ture, we utilize the Open-High-Lane High-Low-Clowe (OHLC) price. For sentiment feature, we opt for using the percentage of three different sentiment labels for each day as features. We chronologically divide the dataset into three parts: 60% for training, 20% for validation, and 20% for testing. We employ an LSTM as the prediction model. It incorporates a time step of 10 and hidden size of 128.
5.2 Contemporaneous Correlation Analysis
We begin by measuring the contemporaneous cor- relation between daily focal cryplocurrency prices and inferred sentiments. To obtain numerical value for the sentiment score, we code positive la- bels as +1, neutral as and negative as -1. Then, We calculate the sum sentiment scores contained in each day's articles and normalize them using min-max normalization.
We consider LSTM model that uses the OHLC price only and LSTM models that incorporate ad- ditional sentiment features inferred from either sequence-level or entity-level models. Table 4 re- veals that the model incorporating entity-level sen- timent features exhibits better accuracy than those utilizing sequence-level sentiment features or ex- cluding sentiment features altogether. This further emphasizes that sequence-level approach is insuffi- cient due to the presence of multiple entities within
For illustration purposes, we present the corre- lation graph for Bitcoin in Figure 1. The graph indicates positive correlation between the price and sentiment score inferred from the entity-level sentiment model FinBERT-CRE Additionally, we observe that both the sentiment and the price of Bit- coin experienced sharp decline in early November 2022, attributable to the bankruptcy of FTX.
Next, we compute the maximum information co- efficient (MIC) between the cryptocurrency price and the inferred sentiment. Figure displays moderate positive correlation. Furthermore, entity- level sentiment exhibits higher correlations than sequence-level sentiment. Cryptocurrency markets are highly interconnected and frequently covered together in the press. Upon examining this dataset,
2275
0.200
0.30
0123
siquence Irvel Jeckl
01000
Bitcoin
Btereum
Dogacoin
Ripple
Figure 2: Correlation hetween sequence-level and entity- level sentiment and different cryptocurrency prices.

---

Peatures
RMSE
Keith Cortis, André Freitas, Tobias Daudert, Manuela Huerlimann, Manel Zarrouk, Siegfried Handschuh. and Brian Davis. 2017. Semeval-2017 task 5: Fine- grained sentiment amalysis on financial microblogs and news. In Proceedings of the 11th international workshop on semantic evaluation (SemEvaal22017 pages 519-535.
OHI + entity-level sentiments
0,08502
OHLC sequence-level sentiments
0.09549
OHLC only
0.11218
Table 4: Bitcoin price prediction performance.
a single sequence, highlighting the practical utility of our manually curated FinEntity dataset.
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv.1810.04805.
6 Conclusion
In this paper, we present the construction of FinEn- tity, dataset with financial entity lagging and sen- timent annotation. We benchmark the performance of PLMs and ChatGPT on entity-level sentiment classification and showcase the practical utility of FinEntity in a case study related to the cryptocur- rency market. We believe that FinEntity can be a valuable resource for financial sentiment analysis in investment and regulatory applications.
Richard Frankel, Jared Jennings, and Joshua Lee. 2022.
Disclosure sentiment: Machine learning vs. dictio- nary methods. Management Science, 68(7):5514 14 5532.
Kilem L Gwet. 2014. Handbook of inter-rater reliabil- ity: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC.
Allen H Huang, Hui Wang, and Yang. 2023. Finbert: A large language model for extracting information from financial text. Contemporary Accounting Re- search, 40(2)8806-841
Limitations
First, our dataset construction relies on Reuters news. We have not extensively investigated the transferability of this news dataset to other financial corpora, such as corporate reports or social media posts. Second, since the dataset is in English, its applicability to non-Einglish financial texts may be limited. Besides, financial decisions should con- sider multiple factors since relying on the above- mentioned forecasting methods may entail risks.
Siavash Kazemian, Shunan Zhao, and Gerald Penn. 2016. Evaluating sentiment analysis in context of securities trading. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2094 -2033.
Pekka Malo, Ankur Sinha. Pekka Korhonen, Jyrki Wal- lenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782-796
Ethics Statement
Hiroki Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation. Software available from https://github. com/chakki workssseiww
The study was conducted in accordance with the ACL Ethics Policy. The annotators were recruited via University Research program. All activities, including data annotation, comply with the Uni- versity and program policies. Confidentiality and anonymity were ensured throughout the study by using unique participant identifiers and securely storing the data.
Anna Rogers, Niranjan Balasubramaniann Leon Der- czynski, Jesse Dodge Alexander Koller, Sasha Luc- cioni, Maarten Sap, Roy Schwartz, Noah A Smith, and Emma Strubell. 2023. Closed models make bad baselines.
Agam Shah, Suvan Paturi, and Sudheer Chava. 2023a. Trillion dollar words: new financial dataset, task & market analysis. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (ACL).
References
Julian Ashwin, Eleni Kalamara, and Lorena Saiz. 2021. Nowcasting euro area gdp with news sentiment: tale of two crises.
Agam Shah, Ruchit Vithani, Abhinav Gullapalli, and Sudheer Chava. 2023b. Finer: Financial named en- tity recognition dataset and weak-supervision model.
Chengyu Chuang and Yi Yang. 2022. Buy tesla, sell ford: Assessing implicit stock market preference in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages 100-105.
Soumya Sharma, Subhendu Khatuya. Manjunath Hegde, Afreen Shaikh Dasgupta, Pawan Goyal, Niloy Gan- guly, al. 2023. Financial numeric extreme la- belling: A dataset and benchmarking for xbrl tagging. arXiv preprint 030.v:2300.03723.

---

Ankur Sinha, Satishwar Kedas, Rishu Kumar, and Pekka Malo. 2022. Sentfin 1.0: Entity-awar aware sen- timent analysis for financial news. Journal of the Association for Information Science and Technology, 73(9):1314-1333
"Discard all the previous instructions. Behave like you are an expert entity recognizer and senti- ment classifier. Identify the entities which are com- panies or organizations from the following content and classify the sentiment of the corresponding en- tities into 'Neutral' 'Positive' or 'Negative' classes. Considering every paragraph as a String in Python, provide the entities with the start and end index to mark the boundaries of it including spaces and punctuation using zero-based indexing. In the out- put, Tag means sentiment; value means entity name. If no entity is found in the paragraph, the response should be empty. The paragraph! (paragraph)." For few-shot in-context learning, we concate- nate three examples, i.e., three-shot, to the above prompt, as follows: "user: "Other U.S. companies have made similar moves, including social media site Reddit Inc and Mobileye, the self-driving car unit of Intel Corp <INTC.0>. "assistant: ("start": 74, "end": 84, "value": "Reddit Inc", "lag": "Neu- 128, "end": 138, "value": "Intel Corp", "tag": "Neutral" Juser: "Kellogg <K.N>, however, based the corporate headquarters for its largest business, snacks, in Chicago after an- nouncing split into three independent companies this summer. [nLAN2Y822D] "assistant: "["start": "end": 11, "value": "Kellogg", "tag": "Neu- tral") "user: "Rival Oracle <ORCLN> says in a statement on its website has withdrawn all products, services and support for Russian and Belarusian companies, subsidiaries and partners. An Oracle spokesperson declined further com- ment. "assistant: "slarr::: ""start"t"""trrt.... 183, "end": 177, "value": "Oracle", "tag": "Neutral"), [s["start": "end": 12, "value": "Oracle". "tag": "Neutral")".
Ankur Sinha and Tanmay Khandait. 2021. Impact of news on the commodity market: Dataset and results. In Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2, pages 589-601. Springer.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of of the-an art natural language processing. CoRR, hts//910.03771.
Ken Wong, Max Kwong, Paul Luk, and Michael Cheng. 2022. A robust textual analysis of the dynamics of hong kong property market. Pacific Economic Review.
Yi Yang, Yixuan Tang, and Kar Yan Tam. 2023. In- vestlm: A large language model for investment using financial domain instruction tuning. arXiv preprint arXiv:2309.1300
Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. CoRR, abs/2006.08097,
Yi Yang. Kunpeng Zhang, and Yangyang Fan. 2022. Analyzing firm reports for volatility prediction: A knowledge-driver text-embedding approach. IN- FORMS Journal Computing, 34(1):522-540.
A Appendix A: Annotation Interface
A screenshot of the annotation interface is shown in Figure 3. Annotators tag one of the three sentiment labels (Positive, Negative, Neutral) the selected named entities.
B Appendix B: A sample of FinEntity
We present sample of FinEntity and its annotation in Table 5.
C Appendix C: Zero-shot and few-shot ChatGPT
We follow prior literature (Shah et al., 2023a) to construct the following prompt query ChatGPT (OpenAl's gpt-3.5-turbo API) to obtain structured responses. Given a text sequence, ChatGPT returns a list of financial named entities (in the format of [begin, end] spans) and their associated sentiments.

---

Schemna
Lovinng warmeeue
723
Targes protitable
4WMM. No (ILANZUS2AP) <DOST.5>. Meanshi
yours givon cholcers retum Binaconccon
Sespite that, Berant enterprin vatue of timee fenward FarTDA, acconding to Betinini helow bent those dusts, Karaet Sontien fhr lte
Figure 3: Entity-leve Sentiment Annotation Interface.
"With S&P 500 earnings expected grow 8.4% 2022, the backdrop stocks appears be solid However, skittish investors have punished companies such Netflix <NFI.X.>>, JPMorgan </PM.N>and Tesla
<TSLA. MISLA.Oxdeliveriig less than stellar news recent weeks, adding the uneasy mood. Another large batch of
reports due week, including from heavyweights Alphahet ZGOOGL.Osaa Amazon <AM7N.02
["start": "nn"""":: 12, "value"; "S&P 500". "tag": "Positive], "start": 164, 'end": 171, "value": "Netflxx. "tag":
"Negative")
("start": 182,"end": 190, "value": "JPMMrran""" "tag:: "Negative"), ), { ("start": 203, "end": 208, "value": "Tesla", "tag": "Negative")
("start": "end"" 381, "value": "Alphabet", "tag": "Neutral") ) ("start": 396,"end "end"; 402, "value": "Amazon". "tag": "Neutral")
Table 5: Top row: A sample of FinEntity. Bottom row: Its entity span tagging and sentiment annotations.


Reviews:
[{"review_id": "tjQj1OfYyE", "reviewer": "Reviewer_R2TV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We are happy that you find our work \"is a clear contribution to the community when publicly available.\" Yes, we will release the dataset.\n\nOur response to your comments is as follows.\n\n1. Regarding your \"ethics concern,\" we want to assure you that the undergraduate students were recruited via a University Research program. All the activities, including the data annotation, comply with the University and program policy. To our knowledge, there is no ethical concern on this matter. We will add an \"Ethics Statement\" to the manuscript to alleviate your concern.\n\n2. Regarding your comment that \"potential problems and even financial risks one may have if they try to implement and use their approach to forecast cryptocurrency prices,\" please note that the primary purpose of constructing such a dataset is that we want to contribute a fine-grained and high-quality dataset to the financial NLP community so that financial sentiment analysis can be conducted in a more robust way. By contributing such a dataset, we hope to minimize potential unexpected financial risk caused by inaccurate sentiment analysis. However, we do agree with your comment, and we will add a discussion to the manuscript. \n\n3. Regarding your comment that \"there are many other entity-level datasets, including SemEval-2022\", we agree that they are many entity-level datasets. However, to our best knowledge, many of them, including SemEval-2022, are not financial domain-specific. Recent literature has noted the importance and the need to evaluate financial NLP tasks using financial-domain specific datasets (Shah et al. 2022). As we discussed in the literature review section, there are a few entity-level datasets in the financial domain. However, none of them provides an entity-level sentiment label. Therefore, we believe FinEntity will be a high-impact, high-quality data set.\n\nShah, Raj, et al. \"When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The dataset created by the authors is interesting, and it is a clear contribution to the community when publicly available, especially since the entity-level dataset is much more scarce than the sentence-level.\n- The authors demonstrate care and attention to quality when creating their dataset. Also, they demonstrate a pertinent application of their dataset in a real-world scenario for stock prices forecast.\n- The paper is quite direct and easy to follow.", "reasons_to_reject": "- There is no Ethical Authors did not provide a dedicated \"Ethics Statement\". My main concern is related to the use of 12 undergraduate students for labeling their dataset. It is not clear if this comply with good scientific ethics standards.\n- The authors does not carefully state the potential problems and even financial risks one may have if they try do implement and use their approach to forecast currency prices.\n- There are overstatements such \"no such public dataset currently exists\". There are many other entity-level datasets, including from SemEval-2022. Authors should clarify who their dataset are unique.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work, the authors introduce a new entity-level sentiment analysis classification dataset focused on Financial Texts. Entity-level sentiment analysis (SA) is a branch of SA where sentiment is associated with specific entities in the text and is not just an overall sentiment classification common in sentence-level sentiment analysis. The authors describe step-by-step how they collected their data and annotated their dataset. Moreover, they propose a specific classification model (FinBERT-CRF) and show that it outperforms other transformed-based models. In a case example, the authors also demonstrate how one may use their dataset to build a model that predicts cryptocurrency prices better than without using the model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Provide references of using conditional random field layer with BERT.\n- In table 1 caption, state which metric the numbers mean, is it accuracy? F1 Score?"}, "extra_text": {"Justification_for_Ethical_Concerns": "Authors did not provide a dedicated \"Ethics Statement\". My main concern is related to the use of 12 undergraduate students for labeling their dataset. It is not clear if this comply with good scientific ethics standards."}}, {"review_id": "e1CqCpHMV9", "reviewer": "Reviewer_NjN2", "summary": "", "questions": "In section 5.1, the entity-level approach used in the paper is not explicitly clarified regarding the grouping of entities. It is unclear whether entities are grouped based on the target words utilized in the sequence-level approach or if the grouping and aggregation of sentiments are done manually. Additional information on the entity-level approach's methodology would help to understand the process better and ensure the validity of the comparison with the sequence-level approach.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. Our response to your comments is as follows.\n\n1. We respectfully disagree with your comment that \"the dataset lacks innovation.\" The primary contribution of this work is a new dataset for financial domain entity-level sentiment analysis. To our knowledge, this is the first dataset of such kind. We do not present a new method for the task. Moreover, the recent NLP community has made several notes on how to evaluate the contribution of a Resource paper [1,2]. We would argue that FinEntity is a high-impact, high-quality data set (see [1, 2]), which makes it an appropriate contribution to the Resources and Evaluation Track. \n\n2. Regarding your \"Experimental Report Style\" comment, we would like to refer to Prof. Anna Roger's note (Myth 1: Resource papers are not science) [2], and we believe that our construction of FinEntity, an entity-level sentiment annotation for the financial domain, makes a clear contribution to the field. Recent literature has also noted the importance and the need to evaluate financial NLP tasks using financial-domain specific datasets (Shah et al. 2022).\n\n3. Regarding your question on the \"dataset size in Section 5.1\", we apologize for any confusion caused. Both sequence-level and entity-level models use the same dataset, composed of sentences from financial news containing focal cryptocurrency entities (e.g., Bitcoin, BTC). This results in an identical data size. The distinction between sequence-level and entity-level models lies in the level of sentiment analysis: entity-level analysis uses the entity-level sentiment as the focal cryptocurrency sentiment, whereas sequence-level analysis uses the entire sequence sentiment as its sentiment. Our experiment shows that entity-level sentiment analysis can better capture the focal cryptocurrency's sentiment than the conventional sequence-level analysis. We will clarify it in the revised manuscript.\n\nWe thank you for your insightful comments. Should you have any further comments, we are happy to address them.\n\n[1]: Advice on Reviewing for EMNLP, EMNLP 2020 Blog post, May 17th, 2020\n[2]: Peer review in NLP: resource papers, Anna Rogers, April 16th, 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear Documentation: The paper provides transparent and detailed documentation of the data collection and sentiment annotation process, allowing easy dataset reproduction and potential expansion by other researchers.\n2. Objective Sentiment Annotation: The method used to assign sentiment labels to each financial entity appears to be objective, ensuring consistency in the dataset construction process.", "reasons_to_reject": "1. Lack of Innovation: The dataset construction method employed in the paper is relatively simple and lacks innovation, leading to concerns about the originality and novelty of the research.\n2. Experimental Report Style: The paper's approach and presentation make it resemble more of an experimental report rather than a research paper, potentially undermining its overall contribution to the field.\n3. Ambiguity in Dataset Size: In section 5.1, it is not explicitly clear whether the dataset size used for sequence-level and entity-level models is the same. The distinction between the two models in terms of data utilization needs clarification to ensure a fair comparison.", "questions_to_authors": "In section 5.1, the entity-level approach used in the paper is not explicitly clarified regarding the grouping of entities. It is unclear whether entities are grouped based on the target words utilized in the sequence-level approach or if the grouping and aggregation of sentiments are done manually. Additional information on the entity-level approach's methodology would help to understand the process better and ensure the validity of the comparison with the sequence-level approach.", "paper_topic_and_main_contributions": "The paper addresses the disparity in sentiment classification within the financial domain by introducing the FinEntity dataset. This dataset annotates financial entities and their corresponding sentiments in financial news, facilitating entity-level sentiment classification. Pre-trained language models fine-tuned with FinEntity outperform zero-/few-shot ChatGPT models in this task. The practical utility of the dataset is demonstrated through a case study on the cryptocurrency market, showcasing stronger correlations between cryptocurrency prices and inferred sentiment. An LSTM model incorporating entity-level sentiment features shows improved accuracy in Bitcoin price prediction. Overall, the FinEntity dataset offers a valuable resource for accurate financial sentiment analysis systems.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "none"}, "extra_text": {"Missing_References": "Du, Chi-Han, Ming-Feng Tsai, and Chuan-Ju Wang. \"Beyond word-level to sentence-level sentiment analysis for financial reports.\" ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.\n\nLin, Sheng-Chieh, et al. \"Self-attentive sentimental sentence embedding for sentiment analysis.\" ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020."}}, {"review_id": "OcsSlkWPEt", "reviewer": "Reviewer_kPZc", "summary": "", "questions": "1. Regarding the FinEntity dataset, it is recommended that the detailed distributions, i.e., entity categories, single entity, and multiple entity distributions, be presented in tabular form.\n2. As claimed in the limitations, only 4000 samples are annotated, and more domains, i.e., corporate reports or social media posts can be added.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We are happy that you find our work \"may be useful to the community.\" Yes, we will release the dataset.\n\nOur response to your comments is as follows.\n\n1. Your suggestion on adding detailed data distribution is well taken. The distribution is shown below. As you can see, about 75% of financial text in the dataset contains multiple entities. Moreover, the sentiment label distribution of entities is fairly balanced. \n\n| |Positive | Negative | Neutral | Total |\n|-------------|-----------|----------|---------|-------|\n| Number     | 1,586     | 1,261    | 3,041    | 5,888  |\n| Percentage | 26.94%| 21.42% | 51.64%  | 100%  |\n\n\n|                    | Single Entity | Multiple Entity | Total |\n|--------------|--------------- |-----------------|-------|\n| Number      | 457              | 1,438               | 1,895  |\n| Percentage |  24.11%      | 75.88%          | 100%  |\n\n2. Regarding your comment that \"there are only three entities are annotated,\" we apologize for the confusion caused. The FinEntity dataset contains 1,921 unique entities (such as Tesla Inc., AAPL, S&P 500) belonging to three types (company, organization, and asset class). We focus on these three types because they are the most prevalent in financial text. Our annotation pipeline allows us to include more financial entity types in future extensions.\n\n3. We agree with your comment that \"corporate reports or social media posts can be added.\" We have already set up the annotation interface so that we can easily extend the dataset in the future. We will discuss this future direction in the revised manuscript. \n\nWe thank you for your insightful comments. Should you have any further comments, we are happy to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The annotated dataset may be useful to the community;\n2. The authors describe the detailed process of structuring the FinEntity dataset.\n", "reasons_to_reject": "1. The detailed distribution of the proposed dataset is unclear;\n2. Only three entities (company, organization, asset class) are annotated;\n3. The experiments are a bit simple.", "questions_to_authors": "1. Regarding the FinEntity dataset, it is recommended that the detailed distributions, i.e., entity categories, single entity, and multiple entity distributions, be presented in tabular form.\n2. As claimed in the limitations, only 4000 samples are annotated, and more domains, i.e., corporate reports or social media posts can be added.", "paper_topic_and_main_contributions": "This paper focuses on constructing the FinEntity dataset for the financial domain, involving financial entity labeling and sentiment annotation. The authors also present the benchmarks for PLMs and ChatGPT in terms of entity-level sentiment classification. Experimental results validate the practice of the proposed dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. line 062, \"Experiment results\" should be \"Experimental ...\"\n2. line2 126 and 279, \"We\" should be \"we\"."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This study introduces an entity-level sentiment analysis dataset tailored for the financial domain. Alongside the dataset, the authors present baseline experiments utilizing various models, including BERT, FinBERT, and ChatGPT.

Reviewer R2TV appreciated the dataset and its development process. However, reviewer raised valid concerns about the ethics statement and potential implications of the model. These concerns can be addressed in the limitations section, emphasizing that the direct application of the models' decisions could have adverse effects. The authors have already indicated in their rebuttal that an ethics statement will be incorporated, which should be added.

Reviewer NjN2 appreciated the detailed description of the data collection and annotation process. To further enhance the paper's readability, it is advised to elaborate on the concerns regarding data size and distribution.
Decision: Accept-Main