RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
As an increasing amount of information is contained within documents and text available online, utilizing a series of natural language processing (NLP) techniques to automate the process of extracting meaningful information from unstructured text has become a critical task, especially in the financial domain Ashtiani and Raahemi ( 2023 ) . Among all, named entity recognition (NER) serves as a foundational first step in identifying key entities, such as persons, organizations, and locations, enabling the construction of knowledge graphs and other applications.

With the surge of large language models (LLMs), LLMs have demonstrated transformative capabilities in generative tasks, leveraging reinforcement learning from human feedback (RLHF) Christiano et al. ( 2017 ) . LLMs achieve remarkable performance across a wide range of NLP tasks with minimal adaptation Qin et al. ( 2024 ) . However, their ability to perform domain-specific tasks, such as NER in the financial domain, remains less explored. For instance, in the sentence “Johnson Brothers rethink plan for St. Paul waterfront Shepard Road Development.” , a generic NER model might incorrectly classify the company "Johnson Brothers" as a person. This understanding is critical, as it could influence numerous applications in finance.

In this paper, we aim to evaluate the capabilities of state-of-the-art LLMs in performing NER tasks within the financial domain, their response to various prompt types, and their limitations in this context. To achieve this, we conduct a systematic analysis and present experimental results, comparing the effectiveness of leading LLMs with recent fine-tuned approaches. Specifically, we evaluate three advanced LLMs with different parameter sizes, GPT-4o OpenAI ( 2024 ) , LLaMA-3.1 Dubey et al. ( 2024 ) , and Gemini-1.5 Google ( 2024 ) —under three distinct prompting techniques: direct prompting, in-context learning, and chain-of-thought (CoT) prompting. We perform our study by investigating the following two research questions (RQs):

RQ1 : How do different LLMs perform in NER tasks under various prompts?

RQ2 : What types of mistakes do LLMs commonly make?

To sum up, the main contributions of this paper are as follows:

To the best of our knowledge, this is the first study to comprehensively compare state-of-the-art generically trained LLMs on NER tasks in the financial domain.

We analyze LLM performance across three distinct prompting techniques, identify their limitations, categorize five representative types of failures and underlying causes, and elicit two future directions based on our findings.
Large Language Models in Finance
LLMs have recently been applied to finance, particularly in automatic information retrieval and financial analysis Li et al. ( 2023b ) . Li et al., 2023a empirically explore ChatGPT and GPT-4’s capabilities in analyzing financial texts and compare them to state-of-the-art fine-tuned models. However, existing research mainly focuses on fine-tuned finance LLMs or individual generic LLMs, lacking comparisons of their performance under various prompt designs. This paper addresses this gap by providing a comprehensive evaluation of state-of-the-art LLMs under various prompting styles in the context of financial NER tasks.
Study Setup
To understand current LLMs’ capabilities in handling financial NER problems, we choose three state-of-the-art LLMs, each with three popular prompting strategies. We further select two representative transformer-based models and fine-tune them on financial data for comparison.
Financial NER Datasets
In this study, we use the FiNER-ORD dataset Shah et al. ( 2023 ) as our benchmark. While the CRA NER dataset Alvarado et al. ( 2015 ) , based on financial agreements from the SEC, is widely used for research Li et al. ( 2023a ) and includes four entity types (person/PER, location/LOC, organization/ORG, and miscellaneous/MISC), it suffers from a skewed distribution of entity types and limited source of data.

FiNER-ORD resolves this imbalance and removes the ambiguous miscellaneous category, consisting of a manually annotated dataset of 201 financial news articles. This provides a more robust and high-quality benchmark for financial NER tasks and has been adopted in recent research Xie et al. ( 2024 ) . As reported by Shah et al., 2023 , the entity ratio in FiNER-ORD for ORG, LOC, and PER is 2.29:1.17:1, compared to the heavily skewed ratio of 0.31:0.22:1 in the CRA dataset.
Models
We evaluate three state-of-the-art LLMs and their lightweight versions on the FiNER-ORD task: GPT-4o, GPT-4o-mini OpenAI ( 2024 ) , LLaMA-3.1-70B-Instruct, LLaMA-3.1-8B-Instruct, Gemini-1.5-flash, and Gemini-1.5-flash-8B Google ( 2024 ) . The model versions are 20240806 for GPT-4o, 20240718 for GPT-4o-mini, 20240723 for LLaMA-3.1, and the latest stable release for Gemini-1.5-flash models as of November. LLaMA-3.1 models are accessed through the DeepInfra API DeepInfra ( 2024 ) . All models use default configurations as per their respective API documentation OpenAI ( 2024 ); Google ( 2024 ); DeepInfra ( 2024 ) .

Additionally, we evaluate transformer-based models for comparison: BERT Devlin ( 2018 ) and RoBERTa Liu ( 2019 ) . These models are initialized with pre-trained versions available in the Hugging Face Transformers library Wolf et al. ( 2020 ) , using a batch size of 16, a learning rate of 1e-05, and 50 epochs. Fine-tuning is performed on an Nvidia Tesla A100 GPU via Google Colab Google ( 2024 ) .
Prompt Design
We design three types of prompt methods: direct prompt, in-context learning Dong et al. ( 2022 ) , and chain-of-thought Wei et al. ( 2022 ) . As shown in Figure 1 , the direct prompt first gives instructions for the NER task, followed by the given text and the answer format. Next, we conduct few-shot learning (five shots) experiments through in-context learning and CoT prompts. The shots are chosen randomly and the same five shots are used in every experiment. For the in-context learning prompt, we simply add the five examples after the NER task instruction of the direct prompt. For the chain-of-thought prompt, we use the instruction "let’s think step by step" to design intermediate steps for identifying each named entity in the text, as shown in Figure 2 .
Evaluation Metrics
After obtaining answers from the generated text, we label the identified entities through word matching. The evaluation metrics include the entity-level F1 score and the weighted F1 score . The formula for entity-level F1 score is described below, where TP , FP , and FN represent the counts of True Positives, False Positives, and False Negatives, respectively.

Precision=\frac{TP}{(TP+FP)},\ Recall=\frac{TP}{(TP+FN)} (1)

F1\_Score=\frac{2*Precision*Recall}{Precision+Recall} (2)

The weighted F1 score is defined as follows:

w_{i}=\frac{No.\_of\_entities\_in\_class_{i}}{Total\_number\_of\_entities} (3)

Weighted\_F1=\sum\limits_{i=1}^{N}(w_{i}*F1\_Score_{i}) (4)
Experiments
In this work, we conduct experiments to answer the following two research questions.
RQ1: How do different LLMs perform in FiNER-ORD tasks under different prompts?
We present the performance results of three leading LLMs under three distinct prompts in Table 1 . The results are measured using the F1 scores for three entity types and the weighted F1 score (shown in the Weighted column). The LLMs are grouped into two sections based on their size, with bold values highlighting the best performance. From these results, we can draw the following observations.

(1) Fine-tuned language models consistently outperform generic LLMs, the performance gap can be narrowed through prompt design, few-shot learning, and model size. Table 1 demonstrates that fine-tuned language models surpass generic LLMs in zero-shot direct prompting. However, the performance of generic LLMs improves significantly with diverse zero-shot prompting styles, surpassing the prompt designs proposed by Shah et al., 2023 . Additionally, few-shot learning and larger LLMs demonstrate notable advantages over their smaller counterparts.

(2) Chain-of-Thought prompting has limited effect on LLMs performance and can sometimes reduce effectiveness. While few-shot learning generally enhances generic LLMs’ performance, Table 1 shows that the difference between prompting styles is marginal. CoT prompting only improves the performance of the GPT-4o-mini model, whereas it significantly degrades the performance of the LLaMA 3.1 series. Notably, LLaMA 3.1 frequently suffers from "implied entities" errors, where it tends to overanalyze and tag words that merely imply a named entity. This failure type is further discussed in subsequent sections.

(3) The Gemini series outperforms the GPT-4o and LLaMA 3.1 series in the FiNER-ORD task after few-shot learning. The Gemini series outperforms the GPT-4o and LLaMA 3.1 series in the FiNER-ORD task after few-shot learning. Experimental results indicate a consistent performance ranking, with the Gemini series achieving the optimal performance, followed closely by the GPT-4o series. The LLaMA 3.1 series exhibits the lowest performance among the three.
RQ2: What types of mistakes do LLMs commonly make?
We manually annotate the failure types, summarize the limitations of LLMs, and analyze the underlying causes based on their responses, as shown in Table 2 . The most common failure cases include:

(1) Contextual misunderstanding of proper noun. LLMs often fail to classify entities that rely on context correctly, such as domain-specific terms or ambiguous entities. For example, person names that overlap with location names, and organizational entities containing person or location names may be incorrectly categorized.

(2) Pronouns and generic terms . Terms such as pronouns ( "he" or "a woman" ), and generic phrases ( "universities" or "automakers" ) are sometimes misclassified as specific entities.

(3) Citizenship Terms. Words related to citizenship, such as "Chinese" or "British" , are often misclassified as locations despite referring to national identities.

(4) Implied entities . LLMs frequently misinterpret terms that imply specific entities. For example, product names like "iPhone" or "Google Maps" are often mislabeled as organizational entities due to their association with companies.

(5) Entity omission and boundary errors. LLMs struggle to recognize certain entities, such as abbreviations or long entities (e.g., long addresses). They may either omit these entities entirely or incorrectly segment them.
Discussion
The findings of our study highlight several potential directions for improving the performance of LLMs on financial NER tasks:

Tuning LLMs for the Financial Domain. A significant proportion of the observed failure cases involve domain-specific proper nouns. Fine-tuning LLMs with financial data could enhance their ability to accurately recognize such entities.

Implementing self-correction strategies. Our analysis in RQ2 identifies common mistakes made by LLMs in the FiNER-ORD task. Developing self-verification prompting strategies could allow LLMs to recognize and address these errors, thereby reducing recurrent failures.
Conclusion
This study presents the first systematic evaluations of generic LLMs in the FiNER-ORD task under different prompt designs, compared to state-of-the-art fine-tuned transformer-based models. Through comprehensive experiments with LLMs and their related lightweight versions, we demonstrate the capabilities and limitations of generic LLMs in handling domain-specific tasks. Our findings categorize five representative types of failures, along with their underlying causes. We release artifcats for future research 1 1 1 https://github.com/Alex-Lyu0419/Financial-Named-Entity-Recognition-How-Far-Can-LLM-Go .
Figure 1: Direct prompt for the NER task.; Direct prompt for the NER task.
Figure 2: The chain-of-thought prompt for experiments.; The chain-of-thought prompt for experiments.
Performance of different fine-tuned language models and LLMs under different prompts on FiNER-ORD task.
Failure types, distributions, and examples. Entities and their wrong recognitions are highlighted with blue and red , respectively.