Argument-Based Sentiment Analysis on Forward-Looking Statements (2024)

Abstract: This paper introduces a novel approach to analyzing the forward-looking statements in equity research reports by integrating argument mining with sentiment analysis. Recognizing the limitations of traditional models in capturing the nuances of future-oriented analysis, we propose a refined categorization of argument units into claims, premises, and scenarios, coupled with a unique sentiment analysis framework. Furthermore, we incorporate a temporal dimension to categorize the anticipated impact duration of market events. To facilitate this study, we present the Equity Argument Mining and Sentiment Analysis (Equity-AMSA) dataset. Our research investigates the extent to which detailed domain-specific annotations can be provided, the necessity of fine-grained human annotations in the era of large language models, and whether our proposed framework can improve performance in downstream tasks over traditional methods. Experimental results reveal the significance of manual annotations, especially for scenario identification and sentiment analysis. The study concludes that our annotation scheme and dataset contribute to a deeper understanding of forward-looking statements in equity research reports.

heading: 1 Introduction
text: The study of argument mining has been a focal point of comprehensive research (Toulmin, 2003; Cabrio and Villata, 2018; Lawrence and Reed, 2019), playing a crucial role in unraveling the complex views of authors or speakers. The foundational Toulmin argument model (Toulmin, 2003) classifies argument units into claims and premises (evidence) and dissects their interrelations. Nevertheless, such a categorization seems overly broad for interpreting the intricacies of forward-looking arguments, those primarily concerned with future analysis (Chen and Takamura, 2024). For example, when predicting future price trends, the Toulmin model treats past events and anticipated fu-
ture scenarios as identical types of premises supporting an investor’s claim. However, under the efficient-market hypothesis (Malkiel, 1989, 2003), events that have already transpired rapidly reflect in the market. Considering this, we propose a more detailed subdivision of premises into two categories: premises and scenarios. This approach aids in a deeper understanding of the author’s futureoriented analysis.
Scenario planning is a critical process for conceptualizing and formulating comprehensive longterm plans (Amer et al., 2013). For instance, given the context of COVID-19 recovery, envisioning the future of public transit and shared mobility is a pressing issue for the transportation industry. To execute scenario planning, Shaheen and Wong (2021) engage numerous experts to deliberate on this topic. This underlines the fact that scenario planning still heavily relies on experts (Shaheen and Wong, 2021) and underscores the significance of identifying expert-written scenarios for understanding future projections. Given this factor, this paper introduces the task of scenario identification, along with the traditional task of argument unit identification, emphasizing the importance of scenarios authored by experts. As equity research reports typically delve into the future, we leverage them as our primary source.
We elevate this approach by merging argument
13804
mining with sentiment analysis, enabling a profound understanding of opinions and their interconnections. Deviating from traditional sentiment analysis that employs a generic label set (positive/negative), we propose distinct label sets for different argument units (claim, premise, and scenario). Table 1 shows an example in the proposed dataset. When making claims, investors reveal their personal perspectives (bullish/bearish/neutral). To support their claims, investors reference objective premises (positive/negative/neutral) and formulate scenario plans to contextualize these events. We utilize Dator’s Four Generic Scenario Archetypes method (Dator and Dator, 2019) to categorize scenarios into four labels (continued growth/ steady state/ transformation/ collapse). This proposed argument-based sentiment analysis framework lays the foundation for future exploration of forwardlooking arguments from a nuanced perspective.
Furthermore, when talking about the future, how long the impacts will continue becomes an important question (Tseng et al., 2023). For example, we may not use the earnings of a company in 2010 to assess the value of this company in 2024. To capture the duration of impacts implied in the arguments, we categorize the temporal dimension into: “within a month,” “1-3 months,” “4-6 months,” “6- 12 months,” and “over a year”. This classification is driven by the observation that most equity research reports discuss impacts spanning the upcoming year. By incorporating the temporal dimension, we distinguish between fleeting market sentiments, which are immediate reactions to news or shortterm events, and more enduring financial analyses that reflect deeper market mechanisms and longterm company performance. Table 2 showcases examples by their anticipated impact durations. The first indicates a temporary rebound for Catcher in July, attributable to Sony Xperia casings, reflecting short-lived market fluctuations. In contrast, the second delves into the high-end smartphone market’s anticipated saturation and a predicted 10% growth over 1-2 years, illustrating a sustained market outlook. Such demarcation provides clarity on the expected duration of an event’s influence, enabling a deeper understanding of forthcoming market shifts.
In summary, this paper presents a novel annotation scheme for the automated interpretation of equity research reports by integrating the methodologies of argument mining and sentiment analy-
sis. Addressing the lack of annotated professional analysis reports available to the research community, we introduce a comprehensive and expansive dataset named Equity Argument Mining and Sentiment Analysis (Equity-AMSA). This dataset covers a wide temporal range and volume, thus broadening the scope of our research community’s endeavors. To conduct a thorough examination of the proposed Equity-AMSA, we aim to address the following three research questions (RQs):
• (RQ1): To what extent can someone with domain knowledge provide annotations in this detailed manner?
• (RQ2): Why are fine-grained human annotations still necessary in an era where large language models demonstrate high performance in many NLP tasks?
• (RQ3): Can the proposed argument-based sentiment analysis scheme enhance performance in downstream tasks compared to traditional sentiment analysis and argument mining approaches?

heading: 2 Related Work
text: Financial news, earnings calls, and social media have been extensively used to create annotated datasets for model training. These annotations usually entail assigning sentiment labels, identifying entities, and performing argument analysis. For example, the Financial Phrase Bank dataset (Malo et al., 2014) comprises nearly 5,000 English sentences sourced from financial news. Each sentence is classified as positive, negative, or neutral, based on its emotional tone. StockTwits (Jaggi et al., 2021), a social media platform, offers labeled
tweets about companies, signaling sentiment as either bullish or bearish. This dataset encompasses various stocks and extends over several years of data collection. Alhamzeh et al. (2022) annotated argument units and structures in a dataset containing 804 documents from financial earnings calls. However, datasets specifically focusing on annotating argument-based equity research reports are seldom found. Despite these resources, none of the previous studies have amalgamated the concepts of argument mining, sentiment analysis, and scenario planning for a detailed understanding of financial opinions. This paper pioneers this research direction and introduces the first dataset for this purpose. Moreover, we turn our attention to equity research reports, a resource rarely included in previous open-access datasets, despite its pivotal role in the financial market. We anticipate that our dataset will prompt diverse discussions on opinion mining in financial documents.

heading: 3 Dataset

heading: 3.1 Data Source
text: We collected English equity research reports from Bloomberg Terminal.1 Our data source comprises 1,876 analyst reports from the period between 2014 and 2022. According to the analysts’ ratings, 47% of the reports are categorized as “Buy,” 33% as “Neutral,” and a mere 18% as “Sell.” This distribution reveals a predisposition towards positive or neutral recommendations in analyst reports, potentially resulting from conflicts of interest, legal stipulations, or investor inclinations. The average word count for each report is 391 words, with the lengthiest report containing 735 words and the briefest one having 118 words. Typically, each report is structured into 3 to 4 paragraphs.

heading: 3.2 Annotation Task Guidelines
text: The primary objective of this annotation task is to identify the premise, scenario, and claim within the text. Annotators then assign various labels based on the type of statement, such as impact duration and sentiment. The labels are defined as follows:
Claim: This term refers to the forecasts or expectations posited by analysts regarding a company’s growth and profitability. When labelling sentiments associated with these claims, annotators are
1https://www.bloomberg.com/professional/ solution/bloomberg-terminal/
instructed to discern whether the analyst’s outlook is bullish, bearish, or neutral towards the company.
• Recognizing whether the analyst is bullish, bearish, or neutral towards the company
• Determining the duration of a bullish or bearish trend
• Deciding if the duration is explicitly stated in the report or requires subjective judgement
Premise: This pertains to events that have already transpired or are expected to transpire. In terms of sentiment labels on premises, annotators are requested to assess the event’s influence on the company, whether it is positive, negative, or neutral.
• Differentiating whether the event is past or futureoriented
• Evaluating the event’s positive, negative, or neutral impact on the company
• Indicating the impact duration from the publication date of the report
• Deciding if the impact duration is explicitly stated in the report or requires subjective judgement
• Assessing whether the event critically impacts or sparks a pivot in the company’s future
Scenario: This encapsulates potential future events predicted by analysts. Concerning the sentiment labels on scenarios, annotators are tasked to gauge the implications for the company’s future performance, i.e., whether it suggests continued growth, obstacles or decline, maintaining stability, or the prospect of significant changes or challenges. The labels are as follows:
• Determining the impact on the company’s future performance: continued growth, facing obstacles or decline, maintaining stability, or encountering significant changes or challenges
• Identifying the scenario’s impact duration from the report’s publication date
• Deciding if the impact duration is explicitly stated in the report or requires subjective judgement
• Assessing whether the scenario critically impacts or sparks a pivot in the company’s future

heading: 3.3 Inter-Annotator Agreement
text: Our annotation campaign involved five master students from the Department of Finance. We initially assigned a manageable set of reports to each annotator weekly, with the quality of their work assessed through inter-annotator agreement and weekly review meetings. In these meetings, we examined each instance that got inconsistent annotations, with the goal of establishing an acceptable
level of uniformity among annotators. On average, each annotator annotated roughly 6,200 instances across 720 reports. Annotators are paid 30% higher than the legal minimum wage.
We gauge the inter-annotator agreement by computing the pairwise F1 Score between two annotators (Yang et al., 2018). Given that annotators are not confined to annotating at the sentence level and may annotate text spans arbitrarily, we estimate the F1 Score using the following two different definitions to evaluate the agreement:
Token-level F1 Score: In this definition, each token is treated as an annotation unit, implying that each token carries a label. We then determine the F1 agreement between the two annotators using the aforementioned method.
Sentence-level F1 Score: Here, we treat each sentence as an annotation unit. If an annotator splits a sentence into two annotations, for the purposes of calculating the F1 Score, both labels are regarded as potential gold annotations. That is, if there is an overlap between the annotations of the two annotators and the labels match, it is deemed correct. Subsequently, we compute the F1 agreement between the two annotators using the described method.
Table 3 presents the agreement level for the task of argument unit identification. The agreement for the scenario label is noticeably lower compared to the other labels. This discrepancy may arise due to analysts often referencing company revenue or profitability indicators while predicting future scenarios, which could potentially be confused with the claim label. Table 4 depicts the level of agreement for the task of argument-based sentiment anal-
ysis. We have observed a lower agreement in the sentiment labeling of premises and claims, particularly concerning the assignment of the neutral label. This variation can be attributed to differing sensitivity levels among annotators when determining whether an event is positive/negative or neutral, and to the presence of both neutral and positive/negative narratives within a single annotation, leading to divergent views among annotators. In the sentiment labeling of scenarios, we notice a significantly lower agreement level than for premises and claims. This is primarily due to inconsistencies in identifying text spans that should be labeled as scenario labels within the same report, leading to an indirect impact on the agreement level for sentiment labels of scenarios.
Table 5 illustrates the agreement on argumentbased impact duration. For short-term impacts, like “within a month,” the agreement is relatively low, suggesting such impact duration might be ambiguously mentioned in reports. In contrast, broader durations such as “over a year” yield higher agreement. This suggests that more extensive time frames are frequently described with greater clarity and emphasis, facilitating easier identification by annotators. In light of the observation, we introduce “explicit” and “implicit” labels to allow annotators to indicate whether the impact duration could be clearly derived from the text or required subjective judgment. In Table 6, we also calculate the agreement based on the two labels independently. For instances where both annotators categorized as “implicit,” the Cohen’s Kappa coefficient (Cohen, 1960) value is 0.302, indicating fair agreement. However, in situations where both annotators identified as “explicit,” the Kappa value increases substantially to 0.613, signifying substantial agreement. In total, we collected 65% of ex-
plicit annotations and 35% of implicit annotations. To enhance consistency and address the aforementioned issues, we also introduced some measures. We instructed the annotators to carefully review the text for any time-related terms and to specify precise time intervals, thereby minimizing ambiguities related to time spans. Additionally, we’ve noticed that some inconsistencies between annotators are due to instances containing two distinct labels. As a result, we instruct annotators to break down the sentence into its smallest unit that explicitly represents a single label to minimize disagreement. Specifically, when annotators encounter a sentence containing two distinct labels, such as positive and negative sentiment labels in the same event, they are required to divide the sentence into two individual instances.
In addition to the F1 Score, we also utilize the Cohen’s Kappa coefficient specifically to measure the overall agreement on instances that were labeled by both. The results of Cohen’s Kappa are presented in Table 7. These results provide the answer to RQ1. The annotation results indicate that the proposed tasks are inherently subjective, particularly in terms of scenario identification and scenario sentiment analysis. This observation aligns with the characteristics of the financial market, where investors may interpret identical research reports differently, influencing their market decisions (buy/sell). During review meetings, reaching consensus on specific cases was challenging due to varying interpretations among annotators. For instance, the statement: “Dow reduced its capital expenditures from $2 billion in 2019 to approximately $1.2 billion in 2020.” was interpreted positively by Annotator A, highlighting the potential for short-term cost savings and profitability enhancements from reduced capital expenditures. Conversely, Annotator B viewed it negatively, suggesting that such a reduction in investment could hinder growth prospects and competitiveness. This discrepancy exemplifies the challenge in determining the “correct” perspective, as both viewpoints
are valid. Recent research underscores the value of learning from multiple perspectives, a promising direction that diverges from the traditional supervised learning paradigm focused on a single ground truth (Bender and Friedman, 2018; Basile et al., 2021). Consistent with this approach, our dataset includes annotations from multiple annotators, rather than providing definitive labels. To support research aimed at a singular ground truth, we organize weekly meetings with annotators to discuss cases with divergent annotations and assign final labels based on these discussions.

heading: 3.4 Dataset Statistics and Analysis
text: We have annotated a total of 37,416 instances, mostly using sentences as the annotating units, classifying these instances into premises, scenarios, or claims. Moreover, we have recognized 10,485 groups that correspond to the count of groups in which premises, scenarios, and claims are deemed correlated (i.e., group(premise, scenario, claim)). Thus, each group, anchored by a claim, represents a comprehensive argument, with supportive premises and potential scenarios related to the central claim. Tables 8 and 9 provide a comprehensive statistical breakdown for each label type. Additionally, we have enumerated the top 5 most frequently occurring indicators annotated within claims. These indicators— “Price target,” “EPS (earnings per share),” “revenue,” “YoY” (Year on Year), and “GM” (Gross Margin)—are critical metrics often scrutinized by
analysts when evaluating companies and formulating recommendations.

heading: 4 Experiment

heading: 4.1 Baseline Models and Results
text: For our experiment, we utilized pre-trained models such as BERT (Devlin et al., 2019), FinBERT (Araci, 2019), RoBERTa (Liu et al., 2019), GPT-42 and ChatGPT (GPT-3.5)3 as our baselines. Specifically, we fine-tune the models, namely BERT, FinBERT, and RoBERTa, on our tasks, with an added linear layer as a classifier leveraging the final hidden state associated with the [CLS] token. The inputs for sentiment analysis and argument unit identification are instances (a text span from a report) from our dataset. For impact duration inference, the inputs are instances (a text span from a report) along with the report’s publication date. The corresponding ground truth labels are used as outputs during model training. It is important to note that, in this paper, the test set we utilized for evaluation comprises only instances that have achieved full agreement among the annotators. The models were fine-tuned using a learning rate of 2e-5, weight decay of 0.01, batch size of 32, 5 epochs, and the AdamW optimizer. To assess the performance of these refined models, we utilized accuracy, precision, recall, and F1-score as evaluation metrics.
Table 10 presents the results on the argument unit identification task. RoBERTa exhibits superior performance among all models and achieves
2https://openai.com/index/gpt-4-research/ 3https://openai.com/blog/chatgpt
an accuracy and F1-score above 90%. Given that ChatGPT and GPT-4 operate under zero-shot and few-shot settings, and we provide guidelines, including some demonstrations for each label in the few-shot setting, to request them to generate predictions, our intention in showcasing the performance of ChatGPT and GPT-4 is not for comparison with supervised models. The performance detailed in Table 10 suggests that GPT-4 can achieve substantial agreement with human annotators. That is, when the annotators’ agreement level reaches 100%, GPT-4 attains 81.2% by following the same guidelines.
Table 10 also presents the results from the argument-based impact duration inference task. RoBERTa consistently outperforms other models across all metrics. With an accuracy, precision, recall, and F1 Score of approximately 78%. Additionally, we carry out an ablation study to understand the importance of including the report’s publication date when fine-tuning the models. The results reveal that integrating the publication date lead to an approximate 0.1 increase in the F1 Score. This underscores the significance of the report’s publication date in assessing the validity period of analysts’ forecasts and the duration of event impacts.
Further analysis of ChatGPT’s performance on specific labels indicates a pattern of low precision combined with high recall. This trend is especially pronounced for the labels: “Neutral” of premise (precision: 0.25, recall: 0.78), “Neutral” of claim (precision: 0.54, recall: 0.86), and “Steady State” of scenario (precision: 0.19, recall: 0.73). In essence, while ChatGPT often identifies a broad range of potential neutral arguments (high recall),
it is less accurate in ensuring that these identifications are correct (low precision). This suggests that ChatGPT tends to misclassify polarized arguments as neutral, resulting in numerous false positives.
Table 11 displays the experimental results of the argument-based sentiment analysis tasks. Overall, RoBERTa continues to outperform the other models when evaluated based on the F1-score. In the argument-based sentiment analysis task, the performance gap between supervised models and ChatGPT narrows, particularly in the sentiment analysis of premises. Our results underscore the potential of employing ChatGPT for sentiment analysis tasks. Nevertheless, regardless of the tasks, a performance gap persists between supervised models and ChatGPT, which affirms the necessity of the proposed manually annotated dataset.

heading: 4.2 Significance of Human Annotation
text: The recent surge in discussions has revolved around the potential of LLMs as substitutes for human annotators in both dataset construction (Latif et al., 2023) and evaluation (Chiang and Lee, 2023). To answer RQ2, we delve into the extent to which we can depend on LLM-generated labels for supervised model training. By contrasting these models with those trained using human annotations, we aim to gauge the efficacy achieved using our proposed dataset. To this end, we construct a new training set, which annotations are provided by ChatGPT. The test set is the same as that in Section 4.1. We follow the same experimental setting as shown in Section 4.1 to train BERT, FinBERT, and RoBERTa with the new training set with ChatGPT-generated labels. It is worth noting that ChatGPT uses the same guidelines as human annotators, and the instances and size in the new training set are the same
as the human-annotated training set.
Table 12 presents the performance of models on the argument unit identification task. The outcomes suggest that using manually-annotated labels consistently yields superior performance, irrespective of the underlying language model. When contrasted with Zero-shot ChatGPT performance, it becomes evident that auto-generating additional instances for training supervised models doesn’t necessarily enhance the task performance. Turning our attention to Table 12, the trends reiterate the significance of manual annotations encapsulated in our Equity-AMSA.
Nevertheless, the insights drawn from the argument-based sentiment analysis task differ to some extent. Table 13 delineates the performance metrics for the argument-based sentiment analysis task. Results for both claim and scenario sentiment analyses emphasize the pivotal role of manual annotation. Notably, there’s a marked performance disparity regardless of the model in play. In contrast, premise sentiment analysis exhibits distinct trends. Models trained on ChatGPTgenerated labels achieve F1 Scores comparable to those fine-tuned with manually-generated labels. Furthermore, supervised models leveraging ChatGPT-generated labels outperform the zeroshot ChatGPT. This suggests the latent potential of ChatGPT-derived labels for premise sentiment analysis.
Synthesizing the insights from this section, it’s evident that manual annotations predominantly elevate performance across tasks in our proposed dataset, with the lone exception being premise sentiment analysis. Given that a majority of extant research (Devitt and Ahmad, 2007; Hamborg
and Donnay, 2021) primarily orbits around sentiment tasks akin to premise sentiment analysis in this study, our findings arguably introduce a nuanced, fine-grained sentiment analysis task centered around forward-looking statements, namely claims and scenarios.

heading: 5 Predictability Assessment

heading: 5.1 Experimental Setup
text: In equity research reports, professional analysts forecast a company’s future performance, and the realization of these forecasts is a criterion for assessing their analysis (Zong et al., 2020). Inspired by Chen et al. (2019), each report encapsulates its analysis into a price target, which is the anticipated stock price level the analysts predict the company will achieve. Given the quarterly release of financial reports by companies, analysts update their forecasts to incorporate significant changes. We aim to evaluate the predictability of equity research reports by identifying the realization of the stated price targets within a three-month period. We used 1,775 reports for this experiment. For each report, we extracted historical stock closing prices from Yahoo Finance and assessed whether the price target was met within the subsequent three months. The dataset was split into a training set (80%) and a test set (20%). We prompted LLMs (ChatGPT, GPT-4, PaLM 24, and Gemini Pro5), and employed language model-based methods, Longformer (Beltagy et al., 2020), and graph neural network (GNN)based method, SAGEConv architecture (Hamilton et al., 2017) as our baselines.
The aim of using Longformer is to address token 4https://ai.google/discover/palm2/ 5https://deepmind.google/technologies/gemini/
pro/
size limitations. For fine-tuning Llama-2-7B and Mistral-7B models, we employed the parameterefficient fine-tuning method LoRA, configuring the settings with a lora_rank of 256 and a lora_alpha of 512. Regarding GNN, we created an opinion graph for each report to predict price target realization. We generated 1,775 graphs, with an average of 10.55 nodes and 13.53 edges per graph. The graph is constructed based on the "group" mentioned in Section 3.4. We connect all correlated premises and scenarios to the corresponding claim, noting that these premises or scenarios are not constrained to only one claim; they can be connected to more than one claim if applicable. Additionally, all claims are connected to a virtual node to facilitate information exchange between nodes. The SAGEConv architecture (Hamilton et al., 2017) was employed to learn node representations by aggregating information from neighboring nodes. Initially, we encode each argument unit (e.g., premises, scenarios, or claims), combined with additional data such as impact duration or sentiment labels, to form graph nodes. We then applied two layers of SAGEConv for node refinement. A comprehensive graph representation was obtained by averaging all node representations, which was then used as input for a linear classifier.

heading: 5.2 Experimental Results
text: Table 14 presents the experimental results. The distribution of achieved price targets or not is 45.61% (achieved) and 54.39% (not achieved). Although LLMs can somehow identify the predictability of the given report, they still perform worse than supervised models. The basic GNN outperforms the Longformer, Llama-2-7B, and Mistral-7B. Enhancements to the GNN architecture with the inclu-
sion of all proposed features significantly boost its performance. To address RQ3, we contrast our annotation schemes against the sentiment labels (positive, negative, neutral) derived from the VADER toolkit (Hutto and Gilbert, 2014) and a GNN utilizing only conventional argument unit definitions (claim/premise) for comparison with our extended schemes (argument-based sentiment analysis and claim/premise/scenario). The findings indicate that our annotation schemes enhance performance in evaluating equity research reports, surpassing previous approaches in sentiment analysis and argument mining. Table 15 shows the label-based evaluation of the GNN model using all labels in EquityAMSA, and it indicates that the model has high precision in identifying the reports that have high predictability and high recall in filtering out the reports that have low predictability.

heading: 5.3 Human Performance
text: To compare the ability of predictability assessment between models and human beings, we conducted a small-scale experiment with three master students from the Department of Finance who are experienced in financial analysis and have academic backgrounds in understanding financial text. We utilize a total of 75 reports, and evaluate the performance of Human, GPT-4 (zero-shot), Mistral-7B, and GNN using all labels in Equity-AMSA in predicting whether a target price would be achieved in the following three months, based on the information in the reports. Table 16 indicates that GPT-4, Mistral, and GNN outperform humans in this task. It shows the potential and capacity of the models to identify reliable reports and filter out unconvincing ones from a large volume of reports, and it helps investors quickly obtain accurate insights, viewpoints, and recommendations on the performance and potential of companies or financial assets.
Moreover, we evaluated human performance on problem sets of varying difficulty, where difficulty refers to how challenging the problems are for models. For example, Easy problems are those reports correctly predicted by all three models (GPT4, Mistral, and GNN using all labels in EquityAMSA), while Hard problems are those reports
that all three models incorrectly predicted. Each category contained 25 questions. As shown in Table 17, humans correctly predicted the outcomes for 16 out of the 25 reports in the easy problems, and for 15 out of the 25 reports in the hard problems. This indicates that humans excel in some cases where models struggle, suggesting that there are unique strengths individually in predictive models and humans for analyzing analysts’ opinions and making predictions, which could result from their different available information, background knowledge or capabilities. This implies the potential to achieve complementary team performance with Human-AI collaboration.

heading: 6 Conclusion
text: This paper proposes a novel and comprehensive approach to the automated interpretation of equity research reports by integrating argument mining with sentiment analysis, and introduces the EquityAMSA dataset. Our results show the indispensable role of human annotation in maintaining highquality data for training machine learning models. Despite the advancements in LLMs, our findings reveal that manual annotations outperform LLMgenerated labels in most tasks, highlighting the nuanced understanding humans bring to the interpretation of financial texts. Moreover, the predictability assessment through experimental evaluations demonstrates the superior performance of our proposed annotation schemes and the importance of task-specific features in enhancing the analysis of equity research reports. Our work serves as a foundation for future exploration, encouraging further investigation into the integration of argument mining and sentiment analysis for the nuanced interpretation.
Limitations
A primary limitation of this paper lies in its singular focus on one type of financial document. As elucidated by Chen et al. (2021), financial sources can be categorized into four clusters based on their providers: company/managers, professional investors, social media users, and journalists. Our choice to center on professional investors’ reports is driven by several considerations. Managers often provide limited insights into a company’s prospective operations, while journalists typically prioritize factual reporting over forecasting. This constraint curtails our ability to delve into forward-looking statements from these two sources. Social media data, on the other hand, is often cluttered and informal, making it less suitable for immediate study. Thus, we earmark it for potential future research. Given these constraints, we posit that initiating our analysis with professional reports offers a reasonable starting point for the nuanced, argument-based sentiment analysis tasks we propose. A secondary limitation pertains to the linguistic scope of our dataset, which exclusively features English. This may curtail discussions encompassing other languages. However, since many models demonstrate peak performance in English, we deem it a pragmatic starting point. Future research endeavors can emulate our task framework and methodologies to delve into other languages, juxtaposing their findings with the insights from this study.
Ethical Considerations
Publishing content on platforms distinct from the original site requires permissions or copyright transfer, hence, researchers often release URLs or tweet IDs with their annotations. Adhering to this, our Equity-AMSA dataset releases annotations with equity report file names under the CCBY-NC-SA 4.0 license. We also offer codes to reconstitute the dataset from original reports.6 Researchers can independently download reports from the Bloomberg Terminal and apply our codes to rebuild the dataset.

heading: Acknowledgement
text: This work was supported by National Science and Technology Council, Taiwan, under grants MOST 110-2221-E-002-128-MY3, NSTC 112-2634-F002-005 -, and Ministry of Education (MOE) in
6https://github.com/CYXup6/Equity_AMSA
Taiwan, under grants NTU-113L900901. The work of Chung-Chi Chen was supported in part by JSPS KAKENHI Grant Number 23K16956 and a project JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).

heading: A Prompt for LLMs
text: We used the prompt below for the experiments of PaLM 2, Gemini Pro, ChatGPT, and GPT-4 in Table 14.
Based on the following equity research report, predict the likelihood of the stated target price being achieved within the next three months, carefully considering the opinions presented in the report. Your response should be either “The target price is likely to be realized” or “The target price is unlikely to be realized.” Equity Research Report: <report>
We used the prompts below for the experiments of ChatGPT in Table 10.
You are a financial analyst. I will give you a sentence from an equity research report. Please classify the sentence into premise, scenario or claim:
Premise: Events that have occurred or are expected to occur Scenario: Possible future events envisioned by analysts Claim: Analysts’ expectations or forecasts regarding company growth and profitability Sentence: <sentence> The sentence can be classified as
You are a financial analyst. I will give you a sentence from an equity research report. Please evaluate the impact duration of the statement <sentence>, given that the report was published on <publish date>. Question: How long will the impact of the statement last? (A) within a month (B) 1-3 month (C) 4-6 month (D) 7-12 month (E) Over a year
We used the prompts below for the experiments of ChatGPT in Table 11.
You are a financial analyst. I will give you a sentence from an equity research report. Please classify the sentence into "Bullish", "Bearish" or "Neutral". Sentence: <sentence> The sentence can be classified as
You are a financial analyst. I will give you a sentence from an equity research report. Please classify the sentence into "Positive", "Negative" or "Neutral". Sentence: <sentence> The sentiment of the sentence is
You are a financial analyst. I will give you a sentence from an equity research report. Please classify the sentence into “Steady State”, “Continued Growth”, “Collapse” or “Transformation”. The definitions of the four types are as follow: Steady State: maintaining stability Continued Growth: continued growth Collapse: encountering obstacles or decline Transformation: facing significant changes or challenges Sentence: <sentence> The sentence can be classified as

heading: Future Work
text: Please classify the sentence into premise, scenario or claim:
Premise: Events that have occurred or are expected to occur Scenario: Possible future events envisioned by analysts Claim: Analysts’ expectations or forecasts regarding company growth and profitability Sentence: <sentence> The sentence can be classified as
You are a financial analyst. I will give you a sentence from an equity research report. Please evaluate the impact duration of the statement <sentence>, given that the report was published on <publish date>. Question: How long will the impact of the statement last? (A) within a month (B) 1-3 month (C) 4-6 month (D) 7-12 month (E) Over a year
We used the prompts below for the experiments of ChatGPT in Table 11. You are a financial analyst. I will give you a sentence from an equity research report. Please classify the sentence into "Bullish", "Bearish" or "Neutral". Sentence: <sentence> The sentence can be classified as
You are a financial analyst. I will give you a sentence from an equity research report. Please classify the sentence into "Positive", "Negative" or "Neutral". Sentence: <sentence> The sentiment of the sentence is
You are a financial analyst. I will give you a sentence from an equity research report. Please classify the sentence into “Steady State”, “Continued Growth”, “Collapse” or “Transformation”. The definitions of the four types are as follow: Steady State: maintaining stability Continued Growth: continued growth Collapse: encountering obstacles or decline Transformation: facing significant changes or challenges Sentence: <sentence> The sentence can be classified as