Measuring Consistency in Text-based Financial Forecasting Models
Year: 2023

Abstract:
Financial forecasting has been an important and active area of machine learning research, as even the most modest advantage in predictive accuracy can be parlayed into significant financial gains. Recent advances in natural language processing (NLP) bring the opportunity to leverage textual data, such as earnings reports of publicly traded companies, to predict the return rate for an asset. However, when dealing with such a sensitive task, the consistency of models – their invariance under meaning-preserving alternations in input – is a crucial property for building user trust. Despite this, current financial forecasting methods do not consider consistency. To address this problem, we propose FinTrust, an evaluation tool that assesses logical consistency in financial text. Using FinTrust, we show that the consistency of state-of-the-art NLP models for financial forecasting is poor. Our analysis of the performance degradation caused by meaning-preserving alternations suggests that current text-based methods are not suitable for robustly predicting market information. All resources are available at https: //github.com/yingpengma/FinTrust.

heading: 2 Related Work
text: Text-based Financial Forecasting. A line of work has leveraged event-based neural networks based on financial news for predicting the stock movement of S&P 500 companies (Ding et al., 2014, 2015; Xu and Cohen, 2018). By taking advantage of recent advances in NLP, recent work has shown potential in predicting stock price movements using PLMs, BERT (Devlin et al., 2018), and FinBERT (Araci, 2019; Yang et al., 2020b), with rich textual information from social media and earnings conference calls (Liu and Tse, 2013; Xing et al., 2020; Chen et al., 2021a). The considerable PLMs mainly include BERT and FinBERT. While BERT is trained on corpora from fairly general domains, FinBERT is trained on financial corpora, including earnings conference calls and analyst reports, under the same architecture as BERT. Although implicit stock market preference is in the masked token predictions task, the implicit preference has been under-explored using a logical behavior test.
In addition to building pre-trained models specially trained for financial domains, researchers have recently proposed myriad neural network architectures aimed at more accurate predictions to produce profitable gains including financial risk (volatility) and return predictions. For example, researchers (Qin and Yang, 2019; Yang et al., 2020a; Sawhney et al., 2021a) have considered predicting the volatility of publicly traded companies based on multi-model earnings conference call datasets. Also, Xu and Cohen (2018); Duan et al. (2018); Yang et al. (2018); Feng et al. (2019) leverage different textual data sources for predicting the stock movement based on the daily closing price. Unfortunately, despite the alarm over the reliance of machine learning systems on spurious patterns that have been found in many classical NLP tasks, the topic of text-based financial forecasting lacks a systematical evaluation regarding the robustness analysis from either an adversarial or consistency perspective. To this end, we present the first critical investigation of popular benchmarks by using FinTrust from the consistency perspective. Consistency Measurement. The inductive bias of machine learning systems is greatly affected by the patterns in training data due to the nature of inductive reasoning. While a flurry of research has highlighted this issue (Gururangan et al., 2018; Srivastava et al., 2020; Garg and Ramakrishnan, 2020; Kaushik et al., 2020), recent work Jang et al. (2022)
shows that possible artefacts in data are more influential than the model design when leading to the problem of lacking trustworthiness. Thus, assessing the influence of data artefacts, such as consistency, becomes a crucial problem for trustworthy NLP. Elazar et al. (2021) study the consistency of PLMs (e.g., BERT, ALBERT, and RoBERTa) with regard to their knowledge extraction ability and conclude that the consistency of these models is generally low. Chuang and Yang (2022) aim to raise awareness of potential implicit stock preferences based on the finding that consistent implicit preference of the stock market exists in PLMs at the whole market.
In addition to evaluating preferences in PLMs, previous methods also attempt to evaluate the consistency of models in downstream NLP tasks, such as visual question answering (Ribeiro et al., 2018), QA (Jia and Liang, 2017; Ribeiro et al., 2019; Gan and Ng, 2019; Asai and Hajishirzi, 2020), named entity recognition (Jia et al., 2019; Wang and Henao, 2021), and natural language inference (Naik et al., 2018; Hossain et al., 2020; Camburu et al., 2020; Sinha et al., 2021). Besides, Ribeiro et al. (2020) consider using consistency for building the behavioural testing benchmark beyond accuracy. Surprisingly, these discussions have not yet been extended to text-based financial forecasting models, which require strong robustness to assist decision-making in the financial market, with the exception of our work.

heading: 3 Method
text: We define the pipeline of FinTrust � in Figure 2. For text-based financial models, there are two salient components, namely text representations and financial behavior. For the former, using PLMs has become a dominant approach, improving the
quality of text representations in many domains. For the latter, various neural models can be built on PLMs. Correspondingly, we have two setups in the consistency evaluation, representation (Setup 1) and behavior (Setup 2), respectively.
Setup 1. In the first stage, we assess the implicit preferences in PLMs via masked token predictions. In particular, we first mask a predictable word from the original input extracted from earning conference call transcripts, such as "the cost of raw materials has been greatly decreased...so the expected return for the next quarter is [MASK]". Then, we predict the masked token using PLMs and compare the probability of predicting "increased" and "decreased" for contexts from different transcripts. A higher probability of predicting "increased" would indicate that the given PLM hold logical consistency with human predictions. Conversely, it suggests that the prediction of the PLM may be influenced by spurious patterns such as favoritism towards a particular stock.
Setup 2. We evaluate text-based financial forecasting models after fine-tuning PLMs on a popularly used earnings conference call dataset (Qin and Yang, 2019). However, the consistency measurement faces significant challenges in defining the relationship between two texts, particularly when the text is a long transcript with complex logical connections, such as earnings conference call transcripts. Incorrectly defining this relationship can render consistency judgments meaningless. In line with prior research (Jang et al., 2022), we develop four logical consistency transformations customized for financial text in this work. By meaning-preserving altering the original text, we ensure generated samples have a logical relationship to the original text, thus ensuring the consistency judgment is meaningful. Below we define
our text level consistency transformation first (Sec 3.1), before introducing the financial tasks for behavior study (Sec 3.2) and a wholistic metric (Sec 3.3) to integrate performance and trustworthiness.

heading: 3.1 Logical Consistency Transformations on Text Data
text: In FinTrust, four logical consistency transformation approaches are defined to evaluate if the model maintains the same logical behavior as humans, representing the consistency in text-based financial forecasting models.
Negation consistency refers to the ability of a model to generate converse predictions for texts with opposite meanings, i.e. f(x) = positive ⇔ f(¬x) = negative, where x is the input transcript, f(x) represents the output of the model, a "positive" outcome means the stock price will increase, and a "negative" outcome means the stock price will decrease. ¬x is a negation consistency transformed test example flipped through predetermined rules based on the bi-grams of the most frequent words and their antonyms. We achieve this by splitting the dataset at the sentence level and flipping the meanings of sentences. Given an input "the cost of raw materials has been greatly decreased, with a change of 30% compared with last year", its counterpart can be "the cost of raw materials has been greatly increased, with a change of 30% compared with last year". In the financial market, a significant cost reduction may lead to optimism about the company’s future prospects and an increase in stock price. Only when the model can give the correct predictions for both pairs of testing data we consider that the model is consistent with non-contradictory predictions. Otherwise, it is considered to lack negation consistency.
Symmetric consistency is the property of a model where the order of the inputs does not affect the output. It is defined as f(Sp1, Sp2) = f(Sp2, Sp1), where S is a sentence in the transcript, Spi represents the part i of the sentence. This can be tested by reordering the segments of each sentence in the transcript and comparing the predictions before and after the reordering. For example, given the sentence "the cost of raw materials has been greatly decreased, with a change of 30% compared with last year", if the prediction is reversed after reordering it to "with a change of 30% compared with last year, the cost of raw materials has been greatly decreased", then the model is regarded
as lacking symmetric consistency. Additive consistency refers to the property of a model to predict the stock movement based on the combination of two inputs, x and y that share the same label. The model is expected to hold the same prediction for x, y, and the concatenation of those inputs x+ y. If the model produces different predictions for the above three kinds of inputs, it can be regarded as lacking additive consistency. For example, if a model gives a positive prediction for the sentence "the cost of raw materials has been greatly decreased, with a change of 30% compared with last year", and also gives a positive prediction for the sentence "we believe that our products can bring convenience to everyone’s life", then it should also make a positive prediction for the combined sentences after the concatenation.
Transitive consistency refers to the ability of a model where the perceived sentiment of a company should be reflected in the performance of the top-valued company in the same industry. It can be expressed as f(x) = f(x′), where x′ represents transitive consistency transformed text. Specifically, for transcripts of a particular company, the top-valued company in the same industry is identified and its name is denoted as "company_name". Then occurrences of words such as "we" and "our" are replaced with "company_name" and "company_name’s" respectively. For example, if the corresponding sector of the company is "Information Technology" and the top-valued company in the S&P 500 is Apple Inc., a sentence such as "we believe that our products can bring convenience to everyone’s life" will be transformed to "Apple Inc. believe that Apple Inc.’s products can bring convenience to everyone’s life" after transitive consistency transformation. Again, we calculate the consistency of models by considering the non-contradictory predictions over transitive instances.

heading: 3.2 Prediction Tasks in FinTrust
text: Consistency Measurement in PLMs. To better assess the implicit preference in PLMs, we extend the previous cloze-style prompts used in assessing stock market preference (Chuang and Yang, 2022) by considering logical changes rather than simply predicting the masked token in the input. This is crucial as if PLMs are biased, the fine-tuned model’s predictions based on features learned by PLMs could be further influenced by spurious pref-
erence tendencies, which would negatively impact the effect of financial forecasting. Stock Prediction Task. Following previous studies (Ding et al., 2015; Duan et al., 2018; Sawhney et al., 2020), we treat the stock movement prediction as a binary classification problem, where the model predicts whether the daily closing price of a given asset will increase or decrease over the next n days (n=3, 7, 15, 30) based on the content of earnings call transcripts. The output is either “increase” (positive) or “decrease” (negative). Trading Simulation Task. We use the predictions to determine whether to buy or sell a stock after n days. For example, if the model predicts that the stock price would increase from day d to day d+30, we would buy the stock on day d and sell it on day d + 30. Otherwise, we execute a short sell. The previous work Sawhney et al. (2021a) simulates the trade of one hand for each stock, which allows for the potential offset of multiple forecast failures if one stock is more valuable. However, this approach is unfair under specific situations since each prediction and trade are treated equally and thus will lose the balance between trades. Therefore, we invest the same amount of money in each stock and calculate the profit ratio instead of the cumulative profit. This method does not affect the calculation of the Sharpe Ratio and allows us to explore the impact of financial forecasting consistency on performance and profitability. Notably, we do not consider the transaction cost in accordance with previous work (Sawhney et al., 2021a).

heading: 3.3 Wholistic Evaluation Metrics
text: We introduce the predictive evaluation metrics and the novel consistency evaluation metrics as elaborated below. Predictive Evaluations. For stock prediction, we use three metrics to measure performance: Accuracy, F1 score, and Matthews correlation coefficient (MCC). These metrics are calculated as follows:
F1 = 2× precision× recall precision+ recall
(1)
For a given confusion matrix:
Accuracy = tp+ tn
tp+ tn+ fp+ fn (2)
MCC = tp× tn− fp× fn√
(tp+ fp)(tp+ fn)(tn+ fp)(tn+ fn) (3)
We use both Profit Ratio and Sharpe Ratio for the trading simulation task as performance indicators.
Return R, and investment I is involved in calculating the Profit Ratio.
ProfitRatio = R
I (4)
The Sharpe Ratio measures the performance of an investment by considering the average return Rx, risk-free return Rf , and standard deviation of the investment σ(Rx).
SharpeRatio = Rx −Rf σ(Rx)
(5)
Consistence Evaluations. Based on logical transformations, we propose the consistency evaluation metrics of consistency, aiming to measure textbased financial forecasting models from a consistency perspective as a complementary metric to accuracy. Assuming that C is a set of four logical consistencies. To begin with, we define the consistency score (Consis), elaborated as follows:
Consis = ∑|C| i=1Ci |C| (6)
where the C set contains Negation consistency ConsisN , Symmetric consistency ConsisS , Additive consistency ConsisA, Transitive consistency ConsisT . We give the formal definition of those four metrics, respectively. The consistency of ConsisN is calculated as:
ConsisN =
∑|D| i=1
{ 0 (f(xi) = f(x N i ))
1 (f(xi) 6= f(xNi )) |D| (7)
where D is the original test set, xi is the test sample in the original test set, i.e. xi ∈ D. xNi is the new test sample obtained by negation consistency transformation on xi, and f(x) is the prediction of the model (positive or negative) for the input x. In terms of the symmetric, additive, and transitive transformations, the value equals 0 when f(xi) 6= f(xNi ) while equals 1 when f(xi) = f(x N i ).

heading: 4 Experiments
text: We first evaluate the explicit preferences in PLMs. Then we assess the ability of text-based models to make consistent predictions on the stock movement and finally test the profitability of these predictions using a trading simulation.

heading: 4.1 Dataset
text: Earnings Call Data. We use the publicly available Earning Conference Calls dataset by (Qin and Yang, 2019), which includes transcripts of 576 earnings calls from S&P 500 companies listed on the American Stock Exchange, obtained from the Seeking Alpha website. It also includes the metainformation on the company affiliations and publication dates. Financial Market information. We also collect historical price data (closing price) for the traded companies listed in S&P 500 from Yahoo Finance for the period from January 1, 2017, to January 31, 2018. This data was used to calculate the label of stock price movement and profitability. Data Processing. Following (Qin and Yang, 2019; Yang et al., 2020a), we split the dataset into mutually exclusive train/validation/test sets in a 7:1:2 ratio in chronological order to ensure that future information is not used to predict past price movements. We also construct logical consistency datasets based on the original test set using the above-mentioned four logical consistency transformations. The size of our evaluation dataset is four times the size of the original one since we ensure that each sample in the original test set corresponds to four logical consistency test samples. To facilitate future research, we release our dataset and the evaluation toolkit in FinTrust.

heading: 4.2 Models
text: Representation Models. We conduct experiments on popular PLMs, including BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), DistilBERT (Sanh et al., 2019), and FinBERT (Yang et al., 2020b). The vocabulary of FinBERT is different from the others as it contains domain-specific terms in the financial market, including company names. Predictive Models. Regarding the forecasting models, we evaluate several baselines, including the traditional machine learning and state-of-the-art transformer-based methods, detailed as follows.
• HTML: Yang et al. (2020a) propose a hierarchical transformer-based framework to address the problem of processing long texts in earnings call data. It utilizes a pre-trained WWM-BERT-Large model to generate sentence representations as inputs for the model.
• MRDM: Qin and Yang (2019) propose the first method to treat volatility prediction as a
multi-modal deep regression problem, building benchmark results and introducing the earnings conference call dataset.
• Event: Ding et al. (2015) adapt Open IE for event-based stock price movement prediction, extracting structured events from large-scale public news without manual efforts.
• XGBoost: Chen et al. (2015) propose a gradient-boosting decision tree known as the classical machine learning baseline.

heading: 5 Results and Discussion
text: We report the results of three tasks defined in Section 3.2 and the consistency score calculated by the consistency evaluation metrics in this section. Furthermore, we present extensive ablation studies and discussions to support in-depth analyses of each component in FinTrust.

heading: 5.1 Predictive Results
text: Consistency Measurement in PLMs. The results of explicit preferences in PLMs are presented in Table 1. In general, we find that all PLMs exhibited relatively low consistency, ranging from 70.13% to 82.7%, which falls significantly short of the level of robustness expected in the financial market. Also, we observe that PLMs typically demonstrated lower consistency when tested on
negative tokens than positive tokens (on average 63.91% – negative vs. 86.09% – positive). This suggests that popular PLMs tend to exhibit stereotypes when predicting negative tokens.
From a model-level perspective, our results indicate that FinBERT, which utilizes a domainspecific training corpus during the pre-training phase, can slightly improve consistency compared to BERT-base. Besides, we show that the increase in parameter size brings significant benefits for improving consistency, given that BERT-large and RoBERTa-large both outperform their base-sized versions (75.67% vs. 71.33% – BERT; 82.70% vs.77.79% –RoBERTa). In particular, RoBERTaachieves the highest consistency across three settings, indicating its high robustness. In contrast, DistilBERT achieves the lowest consistency. Stock Movement Prediction. The results of stock movement prediction over text-based financial forecasting models are shown in Table 2. We evaluate multiple baselines by comparing the results of models on the original test set to the results tested on transformed datasets (shown as +FinTrust). It is noteworthy that the accuracy of some predictions is even lower than that of random guess, especially for the short-time prediction (n=3). Furthermore, we demonstrate that the effect of logical consistency transformations on traditional performance indicators varies depending on the time period, but the average performance of all models decreased significantly over three metrics. In particular, models show extraordinary vulnerability when it comes to predicting the long-term stock return (n=15 and 30), as transformations in all settings decrease accuracy when the time period is 15 and 30 days.
From the model perspective, regarding the ratio of performance decay, XGBoost is the least impacted, and MRDM is the most affected. This can be because traditional machine learning models, such as XGBoost, have fewer parameters than deep learning models and are therefore less affected by
artefacts. Despite this, the accuracy on FinTrust achieved by models is only slightly more accurate than the random guess (e.g., 0.504 on MRDM, 0.507 on XGBoost). The vulnerability of these models, including state-of-the-art methods, hinders the deployment of NLP systems in the real financial market and should be taken more seriously. Trading Simulation. We compare three simple trading strategies (Buy-all, Short-sell-all, and Random) with four baselines. The results are shown in Table 3. It can be seen that HTML and Event have higher yields and can exceed simple trading strategies. However, after conducting consistency transformations, positive returns of these two methods are much reduced, even lower than the simple Buy-all strategy. Methods such as MRDM and XGBoost gain lower returns than Buy-all, with MRDM experiencing the highest drop of about 32%-43%. Even though the returns of XGBoost improved significantly after the transformations, it still remained much lower than the Buy-all strategy and the other three baselines. Hence, we contend that the increase in XGBoost’s returns does not have a strong reference value. We conclude that most methods
show unacceptably poor performance caused by lacking consistent logical behavior.

heading: 5.2 Consistency Score
text: Results. We show the results of the consistency score (defined in Section 3.4) in Table 4. It can be seen that Event has the highest consistency score (Consis) and XGBoost has the lowest Consis. Regarding the average consistency over four transformations, Event achieves three of the four highest consistency scores. XGBoost tends to make contradictory predictions in terms of the lowest scores in three settings. Additionally, all methods perform poorly on negation consistency, consistent with findings in the PLMs evaluation (Table 1). Correlation Analysis. We examine the correlation between the indicators of consistency and accuracy. Importantly, we find that our consistency score does not align with traditional performance indicators such as accuracy, evidenced by the fact that the most consistent model (Event) is not necessarily the highest in accuracy (HTML). The overall Pearson correlation coefficient between the consistency score and accuracy is only 0.314, indicating a low-level correlation. This suggests that the proposed consistency score can be used as a complementary evaluation metric for accuracy in future research on text-based financial forecasting.

heading: 5.3 Discussion
text: Human Evaluation. To assess the effectiveness of our consistency transformation method in preserving the original meaning, we conduct a human an-
notation study. Two annotators are employed from the author list and be required to label each sample and its four consistency transformations. Both of them received an advanced degree in computer science. The Inter-Annotator Agreement score is calculated to be 0.98, based on an evaluation of 40 samples and their 160 transformed samples. The average consistency score for human annotators is 0.975, indicating that our method successfully preserves the original meaning in most cases. Ablation Study. We show the ablation results in stock movement prediction of four transformations in Figure 3. We find that evaluations on the FinTrust lead to significant performance decay for most settings compared to the original performance, which illustrates the individual influence of transformations. In particular, we show that models usually underperform when evaluating the negation transformation, with the exception of MRDM. It suggests that current models lack the ability to provide non-contradictory predictions.

heading: 6 Conclusion
text: We proposed FinTrust, an evaluation tool that assesses the trustworthiness of financial forecasting models in addition to their accuracy. Results on FinTrust show that (1) the consistency of state-ofthe-art models falls significantly short of expectations when applied to stock movement prediction; (2) predictions with such a low logical consistency can lead to severe consequences, as evidenced by poor performance in a trading simulation test. Our empirical results highlight the importance of perceiving such concerns when developing and evaluating text-based financial models, and we release our dataset for facilitating future research. Despite this, how to evaluate the consistency of large-scale language models (LLMs) is still an open question
towards the financial forecasting task.
Limitation
While our pipeline is designed to be applicable to any financial text dataset, the evaluation dataset is transformed solely on earnings conference calls. We will expand the scope of experiments to include other financial text sources such as news articles and social media posts. Finally, the current trading simulation does not take transaction costs into account. Going forward it will be necessary to consider more sophisticated trading policies.
Ethics Statement
This paper honors the ACL Code of Ethics. The dataset used in the paper does not contain any private information. All annotators have received enough labor fees corresponding to their amount of annotated instances. The code and data are opensourced under the CC-BY-NC-SA license.

heading: Acknowledgements
text: We would like to thank anonymous reviewers for their insightful comments and suggestions to help improve the paper. This publication has emanated from research conducted with the financial support of the Pioneer and “Leading Goose" R&D Program of Zhejiang under Grant Number 2022SDXHDX0003, the 72nd round of the Chinese Post-doctoral Science Foundation project 2022M722836, and the financial support from the rxhui.com company. Yue Zhang is the corresponding author.

heading: A Transitive Consistency
text: Example. We show an example to understand better the motivation for using Transitive Consistency when measuring the consistency of FinNLP models. Given "Nektar Therapeutics gave investors strong confidence after Earnings Conference Call on March 1, 2017, and its stock price soared 79.43% in the following month.". As a leading company in the same Sector (Health Care), Johnson & Johnson (JNJ) was also affected by this and increased by 1.91% over the same period, which confirmed the rationality of selecting transitive consistency as one of the measurement methods.

heading: B Full Ablation Results
text: We report the ablation study results of four different types of logical transformation based on the fine-tuned forecasting models in Table 5. We use italics to indicate the performance before consistency transformation, use bold to express the performance that has been reduced after consistency transformation, and do not deal with other parts that have not decreased, for the convenience of readers. All detailed return changes in trading simulation based on text-based fine-tuned forecasting models are also shown in Table 6. "+FinTrust " means the average impact of the four transformations.

heading: C Additional experimental details
text: The model settings involved in the paper are all aligned with the parameters and training details described in the corresponding article Yang et al. (2020a); Qin and Yang (2019); Ding et al. (2015); Chen et al. (2015). The total computational budget is about 50 GPU hours, using a GeForce RTX 3090. All models use the highest performance among ten repeated experiments using different seeds and ensure reproducibility.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation Section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.
C 3 Did you run computational experiments? Section 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Our empirical results highlight the importance of perceiving such concerns when developing and evaluating text-based financial models, and we release our dataset for facilitating future research. Despite this, how to evaluate the consistency of large-scale language models (LLMs) is still an open question
towards the financial forecasting task. Limitation
While our pipeline is designed to be applicable to any financial text dataset, the evaluation dataset is transformed solely on earnings conference calls. We will expand the scope of experiments to include other financial text sources such as news articles and social media posts. Finally, the current trading simulation does not take transaction costs into account. Going forward it will be necessary to consider more sophisticated trading policies