Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledge
Authors: Agam Shah
Liqin Ye
Sebastian Jaskowski
Wei Xu
Sudheer Chava
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Knowledge Cutoff
Model Hallucinations
TL;DR: This paper evaluates Large Language Models' knowledge of historical financial data, finding they know more about larger, recent companies but are also prone to hallucinations about these firms.

Abstract: Large Language Models (LLMs) are frequently utilized as sources of knowledge for question-answering. While it is known that LLMs may lack access to real-time data or newer data produced after the model's cutoff date, it is less clear how their knowledge spans across *historical* information. In this study, we assess the breadth of LLMs' knowledge using financial data of U.S. publicly traded companies by evaluating more than 197k questions and comparing model responses to factual data. We further explore the impact of company characteristics, such as size, retail investment, institutional attention, and readability of financial filings, on the accuracy of knowledge represented in LLMs. Our results reveal that LLMs are less informed about past financial performance, but they display a stronger awareness of larger companies and more recent information. Interestingly, at the same time, our analysis also reveals that LLMs are more likely to hallucinate for larger companies, especially for data from more recent years. The code, prompts, and model outputs are available on [GitHub](https://github.com/gtfintechlab/knowledge-gap).

Published as a conference paper at COLM 2025
Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledge
Agam Shah" Liqin Ye Sebastian Jaskowski, Wei Xu, & Sudheer Chava
College of Computing & Scheller College of Business
Georgia Institute of Technology, Atlanta, GA E Corresponding Authors: (ashah482, jee4teegateh
Abstract
Large Language Models (LLMs) are frequently utilized as sources of knowl- edge for question-answering While it is known that LLMs may lack access to real-time data newer data produced after the model's cutoff date, it is less clear how their knowledge spans across Itistorical information. In this study, we assess the breadth of LLMs' knowledge using financial data of U.S. publicly traded companies by evaluating more than 197k questions and comparing model responses to factual data. We further explore the im- pact of company characteristics, such as size, retail investment, institutional attention, and readability of financial filings, on the accuracy of knowledge represented in LLMs. Our results reveal that LLMs are less informed about past financial performance, but they display stronger awareness of larger companies and more recent information. Interestingly, at the same time, our analysis also reveals that LLMs are more likely hallucinate for larger companies, especially for data from more recent years. The code, prompts, and model outputs are available on GitHub.
1 Introduction
As research and development of Large Language Models (LLMs) continues to progress, evidence has emerged of heavy usage of these models within the financial domain. Cheng al. (2024) find that trading volumes significantly decline during ChatGPT outages, indicat- ing potential reliance on LLMs by the investors, and numerous research papers have been published suggesting that LLMs can provide investment advice or generate investment strategies (Romanko al., 2023; Dong et al., 2024). Notably, Oehler & Horn (2024) com- mented that "ChatGPT provides better financial advice for one-time investments than robo-advisors" in regards to bond and equity index funds. Moreover, Yue et al. (2023) stated that LLMs can serve as a method to democratize financial knowledge. Such growing interest in LLMs in the financial industry warrants a closer investigation into the various biases and inaccuracies that may be present in these models.
In this work, we aim to identify and analyze these biases by examining where knowledge gaps in these models are present. While there has been considerable Natural Language Processing (NLP) research into gender biases (Lu et al., 2020), political biases (Motoki et al., 2023), and cultural biases (Naous et al., 2024), there has been limited study of LLM biases in the financial domain. We bridge this gap by considering how various factors such as time period, size, popularity among retail and institutional investors, market capitalization! and the readability of financial reports affect the answering capabilities of LLMs, and whether these knowledge gaps are biased along certain temporal or cross-sectional factors. Our G.T-4.5 (OpenAI, 2024), Gemin' n (Gemini Team et al., 2024), DeepSeek-va. (Liu et al., 2024), and 1.lama-3-Chen (Touvron et al., 2023), revealing temporal and cross-sectional knowledge gaps in LLMs.
study evaluates leading language models, including GPT-4c, GPT-4o-ming
Definition and further reading for financial terminologies are provided in Appendix A. "Cross-sectional analysis examines data from multiple companies at single point in time.
1

---

Published as a conference paper at COLM 2025
No Anawer
Burorr 2 105: Hallucination
Beror 106: Correct
Terporal Blas
Company Size Bias
what was the revenue of AEXTERA ENERGY in 1984?
Nhal was revenue of ROCKY MOUNTAI CIBCC FACT INC in 20187
Bext5ss Energy I8c., formerly known SP2 Grogp,, Inc.. Mas Sounded in 1925.
happy Helo! However,
clariry that theze is publicly traded company
Rosover,
't rind
Toz 1924 specirically. (Actual zevenue 28
(Macket Cap. 50N Astuai zevenue is 34.5458)
93940.28)
: What was the revenue of NEXTERA ENERGY in 2006?
what the
of SPIRIT AIRLINES
INC in 2018?
According to Securities and Exchange Commission (SEC) filings, Nextfrs Energy, inc. (formezly knewn as #eb Group, inc.) reported revenue of $22.4 brssso 2006. (Actual revenue 615710M)
According the Spirit Airliner inc. annual report for 2010 (Fora 10-R), the company's 's total operating revense for 2018 MaP
billion. (Market Cap. 3.958
Actual 1 ravanbe in $2.3233)
What the INC in 2013?
NEXTERA ENERCY
What the INC in 2015?
of MICROSOFT
According the 2013 Annual Report (Form 10-K), the company's 33 revenue for 2013 was 925. Dillion (Autual
Bnergy, Inc.
According Microsoft's annus! report for 2016 (fiscal year ended June 30, 2018), the company's revenue was $180.33 burtod (aarker Cap. 7608 Actusl
is $15138H)
revenue (110.262)
Figure 1:
Example responses from -03 showcasing knowledge biases towards older (vs. newer) financial information and smaller (vs. bigger) companies.
Our analysis shows several key findings. We see that there is a possible "retrograde" knowl- edge bias, where LLMs are unable to answer financial questions before certain time (e-g., what is a company's 's revenue back in the year 1984? See Figure 1). This complements existing research that highlights LLMs' struggles with knowledge produced after the cut-off dates of pre-training data (Onoe et al., 2022; Kasai et al., 2024). For instance, Llama-3- 3-B0B- Chat accurately answers revenue values for 54.17% of companies in 2017 while answering accurately for only 6.32% of companies in the year 1995 despite financial data being publicly available for all U.S. companies since 1995.
We also find that LLMs demonstrate better accuracy for companies with larger market capitalizations, higher attention from both retail and institutional investors, higher number of SEC filing accesses, and better filing readability. For -0-ma-3-70---- Chat, a tenfold increase in market capitalizations of the company leads to a 1.0091 rise the log odds ratio of accurately answering revenue-related questions. Such cross-sectional bias may lead to sub-optimal decision-making and unintended downstream effects in the markets. For example, bias in LLMs towards favoring larger firms may ultimately cause reduced capital allocation to smaller firms (see Appendix K).
Additionally, extend our analysis to examine how both time and firm factors relate to the model's propensity to produce factuality hallucinations, one of two types of hallucinations defined by Huang et al. (2023). Factuality hallucinations are defined as discrepancies between LLM-generated content and verifiable real-world facts. When the LLM response contains numerical value, we can establish a numerical measurement identify factuality hallucination. We calculate the error rates of LLM answers with respect to ground truth values and consider response a hallucination if the error rate is above certain threshold (10%). Interestingly, we find that hallucination occurs more in those firms which also have higher accuracy; for Ll alama- 3 7-0------ Chat, a tenfold increase in market capitalizations results in 0.1914 rise in the log odds ratio of hallucinating eveeue-aaain despite financial data being widely available.
As result of this work, we put forth the following contributions:
Utilizing diverse financial datasets to construct and analyze over 197k questions, providing insights into the performance and bias of leading L.MMs.
2

---

Published as a conference paper at COLM 2025
Evaluations
Ground Truth Answers
Cap.
Datasets
Formulate Queries
8-AIA
Cross Sectional Analysia
Temporal Analysis
zevenue of Apple 2000?
cornect
the
2000?
What the revenue of IGM Biosciences IGMS) in 2000?
Answers
What was the revenue of Apple Inc. in 1980?
Htllscmation
What was the revenue of IBM in 2018?
Analysis
Queries
T
Prompt LMs
Evaluate
Extract Answers via Regular Expression
Apple Inc. 2000 $7980 million
Apple Inc. 1980 5117.9 million
million
ICM Biosciences 2000 N/A
GFT
LIaMA-3
Figure 2: Experiment pipeline to understand and measure knowledge gap in LLMs.
Introducing a novel, systematic method for analyzing temporal knowledge biases of LLMs. To the best of our knowledge, this is the first study to analyze the retrograde knowledge bias LLMs.
Utilizing our cross-sectional analysis framework to investigate a range of factors that are associated with knowledge bias in LLMs.
Conducting comprehensive hallucination analysis that identifies a notable trend of overconfidence in LLMs, particularly evident in responses concerning specific companies and time periods.
Experimental Dataset Construction
To facilitate our analysis, we construct a novel Revenue Prompt Dataset that consists of over 197k question-answer pairs about more than 17k unique companies, spanning over 43-year period (Section 2.1). We pair this dataset with a diverse set of company characteristic variables to further study the effect of cross-sectional variables (Section 2.2).
2.1 Revenue Prompt Dataset (RPD)
We extract 197,011 revenue data points from the Compustat-IQ dataset (S&P Global Market Intelligence, 2023), spanning the years from 1980 to 2022. Each data point is a triple of the company name, year, and revenue (in million USD). Our dataset contains 17,621 unique companies listed on stock exchanges in the United States, including those that have entered exited the market due to IPOs, bankruptcy, privatization. The change in the number of companies across years in our dataset is shown in Figure 3.
We use the entire dataset of 197k samples in most of our analyses for relatively less expen- sive models (GPT-40) CGPT-40-2024-0 06, GPT-40-mine wtw.mminn2222----11111 L.l am -arra-3-BB-hhar Chat,,
'The data 13 different stock exchanges including NYSE and Nasdaq

---

Published as a conference paper at COLM 2025
and -laaa-3-700---- company of a given year (Section 3). For the more costly (GPT-4.5-previewe www.ee-eww22225------.... 27, DeepSeek-V3, and Gemini 1.5 Pro) models, we use subset (ePrr of 200 firms per year from the full sample. In addition, we create another subset (RPD430) with 430 companies that have a long public history with records available for every year between 1980 and 2022. These subsets of RPD are discussed in further detail in Appendix B.2.
by asking LLMs to answer questions about the revenue of
2.2 Cross-Sectional Variables
Besides revenue, we also draw data from multiple financial resources to consider other key variables, as summarized in Ta- ble These variables include market cap-
Number of Companies Over the Years
First Appearance rinal Appearance Ongoing
Number of Companies
italization, attention from retail investors, attention from institutional investors, the frequency regulatory filing downloads from SEC databases, and the readability of these SEC filings. We use standard com-
ed00
4000
pany identifiers, like the SEC's Central In- dex Key (CIK), to align this data with our revenue data. By examining these diverse and multifaceted factors (Section 4), we aim to shed light on the various elements that potentially shape the knowledge bias in LLMs.
3980
1985
1000
1895
2009
2005
2010
2015
Years
Figure 3: Number of companies from 1980 to 2022. 'First Appearance' counts companies entering the sample each year. 'Final Appear- ance' indicates companies exiting or appear- ing only once. 'Ongoing' tracks companies remaining in future years.
Market Capitalization (MCap): We col- lect data on the stock price and number of outstanding shares from the "Monthly Stock File" database of The Center for Research in Security Prices, LLC (CRSP), which is shared by Wharton Research Data Services. We cover the data starting in 1980 and end-
ing in 2022. After collecting the data calculate the market capitalization for company for a given day by multiplying the number of outstanding shares with the price of the share on the day. For each financial year, we use the market cap value on the last trading day as the value for that year. The market cap values are then adjusted for inflation, and logarithmic transformation is applied for better scaling (further details in B.1).
Data (Source)
Key Variables
Years
#Data Points
#Unique Companies
Annual Financials (Compustat Capital-IQ)
Revenue
1980-2022
197,011
17,621
Market Capitalization (CRSP MSF)
Price and Shares
1980-2022
197,011
17,621
Robinhood (Robintrack) Bloomberg AIA (Bloomberg)
# Holders
2018-200
9,592
3,469
B-AIA
2010-2020
21,303
2,432
SEC Access (SEC-FDGAR)
2003-2017
58,519
7,581
Bog Index (Bonsall IV et al., 2017)
Bog Index
1994-2020
110,161
12,822
Table 1: We utilize wide range of finance datasets from notable sources. More details about each dataset can be found in the Appendix B.1.
Robinhood Retail Attention: To understand how the popularity the stocks among retail investors correlates with the model's ability to answer the question for those companies, we collect from Robintrack, a dataset produced by the popular stock trading platform Robinhood. Robintrack keeps track of how many Robinhood users hold particular stock over time. In particular, we collect the popularity metric which represents the number of unique user accounts that hold at least one share of the stock. To ensure comparability with the other variables in our analysis, we apply standardization to the data, involving mean subtraction followed by division by the standard deviation. We utilize each firm's Ticker (a
4

---

Published as a conference paper at COLM 2025
short string used to identify firm listed on exchange) for matching with other datasets. We have this data available only from 2018 to 2020.
Bloomberg Abnormal Institutional Attention (B-AIA): tention, sourced from Bloomberg, caters to a limited audience of around 320,000 subscribers, mainly institutional investors, who are subject to the high cost of $24,000 per annual sub- scription. We collect the data from Bloomberg Terminal which we have accessed through our institution. We utilize methodology used in Chava & Paradkar (2016), which assigns an hourly attention score for each stock by comparing the average count of news article views and searches over the past 8 hours to the distributions of these values over the past 30 days. By aggregating the highest hourly attention scores into a daily attention score, we gain insights into the institutional investor interest. We apply standardization the data similar to Robinhood data. Refer to Appendix B.1 for more details.
Our measure of institutional at-
SEC Access: SEC filings include financial statements and other formal documents sub- mitted to the U.S Securities and Exchange Commission (SEC). As proxy for a company's popularity among investors, we utilize over 58,000 searches on SÉC.gov from February 2003 to June 2017, recorded in the EDGAR Log File, converting queries into CIK-year pairs to measure average daily filing accesses. We apply standardization to the data similar to Robinhood data.
Bog Index for Readability: We also consider the readability of the financial reports compa- nies filed with the SEC to explore how variations in readability may correlate with language models' ability to answer questions about those companies. We utilize the Bog Index (Bon- sall IV et al., 2017) readability score, which is commonly used for finance fillings (Bonsall et al., 2017), for the 110,161 SEC 10-K filings from 12,822 companies between 1994 and 2020 in the Bog Dataset. A higher score equates to less readable document.
3 Temporal Finance Knowledge Gaps in LLMs
Since 1995, annual regulatory filings of publicly traded companies in the U.S. have been transparently available in the Securities and Exchange Commission (SECYs EDGAR database. Given this, we postulate that the models might have an ascending knowledge proficiency from 1995 onwards, as exemplified with NextEra Energy, Inc. in Figure (left). We use our prompting dataset (RPD) to test four LLMs (GPT- 2024--0--22----00 GPT-40- mini-2024-07-22 Llama-3 3- 8B BB-Chat, and Llama- 3- 70B-Cha C on their temporal knowledge gap, checking if these models have greater revenue knowledge for more recent years, especially years after 1995.
Revenue QA Prompts. We construct LLM prompts (Figure 2) programmatically using papaaniial__arr??? a template: What was the revenue of (company companynaamell in financial year ifinancial yearl? This simplistic prompt design is motivated by the observation that non-professional investors with limited familiarity with AT technology are inclined to rely on readily accessible LLMs. Given the potential for inaccuracies in these models, such vulnerable populations could face significant adverse consequences (Chava et al., 2022). While more extensive prompt engineering with few-shot demonstrations may yield more precise answers, many general users may not employ sophisticated prompting strategies or engage in further fine-tuning of the models. We thus perform zero-shot prompting on the four LLMs with a temperature value of 0.00 (for reproducibility) and max_ mok_e value of 100. Based on the output of the model, we utilize regex to extract the numerical value of revenue, if is available, and standardize currency units to millions of US dollars. Further implementation details for models are provided in Appendix C, while the effectiveness of regex is discussed in Appendix D. More discussion on why we choose "revenue" as a question is provided in the Appendix J. Additionally, more sophisticated revenue prompting using Chain-of- Thouught for stock recommendation is detailed in Appendix K.
Metrics for Temporal Analysis. t based the model's answer to evaluate the model performance. The variable Y₁, takes
We use a ternary outcome variable YG, for firm i and year
5

---

Published as a conference paper at COLM 2025
the value of 2 if absolute % difference of the numerical answer of revenue is less than 10% away from the ground truth as a success, if an absolute difference of the answer is greater than 10% as factual hallucination, and 0 if no numeric answer is provided. In A Appendix E, we show that our results are consistent even if this error threshold is varied.
2 absolute % error < 10%
Yu =
1 absolute % error 2 10%
0 numerical answer
We analyze the temporal trend over the years by calculating the success rate and hallu- cination rate. For each given year, the success rate is ratio of the number of successful responses to the total number of data points, while the hallucination rate is ratio of the number of hallucinatory responses to the number of data points where the model fails. For clarity, we define both success rate and hallucination rate in Equation below.
Eit-T (
Success Rate (T)
Σ11-
(1)
ΣiN= (Y==1)
Hallucination Rate (T) Σ I (8,122
Hallucination Different
25
LLaM--3-SB-Chat
LLaMA 3.88 Chat
LLMNA3-09-ChhB
GPT 4n-nii
GPT-fo-mini
GPT-46
GPT-40
75
50
30
25
25
1360
2000
2020
2000
2020
Figure 4: Success and hallucination rates of GPTs and Llamas over time. The dotted line is drawn at the year 1995. The shadow area around the line the standard deviation of model performance.
Results. In Figure (left), we present the temporal success rate trends of four models. This result underscores our claim that LLMs demonstrate heightened proficiency in answering questions from more recent years as opposed to earlier ones. It's pertinent to highlight the dotted line at the year 1995, signifying the inception of the SEC's EDGAR filing system. After this date, detailed financial information from US public companies became publicly accessible online, thus augmenting the datasets available for model training. Intriguingly, there noticeable dip in the performance of all models in the years 2021 and 2022 when compared against their performance in 2018 to 2020. Similar results for GPT-4. DeepSeek-Ve -V3, and Gemini 1.5 Pro on ReD2n0perYear sample are presented in Figure of Appendix Investigating the underlying reason behind this deviation promises to be compelling avenue for further research.
In Figure (right), we present the percentage of companies for which the model hallucinates revenue information over the years. Among the four models, L1ama-3- 3-------7---- halluci- nates at a much higher rate over recent years. Interestingly, these models are more likely to hallucinate for the same years where they also more likely to provide the correct answer.

---

Published as a conference paper at COLM 2025
4 Measuring LLMs' Cross-Sectional Biases in Finance
In addition to temporal knowledge gap biases (Section 3), we also explored how various cross-sectional factors affect the distribution of knowledge gaps. In financial literature, there have been studies showing relation between analysts' biases and the size of the firm (Van Binsbergen et al., 2023) and the role institutional attention (Ben-Rephael 2017) in helping incorporate information in asset prices. Nevertheless, to the best of our knowledge, there is no study analyzing the relationship between firm-level information (size, institutional attention, popularity, etc.) and knowledge biases in LLMs. We extend our analysis to address this research gap and further explore how cross cross-sectional factors affect LLM knowledge and hallucination rates.
al.,
4.1 Metrics for Cross-Sectional Analysis
We use the same ternary outcome variable Y that indicates model performance discussed in Section for our temporal analysis. To capture the relationship between the model's answer and company-level characteristics, we run the following logistical regression with i1, company-level characteristics:
logit(P(Y;: = y)) aa- B X1 + Dj + Citt
(2)
Here Y is the outcome variable where indicating model success while y indicating hallucination, 6, is year-fixed effect, is a constant term, and E₁ is an error term. The coefficient & will help understand the influence of company-level characteristic X on the outcome variable (Y₁,).
4.2 Results and Analyses
Llama-3-8B a
Llama-3-70B
GPT-to-mini
GPT-4o
B
MCap (log)
-13.0984;
1.046t
-11.8915t
1.0091
-15.1833
1.2879:
-10.9046;
0.9209t
retail_inv (std)
-1.28111
0,20731
0.17551
0.1036
-1.1033
0.57271
0.95621
0.25891
B-AIA (std)
-1.1272t
0.0187
-0.1870t
0.0107t
-1.0425
0.02174
0.6939;
0.0180f
SEC-Access (std)
-2.4444
0.0628$
-1,5922t
0.0782$
-2484003
0,1028;
-0.8490t
0,0770:
Bog Index (std)
-2.5228
-0.09655
-1.83611
-0.06661
-2,4133
-0.0296
-1.5021
-0.07533
Table 2: Empirical cross-sectional regression results for market cap (MCap), number of retail investors on Robinhood (retail inv), Bloomberg abnormal institutional attention (B-AIA), number of access on SEC-EDGAR (SEC-Access), and measure of readability (Bog Index). "std" in the parentheses indicates the data is standard normalized. t, and & indicate p-value at the 10%, 5%, and 1% levels, respectively for regression coefficients.
To investigate the influence of a company's market capitalization (size) the profi- ciency of LLMs in answering the company's financial details, we conducted logistic re- gression analysis as outlined in Equation In this regression, we adopted the loga- rithm (base 10) of the company's market cap as the independent variable, denoted as X The outcomes of this regression analysis are presented in Table The data suggests that LLMs exhibit enhanced performance for companies with larger market capital- ization. A tenfold increase in the market cap corresponds to an increment of 1.2879 in the log odds ratio of the GPT-4o-mint model answering revenue-related questions
Model
Constant (a)
Beta (6)
Llama-3-8B
-6.7754t
0.6053
Llama-3-70B
-2.88591
0.1914
GPT-4o-mini
-6.21881
0.5222
GPT-4o
-2.1171
0.0964
Table 3: Market cap analysis results on halluci- nation based on the empirical regression. *, +, and indicate p-value at the 10%, 5%, and 1 1% levels, respectively for regression coefficients.
7

---

Published as a conference paper at COLM 2025
correctly, and an increase of 1.0546, 1.091 and 0.9209 for the Llama- 70B-Chat, and GPT-4o models respectively. Similar results for GPT-4. 5, DeepSeek-V3, and Gemini 1.5 Pro RPD sample are presented in Table of Appendix G. The Results of Bloomberg AIA, SEC Access, and retail investment from Robinhood data follow a similar trend. As a higher value of the Bog Index represents that the filing is less readable, the variable has negative beta indicating that the lower readability leads to decrease in the performance of LLMs.
L1ama-3-
We similarly regress hallucination on the log market cap using Equation 2 by setting y-1. Based on the results Table 3, we find that the model is more likely to hallucinate for the companies with higher market cap. Similar results for GPT-4. 5, DeepSeek-VV and Gemini 1.5 Pro on RPD200per sample are presented in Table 6 of Appendix G. Com- bining the result of Table 2 and Table 3, we can hypothesize that the model is more likely to hallucinate for the same companies for which it is also more likely to provide the correct answer (have knowledge).
For further analysis, we count the number of years for which a given company pro- duces LLM hallucinations and the number of years for which it provides the correct answer. The visualization of such relation- ship for Llamm-233 3- 3-B Chat in Figure 5, shows that the revenue of companies with higher market cap (indicated with color close to light green) have a higher propen- sity to be answered correctly and halluci- nated at the same time. Similar results for GPT-40 are provided in Figure 8 of Ap- pendix H for the robustness of the finding. We hypothesize these results might be due to the model having higher confidence for relatively recent years and those higher mar- ket cap companies.
Hallucination Analysis for
11.0
70B-Chat
Holluciation Yeans)
3.5
10 5
8.0
10
15
20
25
30
35
SuccasE Count (A) Years)
Figure 5: Each dot represents company with a count on the x-axis indicating the number of years for which correct answer and the y-axis indicating the number of years for which the model hallu- cinated in the answer. We take an average of market cap to assign color to the dot.
Discussion
3- 70B gave the
Our results highlight sigmificant biases in the financial knowledge gaps of LLMs, along both temporal and cross-sectional di- mensions. These biases are likely to have substantial effect on the efficacy of LLM use both in the financial domain and across other domains well. In addition, as LLMs continue to be touted as a "democratizing" force for financial knowledge, is important to consider what effects biases may have on this process.
Investors vs. Researchers: Investors employing LLMs should recognize the bias risks highlighted in our work and avoid relying on LLMs for factual information or open-ended analysis. Biases identified, such as favoring larger firms, may unintentionally reduce capital allocation to smaller firms (see Appendix K). These biases pose particular harm to inexperienced investors, who may struggle to detect inaccuracies from L.M.--gnerated content. For instance, Kumar (2009) finds younger investors prone to investing in "lottery- type" stocks, making them especially susceptible to biases in LLM-driven advice. Financial literacy educators should clearly communicate these bias-related risks in their teachings.
Researchers studying applications of LLMs in finance should carefully consider the various strata their tasks span, ensuring comprehensive experimental coverage. They should include firms across diverse years, sizes, and prominence. Attention to stratified biases strengthens results and ensures findings not artifacts of model biases limited to specific strata.
8

---

Published as a conference paper at COLM 2025
Broader Impacts: While our work focuses finance, our temporal and cross-sectional evaluation framework is widely applicable to assessing LLM knowledge across diverse domains. Many knowledge-intensive LLM tasks can be analyzed temporally or across specific entity-based strata. For example, in legal applications, one could evaluate LLM performance on laws enacted in different periods (e-g., comparing accuracy on laws from 2018 versus 1890) or laws receiving varying media coverage.
To illustrate our framework's versatility, we evaluate Llama3-700 70B on answering questions about annual La Liga soccer championship statistics. Specifically, we assess accuracy on questions like: "In the years La Liga season, how many points did team finishh with?". We observe higher accuracy for recent seasons and more prominent clubs, indicating biases toward recent data and larger entities. This demonstrates our evaluation framework's broader applicability, detailed methodology and results are presented in Appendix L..
Related Work
We briefly review existing research on societal and political biases, as well as hallucinations in LLMs. Our study uniquely contributes as the first, to our knowledge, to retrospectively assess knowledge biases in LLMs. Moreover, our research pioneers the exploration of various factors influencing financial knowledge bias in these models.
There has been much research involving the evaluation of LLMs' societal bias, such as gender, religion, race, politics, etc (Zhao et al., 2017; Lu et al., 2020; Sheng et al., 2021; Nozza et al., 2022; Kotek et al., 2023; Zhao et al., 2023; Blodgett et al., 2020; Galarnyk et 2025). Inspired by software testing, Nozza et al. (2022) proposed a systematic way of integrating societal bias testing into development pipelines. Zhao et al. (2023) unraveled the LLMs' "re-judgment inconsistency" in bias evaluation by leveraging psychological theories. Gender bias, as the issue most frequently addressed, has been resolved by diversified methods. Zhao et al. (2018) curated new benchmark dataset WinoBias for testing and dispelling gender bias in LLMs. Kotek et al. (2023) use a simpler paradigm to identify gender bias and reveal that LLMs still tend to reflect the imbalance of their training dataset even after aligning with human preference. Additionally, Motoki et al. (2023) extends bias evaluations to political domains, revealing significant biases in ChatGPT towards certain political groups, necessitating further scrutiny measures in training processes.
Biases in LLMs:
Research on hallucinations in LLMs has produced several innovative methods for quantifying and addressing the issue. Zellers et al. (2019) introduced Grover, model for generating and detecting "neural fake news," " studying LLMs' tendency to hallucinate. Goyal & Durrett (2020) proposed assessing text-generation factuality by analyzing dependency-level entailment with source content. Lin et al. (2021) developed dataset to test model truthfulness, indirectly evaluating hallucinations by alignment with facts. Huang et al. (2023) provides a comprehensive overview, categorizing hallucinations into factuality and faithfulness types. To our knowledge, our study is the first to measure hallucination in LLMs within the financial domain.
Hallucination in LLMs:
Recent advancements in LLMs have significantly impacted the finan- cial domain. Nie et al. (2024) introduced CFinBench, a comprehensive Chinese financial benchmark designed assess LLMs' financial knowledge across various categories, in- cluding financial subjects, qualifications, practices, and laws. Similarly, Xie et al. (2023) developed PIXTU, a framework comprising financial LLM fine-tuned with instruction data, alongside an evaluation benchmark covering various financial tasks. Kosireddy et al. (2024) explored the readiness of small language models for democratizing financial literacy, analyzing their performance in financial question-answeriig Chava (2023) benchmarked the zero-shot performance of LLMs like ChatGPT on financial tasks, comparing them with fine-tuned models to assess their effectiveness in the financial domain. Collectively, these studies underscore the growing role of LLMs in enhancing financial applications and literacy.
LLMs in Finance
tasks. Additionally, Shah &

---

Published as a conference paper at COLM 2025
7 Conclusion
In this study, we examine both the temporal knowledge bias and bias across firm- characteristic strata in LLMs using financial data from U.S. publicly traded companies. Our findings reveal that LLMs are more proficient with recent financial information, espe- cially after the 1995 introduction of the SEC's EDGAR filing system. However, there's an unexplained dip in performance for 2021 and 2022. Secondly, LLMs demonstrate better accuracy for companies with larger market capitalizations, higher retail investment, higher institutional attention, higher number of SEC filing access, and higher readability. We also find that the years and companies for which LLMs are more likely to provide a correct an- swer, they also more likely to hallucinate. In essence, while LLMs offer valuable insights, their limitations in financial knowledge necessitate careful usage, especially in professional financial domains. Future work should explore the reasons behind these trends and enhance LLMs' performance breadth, as well as expand our methodology to other domains. This study further contributes to the discourse on how the availability of the pre-training corpus of LLMs for independent scientific scrutiny can facilitate scientific advancement.
Contribution
AS conceived the research direction, acquired data, developed the Python for analysis, and conducted the analysis. LY and SJ developed the Python scripts for analysis and constructed the visualizations. AS, LY, SJ, and WX helped write the manuscript. WX and SC supervised and guided the project.
Acknowledgements
We did not receive any specific funding for this work. We thank Arnav Hiray, Michael Galamny,, and Rohan Ganduri for their comments.
Ethics Statement
Our work adheres to ethical considerations, although we acknowledge certain biases and limitations in our study.
The data used in our study, which is derived from publicly available sources, does not raise ethical concerns. All raw data obtained for public compa- nies that are obligated to disclose information under the guidance of the SEC and are subject to public scrutiny.
Data Ethics:
The language models employed (with proper citation) in our research are publicly available and fall under license categories that permit their use for our intended purposes. While most models (Llamas) employed are publicly available, it is important to note that prompt answers of GPT-4o will be made public under OpenAT's terms of use. The terms of use of OpenAl do not allow the use of prompt outputs for building competing models. Given the nature of our data, we believe this condition does not diminish the use of our work. We acknowledge the environmental impact LLMs have but we believe that the impact from just inference is limited.
Language Model Ethics:
Dataset Ethics: We will make any raw data used for the project public but we will make all the revenue prompts (RPD) and their answers public on our GitHub repository.
Limitations: We do not run the analysis for GPT-4. DeepSeek-V3, and Gemini 1.5 Pro on the full sample due to high API cost. While we provide small analysis for Llama-basee financial LLMs in Appendix F, We are unable to run analysis on finance domain-specific pre-trained LLMs like BloombergGPT (Wu et al., 2023) as these models are not publicly available.
10

---

Published as a conference paper at COLM 2025
References
Azi Ben-Rephael, Zhi Da, and Ryan D Israelsen. It depends on where you search: Institu- tional investor attention and underreaction news. The Review of Financial Studies, 30(9): 3009-3047, 2017.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of" bias" in nlp. Xio preprint arXio:2005.14051 2020.
Samuel B. Bonsall, Andrew Leone, Brian P. Miller, and Kristina Rennekamp. A plain english measure of financial reporting readability. Journal of Accounting and Economics, 63
(2):329-357, 2017. ISSN 0165-4101. doi: https:///oo.org//...0////aaacccc 201.:/cccc...222............
URL https: /ttppp:://ww...e./ www.sciencedirect.t..mmmm com/ http://rere.cccccceeeeeerrrtttle /tt.eee///rrriiceee/a/////111111
Samuel B Bonsall IV, Andrew J Leone, Brian P Miller, and Kristina Rennekamp. A plain english measure of financial reporting readability. Journal of Accounting and Economics, 63 (2-3):329-357, 2017.
2017.
Sudheer Chava and Nikhil Paradkar. December doldrums, investor distraction, and stock market reaction to unscheduled news events. Available at SSRN 2962476, 2016.
Sudheer Chava, Fred Hu, and Nikhil Paradkar. Gambling crypto tokens? Journal of Financial and Quantitative Analysis, pp. 1-55, 2022.
Qiang Cheng, Pengkai Lin, and Yue Zhao. Does generative ai facilitate investor trading? evidence from chatgpt outages. Evidence from ChalGPT Outages (June 21, 2024), 2024.
Zihan Dong, Xinyu Fan, and Zhiyuan Peng. Fnspid: A comprehensive financial news dataset in time series. arv preprint arXio:2402.06698 2024.
Edwin J Elton, Martin J Gruber, and Christopher R Blake. Survivor bias and mutual fund performance. The review of financial studies, 14):09771 1996.
Michael Galarnyk, Agam Shah, Dipanwita Guhathakurta, Poojitha Nandigam, and Sud- heer Chava. How inclusively do Ims perceive social and moral norms? arv preprint axXiv:2502.02666, 2025.
Gemini Team et al. Gemini: A family of highly capable multimodal models, 2024. URL https:///aaxi..or/abb////2
11805.
Tanya Goyal and Greg Durrett. Evaluating factuality in generation with dependency-level entailment. Xiv preprint xxXiv:2010.05478, 2020.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination large language models: Principles, taxonomy, challenges, and open questions. Xio preprint ar xrXie:2311.05232, 2023.
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. Realtime qa: What's the answer right now? Advances in Neural Information Processing Systems, 36, 2024.
Tagore Rao Kosireddy, Jeffrey David Wall, and Evan Lucas. Exploring the readiness of prominent small language models for the democratization of financial literacy. In Sachin Kumar, Vidhisha Balachandran, Chan Young Park, Weijia Shi, Shirley Anugrah Hayati, Yulia Tsvetkov, Noah Smith, Hannaneh Hajishirzi, Dongyeop Kang, and David Jurgens (eds.), Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for Domain, Application, Group, or Individual (CustomNLPALD, pp. 124- 149, Miami, Florida, USA, November 2024. Association for Computational Linguistics. www.oloonntooyoooooomooooooooooe URL https: / .erllooyyyooorogg/oee orgg20224 cus mustomnlp44-11
Hadas Kotek, Rikker Dockum, and David Q Sun. Gender bias and stereotypes in large language models. ar Xie preprint arXit:2308.1499 14921, 2023.
11

---

Published as a conference paper at COLM 2025
Alok Kumar. Who gambles in the stock market? The Journal of Finance, 64(4))1889-193) 2009. doi: ary wry.wiley.com cww/doi/abs//iii 10. 1111/j........ 01144066661.200 0e222009..888...
209..009.0........ URL https: /nnlinlbbb
Stephanie Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. Xre preprint urXio:2109.07958, 2021.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint ar Xi4:2441..1943, 2024.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. Logic, Language, and Security; Essays Dedicated to Andre Scedrov on the Occasion of His 65th Birthday, pp. 189-202, 2020.
Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues, More human than human: Measuring chatgpt political bias. Public Choice, pp. 1-21, 2023.
Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16366-16393, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: http::::///alaahhoooooo/o hww.hhooloooooo.o22.....o wtg/2024.acll11 ong.863 862.
Ying Nie, Binwei Yan, Tianyu Guo, Hao Liu, Haoyu Wang, Wei He, Binfan Zheng, Weihao Wang, Qiang Li, Weijian Sun, et al. Cfinbench: A comprehensive chinese financial benchmark for large language models. arXiv preprint arXiv:2407.02300 2024.
Debora Nozza, Federcio Bianchi, Dirk Hovy, et al. Pipelines for social bias testing of large language models. In Proceedings of BigScience Episode# 5- on Challenges & Perspectives in Creating Large Language Models. Association for Computational Linguistics, 2022.
Andreas Oehler and Matthias Horn. Does chatypt provide better advice than robo-advisors? Finance Research Letters, 60:104898, 2024.
Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What LMs know about unseen entities. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 693-702, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18657 vv1 /wn////...iinnss-aaaaaaaa findings naacl.
URL https:
//aclanthology org/2022. 2ealanthologg.org//222...
OpenAL. Gpt-4 technical report. Technical report, OpenAI, 2024. Available at https: //doi.org/10 41811...585//iii 0055500/ariv..333.00777 2303.08774.
Martin Rohleder, Hendrik Scholz, and Marco Wilkens. Survivorship bias and mutual fund
performance: Relevance, significance, and methodical differences. Review of Finance, 15(2): 441-474, 2011.
Oleksandr Romanko, Akhilesh Narayan, and Roy H Kwon. Chatgpt-based investment portfolio selection. In Operations Research Forum, volume 4, pp. 91. Springer, 2023.
Agam Shah and Sudheer Chava. Zero is not hero yet: Benchmarking zero-shot performance of Ilms for financial tasks. XrXiv preprint ar #xXio23355166333 2023.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. Societal biases in language generation: Progress and challenges. Xrv preprini arXio:2105.04055 2021.
S&P Global Market Intelligence. Compustat north america database, 2023, URL https: //www./arkkttettteee www.mmarketplace..co .pp.llolllc...llllllll..... com/en/datasetss www.mpusttt-iinnnncccee (8).
12

---

Published as a conference paper at COLM 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et Llama Open foundation and fine-tuned chat models. arXiv preprint (rXi::2307.09288, 2023.
Jules H Van Binsbergen, Xiao Han, and Alejandro Lopez-Lira. Man versus machine learning: The term structure of earnings expectations and conditional biases. The Review of Financial Studies, 36(6):2361-239) 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https: /wwp///wwww///////////// acwweb.. org /anthology/ 2020. mmllp-demos.. 6.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 175664 2023.
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira and Jimin Huang. Pixiu: A large language model, instruction data and evaluation benchmark for finance. ar arviv preprint ar xrXiv2300..05443, 2023.
Thomas Yue, David Au, Chi Chung Au, and Kwan Yuen Iu. Democratizing financial knowledge with chatgpt by openai: Unleashing the power of technology. Available at SSRN 4346152, 2023.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roes- ner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 2979-2989, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/ 100665/vv1/11// v1/D17-1323
URL httphs////caaatthoooog wtt.:loyyy.og//1112222333
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gen- der bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint
2018
Yachao Zhao, Bo Wang, Dongming Zhao, Kun Huang, Yan Wang, Ruifang He, and Yuexian Hou. Mind mouth: On measuring re-judge inconsistency of social bias in large language models. ar Xiv preprint ar Xiv:2308. 125578888.27 2023.
13

---

Published as a conference paper at COLM 2025
A Financial Lexicon: Definitions and Further Reading
Below we provide standard definitions of the finance-related lexicon used in the paper with the reference to further readings.
The total number of shares that are currently owned by all shareholders, including share blocks held by institutional investors and restricted shares owned by the company's s officers and insiders. Further reading: https: / //ww.innestoopdd. com/ terms/ / outs looosttttiiii wettstaannnnnnnneee asp
Outstanding Shares:
Market Capitalization (MCap): The total market value of a company's 's outstanding shares of stock. It is calculated by multiplying the current market price of one share by the total number of outstanding shares. Further reading: https: http://wwwwwwwtt.o//////// investop edia. ctw.cooo//aoooo ttt.eee//maareetaaaaaa talization.
Consumer Price Index (CPI): A measure that examines the weighted average of prices of a basket of consumer goods and services, such as transportation, food, and medical care. It is calculated by taking price changes for each item in the predetermined basket of goods and averaging them. Further reading: https: / wwt.co/////////e coom/ters/coo wwwoonooeeprieennnneeeoo asp
The total amount of money generated by the sale of goods services related to the company's primary operations. Further reading: https://wwe inve stopedia. comm/ters////eeeeeeeeeeeeeeeeeee httemm//rrrreeuueeee asp
Revenue:
Robinhood: A brokerage firm that offers commission-free trading of stocks and exchange-traded funds. Further reading: https://robinhood. www.ws//e/eaeaeeee/eeetcet.. ha///////abbuuuuu// s/
Ticker: A unique series of letters assigned to security for trading purposes. It is also known as a stock symbol. Further reading: erms/s/sttocksymbot
com/t
IPO (Initial Public Offering): The process by which a private company can go public by the sale of its stocks to the general public. could be a new, young company older company that decides to be listed on an exchange and hence goes public. Further reading: httpp://www/ investopedia com/ terms/ htppp.p.appp asp
Bankruptcy: legal proceeding involving person or business that is unable to repay their outstanding debts. The process begins with petition filed by the debtor or on behalf of creditors. Further reading: https: //ww:. invest com/ terms /b/bankruptcy. asp
Privatization: The transfer of business, industry, or service from public to private ownership and control. Further reading: http:://www.iivssopeelee. cwm/ termsp /privatization.a)
B Supplementary Data Information
B.1 Details of Data Sources
Compustat Capital-IQ: We only keep companies that provide consolidated (i.e. combined accounts for parent and subsidiary) financial statements and which report data in "Standardized" format according to Compustat Capital IQ, This will exclude some companies located outside the U.S. (but listed in the U.S.) as they are not required to report consolidated financial statements.
Inflation Adjustments and Scaling for Market Capitalization: For better scaling when running analysis, we convert the market capitalization values log values with a base of 10. We then normalize these values to adjust for inflation using the Consumer Price Index (CPI) data collected from FRED (Federal Reserve Economic Data). The inflation-adjusted market cap is converted to values corresponding to December 2022. The formula used to adjust the market cap values of company on date for inflation is as follows:
14

---

Published as a conference paper at COLM 2025
MCapaı MCapiss CPI(r)
CPT(f)
where MCapiir represents the adjusted market capitalization of company on date 7, MC denotes the market capitalization company as measured on date t, while CPI(r) and (P(()) are the CPI for dates and respectively.
Robintrack: In this data when we share it only includes normal, long shares (not options).
The methodology employed to quantify institutional investor attention involves process, as outlined by Bloomberg. Each news article read is assigned score while searches are weighted more heavily score of 10. These activities aggregated an hourly basis, and the attention score is derived by comparing these hourly counts to the previous month's average, with adjustments for deviation levels. Scores range from 0 to 4, reflecting varying degrees of attention based on the percentiles of activity compared to the prior month. This process effectively captures the intensity of investor focus stock, with daily scores determined by the peak hourly attention score. For additional methodological details, refer to Chava & Paradkar (2016).
B-AIA:
We use the following identifiers from datasets to merge them for analysis:
Compustat Capital-IQ: GVKEY, Company Name, Ticker, CIK
CRSP MSF: PERMNO, Ticker, Company Name
Robintrack: Ticker
B-AIA: Ticker
SEC Access: CIK
Bog Index: GVKEY
B.2 Data Samples
The sample size for each sample is listed in Table 4.
RPDa Given the high API cost for GPT-4 DeepSeex-V3, and Gemini 1.5 Pro, we create subset representative sample of our data. To do so we categorize the log market cap of the companies into 4 categories (i.e., <8.00, 8.xx, 9.xx, >=10.00) After that, we randomly sample 50 companies from each year and market cap categories making it 200 samples per year. For 43 years, the total number of samples will be 8,600 (200*43). The result of GPT-4. 4.5. 5, DeepSeek- and Gemini 1.5 Pro on RPD200errYer sample is presented in Appendix F and G.
RPD430: As new companies can go public and existing public can get delisted, the set of companies changes every year. To analyze the same set of companies over time, we created a sample of companies that were public for every year from 1980 to 2022. We have 430 companies in sample. For 43 years, the total number of samples will be 18,490 (430+43). The result of these four models on RPD400 sample is presented in Appendix I for the robustness check.
C Model Implementation Details
All the API calls are made between February 15, 2025, and February 28, 2025. We ran inference on "Llama 3- BB-Chat" locally using Transformer (Wolf et al., 2020) library
*hhe Central Index Key (CIK) is unique identifier used by the SEC to identify filing companies. Here 8.00 corresponds to $100 million in 2021 value.
15

---

Published as a conference paper at COLM 2025
Sample Type
Sample Size
Full Sample (RPD)
197,011
RPD2000erYeur
8,600 RPD430
18,490
Table 4: Sample size for different samples used in our analysis.
NVIDIA RTX A40 GPU. For the "Llama-3" 1aammm---70--hat--- and DeepSeek-V3 inference, we use API from together.ai. We are grateful to them for providing free credits and making possible.
D Manual Verification
We manually check the correctness our prompting and regular expression in extracting revenue information from LLMs' outputs. We randomly sampled 100 GPT's answers in 2010. There are 85 numerical answers (answers containing numerical revenue) and 15 no-answers (answers containing no revenue). Our regular expression returns None for all no-answers and correctly retrieves all numerical revenue. For all 100 samples, we observe that the LLM does not have difficulty understanding the question.
E Robustness Error Threshold
We repeat both temporal and cross-sectional analysis by varying the error threshold of 5%, 10%, and 20%. The results in Figure and Table show that our findings are consistent.
Success Rate GPT-4o with Oifferent Error Threshold
2990
2008
2028
Figure 6: Performance of GPT-4o for three different error thresholds over time. The dotted line is drawn at the year 1995. The shadow area around the line is the standard deviation of model performance.
Error Threshold
Constant (a)
Beta (6)
5%
-11.9616ł
0.9558$
10%
-10.90461
0.92091
20%
-10.3530
0.9312
Table 5: Market cap analysis results correctness based the empirical regression for GPT-4o for three different threshold values of error. *, t, and $ indicate p-value at the 10%, 5%, and 1% levels, respectively for regression coefficients.
16

---

Published as a conference paper at COLM 2025
We calculate the raw enror of GPT-4o over time and plot it in Figure 12. The color bar indicates the percentage of companies for which GPT-4o outputs an answer containing numerical revenue information for each specific year. The black dots outside the box are the outliers, representing GPT-4o's hallucination. There are many more outliers in the year where GPT-4o's answer rate is high, which is consistent with our findings in hallucination analysis: 60 model is more likely to hallucinate for the year that it is also more likely to provide the correct answer. The overall trend of raw error also aligns with the error trend in our temporal analysis.
Temporal Analysis of Additional Models on Small Sample
Different Samplee)
Halluzination Rate For Different Models (Sma Sample)
GPT4455
Deestiekk V3
Gemiri
RnMX 78- full
Parecentage
50-
50
1980
2000
2020
1990
2000
2320
Years
Figure 7: Success and hallucination rates of GPT-4. 5, Gemini 1.5 Pro, DeepSeek -V3,, and FinMA-7B MMMMA-BB-flll full over time. The dotted line is drawn at the year 1995. The shadow area around the line is the standard deviation of model performance. The performance is measured for all four models on RPD200per sample.
In Figure 7, we compare the performance of GPT-4. 4.5 ("gpt-4. . 5-preview-2025-02 27"), Gemini 1.5 Pro 1eeemini-1.-ppm together-Al) and FinMA- 7B-full ("Tneeiinnl/iinm-----llllll finuulll-blllll from Xie et al. (2023)) over time. The performance is measured for all four models on RIPD200per sample. The findings are similar to the results of Llamas and GPT-40 for the full sample reported in Figure 4 on the full sample. GPT-4.5 performs very similar the GPT-4o not providing any significant gains for success, but performs better on hallucination for very recent years. We manually analyze the error of FinMA model, and find that almost all the time it provides either no text, number without unit. We exclude FinMA from any further analysis. For Gemini and DeepSeek, the trend is similar to the general trend.
DeepSeek-v3 ("DeepSeek-V33FF)
on
G Cross-Sectional Analysis of Additional Models on Small Sample
The results for GPT-4. Gemini 1.5 Pro, and DeepSeek-V2 on RPDz0perrear sample in Table are accordance with the results of the main paper.
H Hallucination Additional Results
The results for GPT4o Figure 8 are similar to the results for L1ama-3-700- 3-70B-Chat reported in 5.
htttts///weggigfff.e.c/c//ee//// full
17

---

Published as a conference paper at COLM 2025
DeepSeek V3
Gemini 1.5 Pro
GPT-4.55
X
B
B
Success Hallucination
-18.0719
1.5170t
-7.9137t
0.6646t 0.1159:
-7.95331
0.6688t
-8.68661
0.4469t
-1.2108
-0.2099+
0.0915
Table 6: Market cap analysis results success and hallucination based the empirical regression for GPT-4.5 5, Gemini 1.5 Pro, and DeepSeek-V3. ", +, and $ indicate p value at the 10%, 5%, and 1% levels, respectively for regression coefficients. The study is conducted on RPD200perYeur sample.
Hallucination Analysis
11.3
hallucinedie Count le noars!
10.5
10.3
20-
95
15
1.5
20 30
Figure 8: Each dot represents company with count on the x-axis indicating the number of years for which G2T-4e gave the correct answer and the y-axis indicating the number of years for which GPT-4o 4o hallucinated in the answer. We take an average of market cap to assign a color to the dot.
I Robustness Check: Same Companies Over Time
Although our experiments provide substantial evidence showing LLMs' proficiency in answering financial questions for more recent periods and larger market cap companies, is imperative to consider the potential confounding factors. The bankruptcy and establishment of companies throughout the years could introduce variability, thus potentially affecting the outcomes. Figure 9 indicates LLMs' temporal performance on 430 companies that have existed consistently over the 43 years in our sample, aligning consistently with our previous comprehensive companies analysis. Eliminating the bias from bankruptcy and IPOs, we can assert that LLMs exhibit enhanced capability in recent periods. The reason for enhanced performance compared to the full sample can be attributed to survival bias(Elton et al., 1996; Rohleder al., 2011).
Model
Constant (a)
Beta (6)
Llama-3-8B
-10.4140ł
0.8408
Llama-3-70B
-9.0624t
0.77861
GPT-4o-mini
-11.02631
0.9567t
GPT-4o
-8.4113t
0.7051
Table 7: Market cap analysis results on RPDa30 over time based on the empirical regression. t, and $ indicate p-value at the 10%, 5% and 1% levels, respectively for regression coefficients. The results are based on the full sample with year fixed effect.
In terms market capitalization, we ran the same regression analysis on those companies. The result, displayed in Table 7, is similar to the previous market cap analysis. It reaffirms
18

---

Published as a conference paper at COLM 2025
Success Rate For Different Models 15ame Companies)
GPI-Io-minii
75
GFT-46
Percentage %
32-
Figure Performance of GPTs and Llamas on RPD430- The dotted line drawn at the year 1995. The shadow area around the line is the standard deviation of model performance.
the claim that the larger the company's 's market cap is, the more accurate LLMs' answers to its financial questions are.
J Why "Revenue"?
Below we provide further reasoning why we picked "revenue" to form our question prompts.
Goal of our study: As the goal of our study is to understand the temporal and cross- sectional biases, we want to ask questions that can vary across time as well as firms. Given our goal, is not ideal ask questions like "Can you explain what derivative securities mean?". The answer to this question is the same across time and firms.
Why not some financial ratio?: Revenue is top-line number that is directly available in SEC filings. If we form question on financial ratio like return assets, even though we can validate the answer, it will involve two skills of model 1. ability to recall knowledge and 2. ability to calculate the ratio (do the arithmetic). As here are focused on understanding the knowledge gap, simple question on revenue is an appropriate choice.
Revenue Knowledge Bias Affects Stock Recommendation: We conduct study with -haiuht prompting to show that because of lack of knowledge of revenue for small companies, LLM (GPT-4o) is less likely to provide stock recommendation for them in Appendix K.
K News Prediction Analysis
To understand how the knowledge bias relate the model's inability to assess the impact of revenue forecast on the stock recommendation, we construct Chain-of-Thought step-by-step prompt as shown in Figure 10 and ask GPT-4o -"opt-40-2024-08- provide stock recommendation on whether to "BUY", "SELL" or "DNK" (do not have enough knowledge of the company). We filtered out the dataset according the knowledge cut-off time of GPT-407. We also require that those companies have revenue information available to us from 2018 to 2023. In total we end up with 2,751 companies in our dataset for this study.
to
We then evaluate whether the model's recommendation was BUY or SELL if the output label not DNK. We run the same empirical regression used earlier in the paper. In this case, the Y variable is assigned value based whether the recommendation is BUY, SELL, or DNK.
We ask the revenue forecast and stock recommendation after Sep 30, 2023.
19

---

Published as a conference paper at COLM 2025
Multi-turn Stock Recommendation Prompt
[SYSTEM INPUT]
Forget all your previous instructions.
[USER INPUT]
What are the revenues of (cempanyl for each finance year from [start_ [start__yer]] to Lend_yearpy Please return the revenue only.
Example: What are the revenues of APPLE INC. for each finance year from 2018 to 2022? Please return the revenue only.
[EXPECTED OUTPUT]
ONLY the revenue information for (companyl for finance year from |start_year] to
lend_yeart
Example: Here the revenues of Apple Inc. for each financial year from 2018 2022: 2018: $265.6 billion; 2019: $260.2 billion; 2020: $274.5 billion; 2021: $365.8 billion; 2022: $394.3
billion
[USER INPUT]
Based on the revenue information above, please predict the revenue of (companyl in finance
year (end___ year+])) Please return the revenue only.
Example: Based on the revenue information above, please predict the revenue of APPLE INC. in finance year 2023. Please return the revenue only.
[EXPECTED OUTPUT]
ONLY the revenue information for (company) for finance year [end_year+]], Example: $423.1 billion
[SYSTEM INPUT]
Act as a financial expert with experience in stock recommendations.
[USER INPUT]
Based on the information above, give either BUY, SELL, or DNK (do not have enough knowl- edge of the company) recommendation for (company] in finance year (end_ _end---
Example: Based on the information above, give either BUY, SELL, or DNK (do not have enough knowledge of the company) recommendation for APPLE INC. in finance year 2024.
[EXPECTED OUTPUT]
BUY, or SELL, or DNK Example: BUY
Figure 10: Example of multi-turn Chain-of-Thoughts (CoT) prompts for stock recommenda- tion under the impact of model's knowledge in revenue information.
Model
Constant (x)
Beta (6)
No Recommendation (DNK)
1.4804
-0.0654
BUY Recommendation
-0.5766
0.0751$
SELL Recommendation
0.1006
-0.0103
Table 8: Market cap analysis results on a stock recommendation based on the empirical regression for GPT-4o. f, and $ indicate p-value at the 10%, 5%, and 1% levels, respectively for regression coefficients. The study is conducted on a list of companies we have in the years 2018-2023.
The result in Table 8 indicates that the model is less likely to make a decision for smaller market-cap companies while more likely to provide the label "BUY" for large market-cap companies. The result serves tiny experiment to show how the knowledge bias can impact biases investment recommendations, We note that this is just a study on only one year of data due to reported cut-off.
20

---

Published as a conference paper at COLM 2025
L Spanish Soccer Leagues Statistics Study
In order to demonstrate the applicability of our LLM bias evaluation framework to non- finance domains, we examined the application of our framework to assess LLMs' abilities to answer questions about soccer statistics. We compiled dataset of 1,676 samples containing season statistics for given season and team in both the La Liga and Segunda División Spanish soccer leagues. Our dataset spans 41 seasons from 1980 to 2020 and contains statistics about 167 unique teams, though not every team is present every year, as teams are relegated down to promoted from lower leagues based on their performance.
We then assess the ability Llama3-70B (temperature=O) max_ tok_eeooss=)00) to answer the question: "In the [year) [La Liga/Segunda División) season, how many points did (team) finish with?". We assess the quality of responses across both the temporal and cross-sectional dimensions, looking at differences in model performance in different years, well as differences in model performance for clubs with greater prominence (measured by finishing position). Model performance is measured using success rate metric, which for this study is defined as the percentage of model responses which give a point value within 5% of the true point value.
Success Rate For Llama3
Success Rate
1980-81
Figure 11: Success Rate of Point Answering Over Time.
Looking at the performance of the model, temporally (see Figure 11), model performs better in more recent years, suggesting a bias towards more recent years.
that the
Furthermore, assess how the prominence of a soccer club affects the model's performance by running logistic regression on Pos;,, the finishing position of each soccer club. Our logistic regression is defined in Equation 3.
logit(P(Y,, = y)) + f* Posú, + + D2 D 1 (league=Segunda) Ei11
Here Yú, is the outcome variable where y = 2 indicating model success, ó; is a year-fixed effect, constant term, y is league-fixed effect, and E₁ is error term. We find to be 3.1137, reflecting the fact that being in Segunda División decreases the probability of getting the correct answer compared to La Liga.
We find 11.2941 and B -0.542 suggesting that clubs with greater finishing position (lower-prominen clubs) see a lower success rate in our experiment. This highlights knowledge bias stratified along team prominence, where the LLM is more likely to have knowledge of statistics regarding more prominent and successful teams.
Significance at the 1% level
21

---

Published as a conference paper at COLM 2025
GPT-4pts Success
LEDEOCOR 2OAE
Raw Error by Year
Eror Cerronae Row
10
Figure 12: Raw answer error of GPT-4o ("8pt-40-2024-08-00) over time.
22


Reviews:
[{"review_id": "1hhk5sOzpP", "reviewer": "Reviewer_YwGF", "summary": "The paper is exploring an interesting direction for the use of LLMs in the market/stock data understanding in terms of understanding the depth (time wise) of the financial data embedded into the training of the LLM. The authors probe state-of-the-art models (GPT-4o,4.5, Gemini 1.5 Pro, DeepSeek-V3, Llama-3 small and large) with 197k questions related to the financial data from various benchmark datasets (all in a zero shot setting). The authors explore the knowledge gap in the models (in terms of timing, old vs new), biases in terms of favoring large companies over smaller ones, and any hallucinations that may occur given the research is into numbers.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors explore a challenging research domain for LLM given the factual information given in the task with the financial datasets, they also study the impact of the LLM hallucinations in the predictions.   \n2. In terms of evaluation, the work is evaluated across a large number of datasets covering 197k instances but across a diverse range of financial related dataset sources.   \n3. The authors have utilized an evaluation setup (Figure 2\\) based on zero short prompting, that not only evaluates/probes the LLM to be factual but also to be accurate in terms of time.   \n4. The authors have also sub-sectioned their findings section to make it an easy parse for the readers.   \n5. The supplemental data section is also rich in terms of including all the information needed to reproduce their experiments with the prompt templates and additional results in terms of empirical results into their findings.", "reasons_to_reject": "1. A strength of the paper is also a weakness given the limited scope of narrowing down the market data to only US related data since some of the companies are international and financial data is impacted by international events.   \n2. The authors have explored only zero-shot prompting, this is the challenge for LLMs but it would have been helpful to see how it does change if there was one-shot prompting.\n3. Seems like not all models are evaluated with all the 197k instances and the large closed sourced models are limited to a sub-sampled dataset due to cost constraints. The highlight of the paper should have been to this smaller dataset in such case.", "questions_to_authors": "1. What is the reason for the drip in 2021-2022 (line \\#200), is it due to COVID-19? But even then, COVID-19 was peaked in 2020\\.", "extra_scores": {}}, {"review_id": "2f26ALTeHG", "reviewer": "Reviewer_hcdm", "summary": "The paper makes a valuable contribution by evaluating the financial knowledge of prominent Large Language Models (LLMs) and focusing on two kinds of bias: temporal (knowledge gaps variation across time) and cross-sectional (variation across firms sizes). Using a new dataset of over 197,000 revenue-related prompts covering 17,000+ U.S. public companies from 1980 to 2022, the authors benchmark several frontier LLMs, including GPT-4, Gemini, DeepSeek-V3, and Llama-3. The findings reveal systematic weaknesses in LLMs: they perform better on larger companies and more recent data, but paradoxically also hallucinate more on those same entities. This is one of the few papers I am aware of evaluating factuality and correctness of LLMs in narrow domains.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel dataset: \n  - the paper introduces a semi automatically generated Revenue Prompt Dataset, spanning multiple decades and company profiles, \n    enabling a high quality evaluation.\n  - the inclusion of auxiliary firm-level features (market capitalization, retail/institutional attention, readability) enhances the analysis.\n- Rigorous evaluation:\n  - the ternary outcome classification (correct, hallucination, no answer) is interesting and quantitatively robust.\n  - logistic regression analysis helps isolate key influencing factors on LLM behavior.\n- Clear presentation and high quality argument:\n  - demonstration of a retrograde knowledge bias adds a new dimension to the discussion of LLM limitations.\n  - the identification of a paradoxical phenomenon - higher accuracy and hallucination likelihood in high-market-cap companies - adds to \n    the discussion of important questions about model confidence and reliability.\n- Potential broader relevance:\n  - the evaluation framework is applied outside finance illustrating potential for broader domain adaptation.", "reasons_to_reject": "- Limited evaluation for full dataset:\n  - high-performing models like GPT-4.5, Gemini, etc. are only evaluated on a subset of the data due to API cost constraints. This leads me \n    to wonder about the generalizability of those results, albeit I of course understand the reasoning.\n- Narrow focus:\n  - the analysis is limited only to revenue-based questions. This does not capture how LLMs perform on more complex financial reasoning \n    tasks (i.e. interpreting cash flows, earnings reports, etc.) I would like to have seen one or two other, qualitatively different questions to \n    make the analysis more comprehensive.\n- Lack of causal explanation:\n  - while the retrograde bias is well-documented, causal analysis (ablation studies, retraining experiments) on why these biases exist is \n    absent. I am guessing part of the reason for this behavior is the change in volume of training data over time (the amount of machine \n    readable text has increased drastically since the 90s, not to mention the 80s). I don't have an explanation for the paradoxical behavior \n    where rate of hallucinations goes up with time. Some hypotheses on this would be good to see.", "questions_to_authors": "See the Reasons to Reject section.", "extra_scores": {}}, {"review_id": "NNJGbyeW7w", "reviewer": "Reviewer_2yqz", "summary": "This paper investigates temporal and cross-sectional knowledge biases of large language models (LLMs) in the financial domain. Authors construct a test set of 197K company-year revenue queries to evaluate several LLMs such as GPT-4o, Llama , DeepSeek-V3. The results show that LLMs perform  better on more recent years and on larger, more prominent companies, while hallucinations also increase for such companies. The paper also studies cross-sectional factors like market cap and show they they can predict both success and hallucination rate.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Construction of a large-scale dataset (197K QA pairs) enabling  analysis of LLM financial knowledge over four decades.\n- A systematic study of “retrograde” knowledge gaps and cross-sectional biases in finance.\n- Providing evidence that LLMs favor recent data and large companies.\n- Providing proof that the same reasons that drive success rate, also increases hallucination", "reasons_to_reject": "The study still remains very shallow to be published in an LLM conference, and there is a lack of connecting the results to the theory of LLM and possibly to the architectural choices of LLMs studied.\n\nStudied LLMs are still generalist one, while there exists many other more finance-specialized one which should have been taken into account in the study.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jOlGy4ta2", "reviewer": "Reviewer_xcLv", "summary": "In this paper, the authors investigate factors influencing the financial knowledge recall of large language models (LLMs). They find that LLMs exhibit more accurate financial knowledge regarding recent events, with a noticeable \"cut-off\" period before which factual recall significantly deteriorates. Additionally, the authors show that LLMs' factual recall tends to be biased toward companies with larger market capitalization. The paper clearly outlines the analytical methods and dataset construction procedures employed in their study.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clearly written and includes sufficient methodological details to allow other researchers to replicate or build upon the authors' work.\n\n2. The in-depth analysis of this focused topic may be particularly valuable and of interest to researchers working in financial applications of large language models", "reasons_to_reject": "1. The paper's contribution is incremental in nature. It lacks a novel approach or an innovative analytical method, and does not provide a new empirical application of natural language processing techniques to finance.\n\n2. The method employed by the authors to verify the factual recall ability of the LLM involves prompting the model to recall company revenues for specific years. Although raw LLMs are frequently treated as knowledge bases in various scenarios, empirically, precise questions of this granularity typically require integration with external tools, such as retrieval-augmented generation or search engines, to achieve satisfactory accuracy. Analyzing raw model behavior alone may therefore offer limited insights into assessing or enhancing real-world system performance. Furthermore, focusing solely on revenue recall is only a narrow reflection of an LLM's overall financial knowledge, making it insufficient for drawing conclusive insights about the model’s broader financial knowledge capabilities.\n\n3. The conclusions presented by the paper may lack depth. Given that an LLM’s knowledge originates from training data, it is expected that the model will exhibit a knowledge cut-off based on the availability of machine-readable data. Additionally, it is unsurprising that companies with larger market capitalization, receiving more extensive media coverage, would naturally have better representation in the LLM's knowledge base.\n\n4. The target audience for this topic appears highly specialized within finance. While the analysis could certainly interest researchers in financial applications, it is unclear whether the topic aligns sufficiently with the broader interests of the general audience at COLM.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a large-scale empirical study evaluating the financial knowledge of several state-of-the-art Large Language Models (LLMs). The authors have constructed a dataset of over 197,000 revenue-related questions spanning four decades and over 17,000 U.S. public companies. The core contribution lies in systematically probing for and identifying two key biases: a "retrograde" temporal bias, where models perform better on more recent data, and a cross-sectional bias, where models are more accurate for larger, more prominent companies.

Consolidated Strengths
* Novel and Valuable Dataset: All reviewers acknowledged the significant effort and value in creating the 197k-instance Revenue Prompt Dataset. This is seen as a durable contribution to the community, enabling future research on LLM factuality in specialized domains.
* Rigorous and Large-Scale Evaluation: The scale of the experiment is a clear strength. The systematic approach, clear evaluation setup (including the ternary outcome classification of correct/hallucination/no answer), and subsequent regression analysis were praised for their robustness.

Consolidated Weaknesses
* Limited Scope: A key criticism centered on the study's scope. The focus is exclusively on U.S. companies, revenue-based questions, and a zero-shot prompting strategy. 
* Generalizability and Model Choices: The fact that the most advanced closed-source models were only tested on a subset of the data due to cost constraints was a recurring concern. Additionally, Reviewer 2yqz pointed out that the study could have been strengthened by including finance-specialized LLMs, not just generalist ones.

Conclusion
This paper makes a valuable empirical contribution to a critical and underexplored area: the factuality of LLMs in the financial domain. The creation of a large-scale, focused dataset is a good contribution itself. The findings on temporal and size-based biases are clear, well-supported, and add an important, nuanced discovery about the nature of hallucinations in the financial area. The majority of reviewers rightfully identify it as an accept.
Decision: Accept