FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
Authors: LIANG HU
Jianpeng Jiao
Jiashuo Liu
Dongyuan Mutu
Yanle Ren
Zhoufutu Wen
Kaiyuan Zhang
Xuanliang Zhang
Xiang Gao
Tianci He
FEI HU
Yali Liao
Zaiyuan Wang
Jingkai Liu
Sun Daibin
Ziqing Zeng
Zhiyuan Zeng
Chenghao Yang
Qianyu Yang
Mingren Yin
Ge Zhang
Xinyi zhang
Xiying ZHAO
Zhu Zhenwei
Hongseok Namkoong
Wenhao Huang
Venue: ICLR 2026 Poster (2026)
Keywords: Agent Benchmark
Financial Search
Financial Reasoning
Primary area: datasets and benchmarks

Abstract: Search has emerged as core infrastructure for LLM-based agents and is widely viewed as critical on the path toward more general intelligence. Finance is a particularly demanding proving ground: analysts routinely conduct complex, multi-step searches over time-sensitive, domain-specific data, making it ideal for assessing both search proficiency and knowledge-grounded reasoning. Yet no existing open financial datasets evaluate data searching capability of end-to-end agents, largely because constructing realistic, complicated tasks requires deep financial expertise and time-sensitive data is hard to evaluate. We present FinSearchComp, the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. FinSearchComp comprises three tasks, Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation, closely reproducing real-world financial analyst workflows. To ensure difficulty and reliability, we engage $70$ professional financial experts for annotation and implement a rigorous multi-stage quality-assurance pipeline. The benchmark includes $635$ questions spanning global and Greater China markets, and we evaluate $21$ models (products) on it. Grok 4 (web) tops the global subset, approaching expert-level accuracy. DouBao (web) leads on the Greater China subset. Experimental analyses show that equipping agents with web search and financial plugins substantially improves results on FinSearchComp, and the country origin of models and tools impact performance significantly. By aligning with realistic analyst tasks and providing end-to-end evaluation, FinSearchComp offers a professional, high-difficulty testbed for complex financial search and reasoning.

Published as a conference paper at ICLR 2026
FINSEARCHCOMP: TOWARDS A REALISTIC, EXPERT- LEVEL EVALUATION OF FINANCIAL SEARCH AND REASONING
Liang Hu!; Jianpeng Jiao! Jiashuo Liu', Dongyuan Mutu', Yanle Ren', Zhoufutu Wen!", Kaiyuan Zhang's Xuanliang Zhang ByteDance Seed
[huliang. will, Iiminnit@@byteddaceecoo
Xiang Gao Tianci He* Fei Hu', Yali Liao", Zaiyuan Wang*, Jingkai Liu", Daibin Sun', Ziqing Zeng', Zhiyuan Zeng', Chenghao Yang', Qianyu Yang', Mingren Yin', Xinying Feng', Ge Zhang', Xinyi Zhang', Xiying Zhao?, Zhenwei Zhu
ByteDDnce Seed
Hongseok Namkoong
Wenhao Huang
Columbia Business School
ByteDance Seed
ABSTRACT
Search has emerged as core infrastructure for LLM-based agents and is widely viewed critical the path toward more general intelligence. Finance is a partic- ularly demanding proving ground: analysts routinely conduct complex, multi-step searches over time-sensitive, domain-specific data, making it ideal for assessing both search proficiency and showledge-grounden reasoning, Yet no existing open financial datasets evaluate data searching capability of end-to-end agents, largely because constructing realistic, complicated tasks requires deep financial exper- tise and sensisevi data is hard to evaluate. We present FINSEARCHCOMP. the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. FINSEARCHCOMP comprises three tasks, Tine-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation, closely reproducing real-world financial analyst workflows. To ensure difficulty and reliability, engage 70 professional financial experts for annotation and implement rigorous multi-stage quality-assuranee pipeline. The benchmark in- cludes 635 questions spanning global and Greater China markets, and evaluate 21 models (products) on it. Grok 4 (web) tops the global subset, approaching expert-level accuracy. DouBao (web) leads on the Greater China subset. Exper- imental analyses show that equipping agents with web search and financial plu- gins substantially improves results on FINSEARCHCOMP, and the country origin of models and tools impact performance significantly. By aligning with realis- tie analyst tasks and providing end-to-end evaluation, FINSEARCHCOMP offers a professional, high-difficulty testbed for complex financial search and reasoning. We provide the data of FINSEARCHCOMP in Github and Huggingface
INTRODUCTION
Search is a core capability for agentic LLMs. Without accurate and timely retrieval, reasoning and decisions are limited. However, the community still lacks realistic benchmarks assess search in domain- domain-groundee grounded and decision-relevant settings. General- browsing benchmarks, such as
"Corresponding authors. 1 Core contributors, a-3 order. Contributors, a-3 order.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of FINSEARCHCOMP with existing financial benchmarks.
Open-domain
Tool
Time-sensitive
End-to-end Agent Evaluation
Holistic Evaluation
Benchmark
Search
Use
Data
FinQA (Chen et al. 2021
X
X
ConvFinQA Chen el al. 2022
X
MultiFinBen [Png ct al 2025
X
X
FinanceQA (Mateega of a.. 2025
X
X
X
X
BizFinBench (Lu
X
X
X
X
FinEval (Guo
2025
X
X
x
CPA-QKA (Kuang et al. 2025)
X
X
Finance Agent (Bigeard et al. 2025
X
X
FINSEARCHCOMP (Ours)
/
BrowseComp Wei et al., 2025). mainly test if agents can complete multi-step navigation to find short, verifiable facts. They avoid long-form synthesis, ambiguity resolution, and do not assess use of domain knowledge or multi-source evidence, limiting for domain-intensive support. In finance, search central because analysts spend much time searching, validating, and synthesizing informa- tion across the web and databases, where timeliness and precision have direct value (Nie al.) 2024] Ding et al. 2024). To address this gap, we study open-domain financial data search, where agents retrieve from web and professional databases to answer queries, requiring tool usc, time-sensitive
data, and integration of heterogeneous evidence (Egg et al. 2025 Ferrag et al. 2025 Peng et al, 2025).
However, existing financial DnuestionAAsweele (QA) benchmarks usually pre-collect relevant con- text and bypass open-domain search and tool use, thereby under-assessing agents' search compe- tence and diverging from analyst workflows Wang et al. (2023); Yang et al.] (2024) (see Table IT. Among them, the Finance Agent Benchmark (Bigeard et al./ (225)) offers an end-to-end evaluation but is confined to a self-constructed system, which is a base model augmented with a retrieval mod- ule. Moreover, the benchmark's use of only historical data permits success via memorization, not necessarily real-time information retrieval.
To address this need, we introduce FINSEARCHCOMP. the first open-domain benchmark for real- istic, analyst-style financial search, comprising 635 questions that require time-sensitive acquisition and multi-source evidence integration. To mirror the day-to-day needs of professional analysts, we design three task
Sensi-veeii Data Fetching, Simple Historical Lookup. and Complex Historical Investigation (Table 2}. These tasks correspond core analyst workflows. Sene-Sensitive Data Fetching tasks capture situations where the correct response changes rapidly (e.g.. stock prices, exchange rates, and gold prices), emphasizing rapid retrieval and verification under tight time con- straints. Simple Historical Lookup tasks reflect frequent point-in-time lookups. These questions require going back to the correct historical disclosure and aligning with the company's reporting calendar. Complex Historical Investigation tasks involve building multi-period views that integrate different sources. Such queries demand stitching together multiple reports, checking consistency across sources, and ensuring that values are comparable across time.
During construction, we engaged 70 professional financial experts for data annotation, conducted multi-stage verification prompts and reference answers, and specified detailed, rubric-based scor- ing guidelines. To account for differences in data sources and reporting conventions. FINSEARCH- CомP covers two subsets: Global and Greater China. Further, since answers of different LLMs may have different formats and exhibit legitimate minor fluctuations (e.g., revisions or rounding across various sources), we adopt an L.L -.L.M evaluation with rubric-guided judging and explicit tolerance bands, complemented by expert spot checks to ensure the overall correctness.
We evaluate models (products) on FINSEARCHICOMP spanning web-enabled products and API endpoints. On the Global subset Grok 4 (web) attains the highest overall score (68.9)) outperform- ing the runner-up, experts by 6.1 pp. On the Greater China subset, Chinese models perform markedly better: DouBao (web) leads the leaderboard, followed closely by YuanBao- -eeeBe-eeeeeeek-- - (web) and Grok 4 (web); nevertheless, all remain more than 34 pp below human performance.
-rinkkig (web), by 5.0 percentage points (pp), yet still trailing human
2

---

Published as a conference paper at ICLR 2026
Further analyses indicate that equipping agents with web search capabilities and financial plugins improves their performance on FINSEARCHCOMP. Our analysis further identifies recurring fail- modes: shallow search, stale or mis mis-timestamped evidence, cross-umit/uurrency aggregation, and report-calendar misalignment, offering concrete targets for future improvement. For example, common failures include neglecting to call specialized data plugins in favor of less reliable web searches, extracting incorrect data from a valid source (e.g., confusing opening vs. closing prices), and unnecessarily over-complicating simple queries like "market cap" into multiple steps.
Our contributions are as follows:
We introduce FINSEARCHCOMP, the first fully open-source, end-to-end agent benchmark for open-domain financial data search. FINSEARCHCOMP comprises 635 expert-curated queries spanning global and Greater China markets and three analyst-style task families (Time-Sensitive, Simple Historical, Complex Historical), with multi-stage quality control.
2. We release a carefully curated benchmark dataset with deterministic gold answers and an fully open-source evaluation harness.
3. We conduct comprehensive study of 21 models (web-enabled products and APIs), show- ing that equipping agents with web search and financial plugins consistently improves per- formance. A case study further reveals that poor performance often stems from insufficient search depth and the retrieval of outdated information.
2 FINSEARCHCOMP
We begin by outlining our design principles and choices. We then describe FINSEARCHCCOMP 's construction, quality-control measures, and some descriptive statisties.
2.1 DESIGN PRINCIPLES
Before introducing FINSEARCHCOMP. we outline the desiderata for a high-quality financial search benchmark and describe how our design addresses them.
1. Task Professionalism & Diversity. Financial data retrieval spans tasks of varying complexity, from real-time queries to complex historical analyses. The nuances of financial metrics necessi- tate domain expert oversight ensure professional accuracy and relevance.
Our design: In consultation with financial analysts, designed three tasks mirroring their daily workflows: sensitive data fetching, simple historical lookup, and complex historical investigation, detailed in (Secciion 2.2]
2. High Quality of Questions. Ambiguity arising from varied data sources and definitions can undermine evaluation. Unambiguous questions with clear provenance and reproducible grading criteria are thus crucial for reliability and fairness.
We implement rigorous quality control process, including reliable data source selection, ambiguity mitigation through precise definitions, and multi-expert answer verifica- tion (Section 2.4).
Our design:
3. Broad Market Coverage. To ensure external validity and assess real-world readiness, a bench- mark must cover diverse markets, languages, and reporting conventions, revealing generalization failures that single-market tests often miss.
* Our design:
FINSEARCHCOMP includes Global (Western) and Greater China subsets, with questions in English and Chinese. To enable fair comparison, mirror task templates and balance entity coverage, supporting lingual evaluation (see [Section 2.5)
2.2 TASK DESIGN OF FINSEARCHCOMP
We define three task types aligned with daily analyst workflows. Each requires at least one external tool call, and each question has single, fully objective answer. These tasks test core search and reasoning skills that are not only central to financial analysis but also broadly important knowledge work in many disciplines. Detailed comparison is shown in Table 2
3

---

Published as a conference paper at ICLR 2026
Table 2: Examples of the three tasks in FINSEARCHCOMP, with retrieval depth, temporal span, reasoning complexity, and typical data types.
Retrieval Depth
Temporal Span
Reasoning Complexity
Typical Data Types /
Task
Exumple
Examples
TI. Time
Stock prices, rates, gold prices
IBM latest close price. Obtained from a real-time query of IBM
Sensitive Data
day
Easy
Fetching
(real-time quotes)
YoY (Year-over-
What was the total assets of Starbucks as of September 27, 2020? (Answer: $29374.5 million, rounding emors al- lowed.)
Year). HoH (Half-on-
Simple Historical Lookup
Half) TTM (Trailing
day
Medium
Twelve Months),
FY (Fiscul Year),
quarterly reports
From 2010 to Apr 2025, which month did the S&P 500 in- dex experience the largest single- month increase? (Answer: Apr 2020 (12 68%), error =0.1% allowed.)
Multi-period views, currency/umit malization, corporate action adjustments, data provenance
T3. Complex Historical Investigation
>1
Hard
months
T1 Time-Sensitive Data Fetching. Requests intraday/daily data (e.g.. close price, filings, guidance), supporting trading, monitoring, and event reactions. Challenges: freshness, calendar alignment, ticker aliasing, conflict resolution. Also relevant to journalism, policy monitoring, clinical trials.
T2 Simple Historical Lookup. Requests lixed-point facts (e.g., FY2024 R&D expense TTM rev-
enue") Enables baselining, Yo Yo HoH comparisons, event studies, backtests. Challenges: reporting conventions (FY, TTM, quarterly), restatements, unit/currency consistency. Similar needs in policy, medicine, and academia.
T3 Complex Historical Investigation. Requests multi-period synthesis (e.g., largest single-month index gain over decades), supporting trend analysis, factor research, valuation comps, risk mon- itoring. Challenges: long horizons, corporate actions, unit normalization, multi-step reasoning. Also applies to climate science, history, and epidemiology.
2.3 CONSTRUCTION OF FINSEARCHCOMP
To accommodate the unique characteristics of different tasks, we employ a variety data con- struction strategies to ensure both diversity and quality, shown in Figure (left side). Example questions of each task can be seen in SScciion B.4 And we show time cost of financial analysts on three tasks in Section B.3
Time-Sensitive Data Fetching. Financial experts first mannually design questions asking for time-sensitive data that be verified through API. Time-sensitive data changes time, actually prepare the code for API calls that obtain real-time data points for each question. Finally, financial experts check each API result against the real-time data ensure the retrieved result is correct and on-time, for accurate evaluation. We establish permissible error margins for evaluation based on the specific volatility characteristics of different assets (e.g., equities, forex).
Simple Historical Lookup. We collect data for this task from two sources to enrich the diversity. (i) Financial experts select documents from official sources (e.g., listed company filings, regulatory authority websites, statistics bureau) and extract key data points to formulate questions and their corresponding answers. (ii) We also design questions using historical data with consistent defini- tions from professional financial databases. To mitigate the risk of data revisions for certain official statistics (e.g., macroeconomic indicators), we address potential ambiguity by setting a reasonable answer range by explicitly specifying the reference time point in the question.
Complex Historical Investigation. The construction process involves two primary methods. (i) Financial experts design questions based on their real-world professional scenarios and get answers
Researrh and development spending in financial statements. FY2024 means fiscal year 2024. "Trailing Twelve Months revenue, over most recent 12 months.
'Year-over- Year, compares with same period prior year. HHalf-over-Hall compares with prior half-year.
4

---

Published as a conference paper at ICLR 2026
Data Collection and Processing
Quality Contrul
Preffiient Finowial E Exparts
77 Sendtie Puita Frisking
S0 Experts
Senior Experts
Scelllie Qutilies
Public Fiansral APIS
Calle Dyromac Anster)
Provide Sorple A
RySatble iutt Souree Selectiou
Simple Mistorical Lookey
Campay Fllingo
Cress validation
Frencid Dutskses
Doige Sieple
Compaey
Asthone
Axthoriy Websitss
Massaly Verifee Abouens
Mittiaating Amaiguity
Finazeid Datikac
Saat deciziionnl
Sci BEAECT Ea rsSace cridefae kolenane
Andid metrer proce atjucnures
Complay)ittriall Investiaatinn
sknakith is quciiAe
MurissEEuuatt Asunser Vanifccttinn
Comppayy Filiag
èB
Qudivon (MattiSSarrss Resaning Cilcaltion)
Arguss
ArmurA
Marsaly Vedifid ASSHCIS
Eiver di
Amarral
Diticone
Firancid Daxieses
Camper
Sigert
Figure 1: The overview of the construction process. The construction of this benchmark involves three tasks. The data for each task originate from different sources and undergo separate processing pipelines. A uniform quality control procedure applied across all tasks.
using reliable financial data sources. (ii) Financial experts download tables from a verified and reliable financial database and annotate questions based on the tables. Furthermore, financial ex- perts screen and refine these candidates, finalizing 2 to 5 high-quality questions on each table and recording their definitive answers.
2.4 QUALITY CONTROL
To ensure quality, we implemented a rigorous quality-control process throughout construction and summarize the key measures below.
Our annotation process is managed by financial experts, organized into a 50-member annotation panel and a 20-member senior review panel for arbitration. All experts hold advanced finance degrees and possess active industry experience from leading institutions such Citadel, J.P. Morgan, and CITIC Securities. Panelists are selected through a rigorous domain- specific assessment to ensure high annotation quality and inter-annotator reliability.
Proficient Financial Experts.
Reliable Data Source Selection. To ensure data authority and accuracy, answers sourced from highly reliable channels, including official company filings, government websites, and professional financial databases. We employ multi-source pross-validation method verify data and eliminate ambiguity. This rigorous process allows us to identify and avoid inconsistencies or calculation errors, even those present in widely recognized professional databases.
Mitigating Ambiguity. We implement several strategies to mitigate ambiguity. First, we avoid questions involving metrics with inconsistent calculation standards across institutions, such as forward-adjusted stock prices (see [Section B.2). Second, we ensure question clarity by explicitly stating definitional standards (e.g., Static PE vs. PE TTM), setting answers as numerical ranges to accommodate minor calculation discrepancies, and avoiding metrics prone retrospective data revisions where possible. A detailed guide is summarized in Table
Multi- Multi-Expert Answer Verification. Our verification protocol utilizes a blind review mechanism to ensure accuracy. After an expert drafts a question and answer, one or two other experts solve the question independently without access to the solution. A senior expert arbitrates any discrepancies or ambiguities. leading to the modification or rejection of the question. The entire dataset construction involved approximately 240 expert-hours, ensuring comprehensive professional oversight.
5

---

Published as a conference paper at ICLR 2026
2.5 DATA OF FINSEARCHCOMP
Most evaluation metrics in FINSEARCHCOMP rely on problem-specific, expert-annotated ranges of acceptable values, rather than single ground truth. Furthermore, we illustrate the distribution of topics in Figure 7 which covers 10 distinct topics, demonstrating the diversity of FINSEARCHCOMP.
We summarize the basic statistics in Figure 6
2.6 EVALUATION OF FINSEARCHCOMP
Considering the dynamic nature of answers and the need for numerical tol- erance in FINSEARCHCOMP, we adopt LLM-as-a-Jute (Zheng et al.) (023) for evaluation. The specific evaluation methods for the three tasks are detailed below, with the corresponding prompts provided in the Section
Evaluation Protocol.
For Time-Sensitive Data Fetching, we address several challenges: (i) time lags between the model response and the evaluation, (ii) potential data latency from some financial APIs, and (ii)) the inabil- ity of most APIs to query prices specific second. To mitigate these time-sensitivity issues, we initiate the evaluation process uniformly after the relevant markets have closed. To ensure fairness and accuracy, we establish differentiated evaluation rubries based the characteristics of various asset classes, with details in |Section C.T For Simple Historical Lookup and Complex Historical Investigation, which feature static and deterministic answers, we also annotate rubrics, such predefined error margin.
the 0-1 error metric. Ground truth answers are static, except for the Time-Sensitive Data Fetching task, which requires real-time API execution. We employ an LLM judge to assess each candidate answer A against predefined rubrics R. This evaluation is modeled by a judgment function, [[A, R). which returns for a correct answer and 0 otherwise. The final score S is thus defined as:
Evaluation Metrics. In FINSEARCHCOMP.
SAA, R) [A,
Evaluation Accuracy. To validate the reliability of LLMM-as-a-Judge, we conduct human evalua- tion representative subset. For each bench- mark dataset, we selected 5 models and verified their complete evaluation sets, resulting in roughly 400 instances per dataset. On this sample, the judg- ments of LLM-as-a-Judge reached 95% agreement with human-verified labels, confirming the robust- ness of evaluation protocol. Detailed results are summarized in Table 3
Table 3: Accuracy LLM evaluation com-
pared with human evaluation.
Task Subset
Accuracy (%)
T1
Global
91.5
Greater China
91.7
T2
Global
96.8
Greater China
95.5
T3
Global
97.4
Greater China
99.8
3 EXPERIMENTS
In this section, we present the main results for FINSEARCHCOMP. We evaluate the performance of 21 mainstream models (products) on FINSEARCHCOMP from August 1st to 20th and to ensure temporal comparability across models, all TI evaluations were conducted after the official market close (local market time) on each evaluation day. These models are categorized into two groups:
Web-based products (12 models): Grok 4 (web) [AAl 2025). GPT-5-Thinking (web) (abbr.
GPT-5-T (web)), Gemini-2.5-pro (web) (Google 2025) (abbr. Gemini (web)), Qwen3-235B.
A22B-2507 (web) (abbr. Qwen3 (web)) (Qwen Team, (025). DeepSeek-RI (web) (DeepSeek-A)
et al.), DouBao (web) [ByteDance 2025). DouBao- -ouuBao-Thinknn (web) (abbr. DouBao-T (web)),
YuanBao-Haun - Tri-Thiniing (web) (abbr. Hun Yuan-Ti (web)) (Tencent 2025). YuanBao-
DeepSeek- V3 (web) (abbr. YuanBao V3 (web)),
SeenBaa-DeepSeek-... (web) (abbr. YuanBao-
RI (web)), Emiee-11 (web) (Baidu 2025). and Kimi k2 (web) (MoonShot AT 2025).
APIs (9 models): Gemini2.5 (API) (abbr. Gemini (APD), Gemini-2.5-pa (Google Search) (API) (abbr. Gemini-G (APD), DouBao (APD), DouBao-Thinking (API) (abbr. DouBao-T (APD), Dwen3-235B-A222B -5077 (API) (abbr. Qwen3 (APD), DeepSeek-Ri (API), Hun Yuan-TI (API),
Emie-X1 (API), and Kimi k2 (API).

---

Published as a conference paper at ICLR 2026
To establish a human performance baseline, engage another 50 financial experts who are not involved the data construction. These experts perform the benchmark tasks while utilizing search tools. Their average score is taken the human bascline.
3.1 MAIN RESULTS
We report the overall performance of the strongest model from each provider in Figure 8 with the whole performance in [Section C The models form a clear performance hierarchy, with sizable gap human experts remaining. Rankings differ between the global and Greater China subsets, likely reflecting differences in training-corpus coverage, language/domain alignment, and retrieval infrastructure. On the global subset, Grok-4 (web) and GPI:5-Thinkkigg form clear leading tier, with Grok-4 (web) securing the top score and approaching expert-level accuracy. While Gemini-2.5- pro (web) decline when moving from the global the Greater China subset, Grok (web) remains competitive. On the Greater China subset, DouBao (web) and YuanBao- --uan-T--TTiien -Thhinnnggg (web) strong the Greater China subset, though they still trail human experts by substantial margin. Detailed results and analyses are shown in Section 3.2) and Section 4
3.2 RESULTS ACROSS DIFFERENT TASKS
We show the results of each tasks respectively in Figure 2
Our main findings are as follows.
Finding 1. Task difficulty increases from T1 to T3. We observe monotonic performance decline from T1 to T3 across all models, confirming the escalating difficulty of our tasks. The complexity of '3 stems from its requirements for multi-hop retrieval, temporal reasoning (e.g., event dating, fiscal-calendar alignment), and fine-grained entity resolution.
Finding Grok-4 and GPT-S-Thinking approach expert-level reasoning. While most mod- els trail human performance, Grok-4 and GPT-S-Thinking achieve near-expert results on the global subset. Their performance advantage grows substantially with task difficulty, peaking on T3. This suggests their superiority stems from advanced capabilities in multi-step reasoning, temporal align- ment, and entity disambiguation, rather than retrieval alone. Grok-4's 's top score on T3 for the Greater China subset further corroborates its advanced reasoning capabilities.
4
CASE STUDY
In this section, we conduct case studies to analyze the performances in detail. We also provide discussion on the effect of reasoning capabilities in Section
4.1 How MUCII DO SEARCII CAPABILITIES IMPACT PERFORMANCE ON FINSEARCHCCMPP?
As shown in Figure 3 models without search uniformly score 0 on T1, as they cannot retrieve current financial data. Without search, they still obtain non-zero but low scores on T2 and T3; we attribute this to parametric memory from pre-training (e.g., annual reports and statistical-ageney releases), which surfaces approximate facts that are often outdated or misaligned, yielding higher error rates. With search enabled, average gains of 40.8, 29.0, and 8.1 points are observed on T1, T2, and T3, respectively-largee for timessensitvve tasks but still material for complex historical investigations.
These patterns indicate that FINSEARCHCOME stresses complex search and reasoning, where suc- cess requires planning multi-step queries, aligning timelines and identifiers across sources, and re- solving conflicting evidence. In turn, performance reflects not only access to documents but also the ability verify, synthesize, and reason.
4.2 HOw DO FINANCIAL PLUGINS IMPROVE THE PERFORMANCE ON TOP OF GENERAL SEARCH TOOLS?
A comparative analysis reveals that Deepseek RI on the YuanBao platform, augmented with a fi- nancial plugin, significantly outperforms its counterpart on the DeepSeek web interface for financial tasks, as shown in Figure 4 This enhancement particularly evident in T1, with a 31.9 percent- age point improvement. The plugin provides direct access real-time and historical financial data,

---

Published as a conference paper at ICLR 2026
Global Subect
Greater China Subact
Itures Pertermace
(eves)
Harther eeer couke (ons apror Gat (ree)
Ioen caryy Doen ente capp Cartir Taary Conter
the come
caser pour
Trmet
Got dreed
dhe ereal
they eante
Foom
the dhean
(a) Task 1. Time sensitive data fetching.
Creater
lia
Propoes cort
sweld
(art)
Fwedd
Almo srr apcc Frwer rleed)
ovel
Croca Anst
Dogher coog
awell
owel
dis card
Footh coutd Cowll
Ci fooll
Feet
(b) Task 2. Simple historical data lookup.
Greater Subat
Usem Notermass
the By
ais Malk
(sem
feetian Lave Veetio
Coms foodd
Cencel caary
Dhaike Cye reeal
feette cteyy
Westerr cueet
werterr mowe vecer
restiie darven taa
eprce Cod
(c) Task 3. Complex historical investigation
Figure 2: The performance of various models across the three tasks on FINSEARCICOMP. Models with 0 scores are all APIS.
Ixcesae
32.2
$7.0
34.6
11.1 24.6
1:21
15.
Genaii
ERNIE-XI
Figure 3: Performance improvement of search-augmented models over models without search ca- pabilities.
such as asset prices and corporate income statements, overcoming the limitations of standard web searches which often yield outdated information. Despite these gains, the performance of YuanBao- RI (web) remains suboptimal. This suggests that while specialized tools are beneficial, model's intrinsic capabilities are also critical for effectively searching and reasoning with financial data.
4.3 How DOES MODEL ORIGIN IMPACT THE PERFORMANCE?
We compare the scores the US models and the Chinese models on the Global and Greater China subsets in Figure 5] We define asset origin ratio Global subset scores / Greater China subset scores. Higher ratio means better performance on tasks related to global assets than tasks related to Chinese assets, and vice versa. We observe that models from US tend to show stronger data search performance for global assets, while models of Chinese origin appear more proficient with Chinese assets. This pattern suggests underlying differences in search tool integration or model capabil- ities between the two groups. Specially, for tasks TI and T2, US models consistently yield an asset origin ratio exceeding 100%. In contrast, Chinese models register ratio below 100%. However, for
8

---

Published as a conference paper at ICLR 2026
Bato of Global to Chinss Suboet Scoees
DeepSeck-R1 (weh)
Uniled States Models
244%
YannBa--R (weh)
Chinese Models
200
Perfrmsanc
30
40
37.3
35.2
1233%
91%
102%
50%
50
10
Figure 5: A comparison of US and Chinese models on the ratio of Global to Chinese scores.
Figure Average performance change on DeepSeek RI induced by financial plugins.
task T3, majority of models achieve ratio greater than 100%. This implies that the global asset challenges within T3 are less demanding than the Chinese asset challenges. Additionally, among the Chinese models, Doubao and Kimi k2 achieve the highest asset origin ratios. The rank suggests their capabilities are more balanced and less skewed toward domestic assets when compared to other models from the same region.
4.4 WHERE DO MODELS EXCEL AND FALTER ON FINSEARCICCOM??
T1. Time-Sensitive Data Fetching. Financial plugin-augmented products (GPT-S-Thinking, Hun Hlun-uan-TY DouBao; all web) exceed 70% accuracy, outperforming base LLMs. See Section for details. Deepseek-R1 (web) on its official site scored 28.8, 12.4% lower than with YuanBao, highlighting the value of specialized retrieval. Failures arise from antiacttvaiion outdated content, or misselection among conflicting sources.
T2. Simple Historical Lookup. Grok 4 (web) ranks highest via diverse reliable sources. Some products rely on parametric memory without search, yielding inaccuracies. Most extract from news rather than filings, missing details like prepaid expense.
T3. Complex Historical Investigation. Only Grok 4 (web) and GPT-5-Thinking (web) surpass
30. The challenge is structured retrieval (API/SQL), lacking in search-only systems. Successes appear only in simple cases (c.g., difference of two weekly closing prices).
5 RELATED WORK
Financial Benchmarks. Early efforts such as FINQA (Chen al. 2021) and its conversational extension CONVFINQA (Chen et al. 2022) target numerical reasoning over annual reports by re- quiring models to compose multi-step programs that combine text and tabular evidence. Subsequent
suites widened both task type and language coverage: FLUE (Shah et al. 2022) aggregated classic tasks in finance, while FINEVAL (Guo al. (0255) and MULTIFINBEN (Peng al. 2025) intro- duced large-scale Chinese and multilingual collections spanning classification, extraction, genera- tion and multimodality. Moving closer to real-world practice, FINANCEQA (Mateega et 2025)
gathered zero-tor crr--tolcrance questions written by buy-side professionals, and BIzFIN BENCH (Lu et al. 2025) distilled 6.7K genuine queries from a popular investment-searde app probe long-context and noisy scenarios.
However, these datasets provide relevant financial data by default, which substantially mitigates the challenge of financial data search from open-domain sources (Wei et al.) 2025). While Finance Agent Benchmark (Bigeard et al., (025) incorporates open-domain financial data search, it limited to scarching static historical data. This design introduces the possibility of data memorization by models, thus failing to adequately evaluate their financial data search capabilities.
Agentic Benchmarks. To assess end-to-end decision-making several works frame evaluation as goal-directed interaction with external tools. In finance, the FINEVAL (Guo et al. 2025 agentic track scores models planning, API use and long-horizan reasoning across tasks such as financial question and answering, financial text classification Beyond finance, BrowseComp (Wei et al. 2025), BrowseComp-TH (Zhou al. 2025), and BrowseComp-Plus (Chen al. 2025) pose ques-

---

Published as a conference paper at ICLR 2026
tions that require persistent web navigation and creative search strategies, offering challenging yardstick for browsing agents. These studies highlight the gap between token-level metrics and practical autonomy, motivating evaluation that couples financial expertise with realistic tool use.
CONCLUSION
In this paper, address the critical lack of an end-to-end benchmark for evaluating LLM-based agents in financial data search, as prior work failed to assess agent capabilities in realistic, context- free scenarios. To fill this gap, introduce FINSEARCHCOMP, the first fully publicly available benchmark designed for this purpose. It comprises 635 questions curated by experts across three de- manding tasks that require agents to orchestrate various tools, such SQL, APIs, and web search, to procure verifiable answers. Our holistic evaluation reveals that even state-of ofte-offthhearr the- agents signif- icantly underperform humans, often failing due insufficient search depth and the use of outdated information. We release FINSEARCHCOMP as vital resource to drive the development of more robust and reliable financial agents.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY
We have provided the prompts of this paper in Appendix D We provide the data of FINSEARCH- CОMP in Github and Huggingface
ETHICS
All models (products) used in this paper are publicly available, and our usage follows their licenses and terms. Additionally, we confirm that the compensation provided to annotators is significantly higher than the local minimum wage.
REFERENCES
Pranjal Aggarwal, Seungone Kim, Jack Lanchantin, Sean Welleck, Jason Weston, Ili Kulikov, and Swarnadeep Saha. Optmmalhinnknnbeechh Evaluating over and underthinking in lims, 2025. URL
https://arxiv.. http:://axxx..orr//a///22232 13141
Baidu. attps://ernie.aaion
2025.
Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. Finance agent benchmark:
Benchmarking Ilms on real-world financial research tasks, 2025.
abs/2508. 00828
ByteDance. https: www.:dduba..com wwwp:ddouuab..om/chaaaaaa chat/ 2025.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Maaii-Franccie Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3697-3711 1, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021. wwtp://.../oo//oo/oo.
emnlp-main.300) URL
main. main.3007
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang- ConvFinQA: Exploring the chain numerical reasoning in conversational finance question answering. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6279-6292, Abu Dhabi, United Arab Emirates, December 2022, Association for Computational Linguis- tics. doi: 10. 10.18653/w1/2022. http/22222..m2mmaiiii... 421. URL (RLL:ttts////eeaattoooooo conlp-maing 421/|
Zijian Chen, Xueguang Ma, Shengyao Zhuang. Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. Browsecomp-pile A more fair and transparent evaluation benchmark of deep-research agent, 2025. URL httpt:://///..............e. :/tp:::::///xvv..///////aa httparxvivvv// wwg. absa///b/// 2500 06600
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu. Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng
11

---

Published as a conference paper at ICLR 2026
Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. 7. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-RI: Incentivizing Reasoning Capability in LLMs via Rein- forcement Learning. URLhhtt:://axxtt org. abwpg//ab//a/ 25C1 12948
Accessed: May 2025.
Han Ding, Yinheng Li, Junhao Wang, and Hang Chen. Large language model agent in financial trading: A survey, 2024.
Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. Dabstep: Data agent benchmark for multi-step reasoning, 2025. URLhtttpp //arxiv.
org/abs/ 2506. 23719
Mohamed Amine Ferrag, Norbert Tihanyi, and Merouane Debbah. From Ilm reasoning tonomous ai agents: A comprehensive review, 2025. URL ht.tp:://arxvv.or///bt
2554.19677
Google. Proppreeie 05-06): A large language model. https://cloud, google. htt.:/aateeeeaaaa/ccc///c/o////s
com/vertex-ae cessed: May 2025.
2025. Ac-
Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang. Zhiqiang Liu, Sizhe Wang. Jinyi Niu, Chuqi Wang. Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, and Liwen Zhang. FinEval: A Chinese financial domain knowl- edge evaluation benchmark for large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Albuquerque, New Mexico, April 2025. Association for Compu- doi: 11118633v1/2255 hatp5/////2225.maal-ooo......3. URL
Long Papers), pp. tational Linguistics. ISBN 779-8-88176--88-66 https::/aclaamthhooorro.oo naacl -ong. 3187
Ziyan Kuang. Feiyu Zhu, Maowei Jiang, Yanzhao Lai, Zelin Wang, Zhitong Wang. Meikang Qiu, Jiajia Huang. Min Peng, Qianqian Xie, and Sophia Ananiadou. From scores to skills: A cognitive diagnosis framework for evaluating financial large language models, 2025. URL https:///
arxiv. absg///b/// /albs///550... 13491
Guilong Lu, Xuntao Guo, Rongjunchen Zhang, Wenqiao Zhu, and Ji Liu. Bizfinbench: business- driven world financial benchmark for evaluating Ilms, 2025. URLlhttss://acxivorr abs/2505..0 19457
Spencer Mateega, Carlos Georgeseu, and Danny Tang. Financeqa: benchmark for evaluating financial analysis capabilities of large language models, 2025. Utt/httsss//axxiv.co
abs/ 2501. 18062
MoonShot https:://///.... //km::////mm.... www.oooooooo.oooo..m htltps/s/kiin.mooossttt.com
2025.
Yugi Nie, Yaxuan Kong, Xiaowen Dong, John M. Mulvey, H. Vincent Poor, Qingsong Wen, and Stefan Zohren. A survey of large language models for financial applications: Progress, prospects and challenges, 2024. URL https::/p/.. /ttpp::////xiv..../ wwwp://wwwwoo///////////// 11903
12

---

Published as a conference paper at ICLR 2026
Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Jeff Zhao, Huan He, Yi Han, Yun Feng, Yuechen Jiang, Yupeng Cao, Haohang Li, Yangyang Yu, Xiaoyu Wang, Penglei Gao, Shengyuan Lin, Keyi Wang. Shanshan Yang. Yilun Zhao, Zhi- wei Liu, Peng Lu, Jerry Huang, Suyuchen Wang. Triantafillos Papadopoulos, Polydoros Gian- nouris, Efstathia Soufleri, Nuo Chen, Guojun Xiong, Zhiyang Deng. Yijia Zhao, Mingquan Lin, Meikang Qiu, Kaleb E Smith, Arman Cohan, Xiao-Yang Liu, Jimin Huang, Alejandro Lopez- Lira, Xi Chen, Junichi Tsujii, Jian-Yur Nie, Sophia Ananiadou, and Qianqian Xie. Multifinben: A multilingual, multimodal, and difficulty- aware benchmark for financial Ilm evaluation, 2025. URL httph:t/tasaaattee org/ httpg//b//20....222/ 14028
Qwen Team. Qwen3-235B: A large language model, 2025. URL Accessed: May 2025.
co/
Qwen/Qwent - 233B A22B
Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Ra- man, Charese Smiley, Jiaao Chen, and Diyi Yang. When FLUE meets FLANG: Benchmarks and large pretrained language model for financial domain. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pp. 2322-2335, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 211.16633/122222.mmmll maimmppmmii.14ee URL
httpss//belaatbboooo..orr////2/ wttp:///........m............... 148/
Tencent, httts://yuanbbc..eennttt..oo
2025.
Dingzirui Wang, Longxu Dou, and Wanxiang Che. A survey on table-and-text hybridqa: Concepts,
methods, challenges and future directions, 2023. URL http:tttaa///axi...rrr////// wtt..//////bb211222/.
13465
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsccomp: simple yet challenging benchmark for browsing agents, 2025. [tll:ttpp.//aaxxx..or//////////s
12516
xAI. Grok4: A large language model. https://doc....../.c/////// wx.aiidoca//odeeeee
x. /..i///a
models-and andeelss-nnd-ppiiinggg- pricingi 2025. Accessed: August 2025.
Hongyang Yang. Boyu Zhang, Neng Wang, Cheng Guo, Xiaoli Zhang, Likun Lin, Junlin Wang. Tianyu Zhou, Mao Guan, Runjia Zhang, and Christina Dan Wang. Finrobot: An open-source ai agent platform for financial applications using large language models, 2024. URL https:
7/ /arxiv. org/ abs /4055 14767
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LuMeee-a-judge with MT-bench and chatbot arena. In Thirty- Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //openreview.rr net. forum? dww.cccGDlaaaa....
Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. Browsecomp-zhh Benchmarking web browsing ability of large language models in chinese,
2025. URL http::::: // /aaxiiiv.. org/ wwwggabb//aae
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs only polish our paper for better clarity and fluency, without involving the core research content. All contents were checked and edited by the authors to ensure the quality and alignment. The authors take full responsibility for the final version the paper.
B DETAILS OF FINSEARCHCOMP
B. RELEASE FORMAT
The dataset is distributed as set of JSONL files containing (ID, TIER, QUESTION, TOOL..EMPPATT ANSWER, TRACE) plus an evaluation harness that replays traces a sandboxed environment. Detailed documentation and citation files accompany the release.
B.2 ILLUSTRATION OF INCONSISTENT CALCULATION METHODS FOR THE SAME METRIC
Stock Price Adjustment: Due to significant discrepancies in the calculation of forward- adjusted and beckward-adjustee prices across different databases, we uniformly query for non-adjusted prices only.
PE (TTM): The definition of "Earnings" can vary among different institutions.
Market Capitalization of Dual-Listed Companies: We either specify the calculation method (e.g.. "Price Share Class Price 2 Share Class 2" vs. "Price x Total Shares") or avoid such questions.
The timing for switching the main contract and the algorithm for constructing continuous contracts differ across institutions.
Futures Contracts:
Cryptocurrency: Prices vary across different exchanges.
B.3 TIME COST OF FINANCIAL ANALYSTS
Financial analysts worldwide dedicate substantial resources to these core information retrieval activ- ities. There are approximately 370,000 financial professionals in the US (based Bureau of Labor Statistics) and probably over million globally-inehuding equity researchers, portfolio managers, risk analysts, and investment bankers- regularly perform these tasks as part of their daily work- flow.
For TI (Time-Sensitive Data Fetching), analysts typically spend around 1-2 minutes per query for quick references. This task is also commonly performed by non-expert investors in scenarios such checking the current price of their investments.
For T2 (Simple Historical Lookup) represents the most frequent task type, with individual analysts performing 10-30 such queries daily for financial analysis. peer comparisons, and modeling. Each lookup averages 5-10 minutes, accounting for data validation (faster for standard financials that can be retrieved from filings, slower for less common financial data such as operational, macroeconomic, and industry data).
For T3 (Complex Historical Investigation) demands the highest time investment per-query, often requiring 15-60 minutes for comprehensive data retrieval and calculation, based on the complexity of the data retrieval and calculation steps. While less frequent (fewer than 20 queries per analyst monthly), these investigations and calculations are important building blocks in financial analysis and reports.
While standardized templates and automated tools already exist to facilitate these tasks-such as comparable company analysis frameworks that can be efficiently updated-appe approximately half of these information retrieval activities still inevitably require manual data collection and custom alytical framework development. If models could accurately perform such tasks, analysts could further automate these processes and significantly enhance overall productivity.
14

---

Published as a conference paper at ICLR 2026
B.4
CASES OF THREE TASKS
For Time-Sensitive Data Fetching, Some of the example questions include:
Latest closing price of Bloom Energy(NYSE)"
The latest opening price of Starbucks"
"The latest percentage change of Rivian (NASDAQ), based on the latest closing price and the previous closing price"
"USD/THB price today"
"Latest price of S&P 500"
For Simple Historical Lookup.
Some of the example questions include:
"What was the additional paid-in capital of Lands' End as of the end of the fiscal year 2020? (answer in thousand dollars, rounded the nearest integer)"
"What was the closing value of the VIX April 25, 2022? (rounded places)"
decimal "For the fiscal year 2023, what was Planet Labs' Net cash provided by investing activities? (please answer in thousands of dollars, rounded to the nearest integer)"
"What was the U.S. Housing Market Index (HMI) in November 2014? (answer rounded to the nearest integer)"
"In April 2015, what was the CPI of Russia? (base year 2015=100, rounded two decimal places)"
For Complex Historical Investigation. Some of the example questions include:
"What were the specific dates from January 1, 2020, December 31, 2024, when London Gold (XAUUSD) dropped by more than $80 in a single day? Please list these dates and the corresponding daily drop in USD (rounded to the nearest integer), presented in table sorted by date in ascending order."
"During April 2025, did the daily changes (compared to the previous day) in the central parity rates of EUR/CNY, HKD/CNY, and USD/CNY always occur in the same direction (i.e., all rates increasing together or all rates decreasing together)? Among the days when these rates did not move in unison, find the exact date when the USD/CNY central parity rate experienced its greatest single increase. List the three central parity rates (EUR/CNY, HKD/CNY, USD/CNY) for that specific day, rounded four decimal places."
"Which constituent stock of the Nasdaq 100 Index (NDX), with a Price-to-Earning Trail- ing Twelve Months (PE-TTM) greater than and less than as of the market close on September 30, 2024, reported the largest operating revenue (consolidated financial state- ments, in billions of USD) in its 2024 interim report? Please provide the stock name and its operating revenue. Unit: billion USD. rounded to two decimal places."
"What is the year-over-year change in the proportion of Johnson & Johnson's revenue from international markets (excluding the United States) for each the past three years (2022-2024)? Please provide the change in percentage points, rounded to two decimal places."
B.5 GUIDE FOR MITIGATING AMBIGUITY
15

---

Published as a conference paper at ICLR 2026
Table 4: Consolidated guide for annotation in FINSEARCHCOMP for mitigating ambiguity.
Category Topic
Description
Bad Example
Good Example
Corporate Calendar vs.
Questions must differentiate
"What was NVIDIA's revenue in 2024?"
"What was NVIDIA's
Fiscal Year
between calendar and fiscal
Funda- mentals
years. Many companies (e.g., NVIDIA) do not align their fiscal year with the calendar year. Default to "fiscal year" for consistency.
revenue for fiscal year 2024?"
Timing Description for Financial Statements
Use precise language for time periods. Income statement and cash flow items occur "over period", while balance sheet items snapshot "at point in time".
"What were the company's s assets in fiscal year 2023?"
"What were the company' 's total assets as of the end of fiscal year 2023?"
Financial
The variable name in the question must match the terminology used in the financial statements to avoid ambiguity (c.g.. "operating income" vs. "operating revenue").
"What was the company's operating revenue?" (when the report lists "operating income")
"What was the company's operating income?"
Item Naming
GAAP Non-GAAP
To prevent evaluation mismatches, questions must specify the standard (GAAP or Non-GAAP), This ensures the ground truth and the answer are based on the same accounting principles.
"What was the company's 's income?"
"What the company's ' net income, based on U.S. GAAP standards?"
Market Data
(Ambiguous; the ground truth might be GAAP while the answer is Non-GAAP)
Currency
To prevent evaluation mismatches, questions must specify the currency (e.g.. USD, CNY). This ensures the answer can be directly compared to the ground truth.
"What was the company's revenue?"
"What was the company's s revenue in millions USD?"
Industry Classification
a company's industry is mentioned, specify the classification standard (e.g.. a specific level of Shenwan CSRC industry codes) ensure consistency.
"What industry is the company in?"
"What is the company's industry classification according the Shenwan Level standard?"
Continued on next page
16

---

Published as a conference paper at ICLR 2026
Prompt Length
Length
Answer Type
Match Motric
90.5
50-
47.4
0.8%
40-
59.4%
$908%%
78,0%5
0-
Envater Caraa
Gineel
Gratter Caina
(a) Distribution of the answer type and evaluation rubrics.
(b) Average length of prompts and answers on each subset.
Figure Data statistics of FINSEARCHCOMP.
Table 4 Continued from previous page
Category Topic
Description
Bad Example
Good Example
Market Capitalization
For multi-listed companies, specify the exact calculation method. A simple "Market Cap = Total Shares" is ambiguous; a sum of market values from each listing is precise.
"What is the total market cap of a company dual-listed in Hong Kong and Shanghai?"
"What is the total market cap of the dual-listed company,
calculated as
(A-share price x A-share count)
(H-share price x
H-share count)?"
Futures Quote Notation
Futures quotes special hexadecimal notation (e.g., 113'0 The answer should accept both this format and the standard decimal equivalent be robust.
Answer requires "I 113.265562 only, but "I "113'08'5 5" is also acceptable.
The reference answer is "113. 11..265625, but 113'08' 5 is also acceptable."
Fixed Income & Macro
Currency Exchange Rates
Specify the type RMB exchange rate: onshore (CNY), offshore (CNH), or interbank, their values differ.
"What is the USD to RMB exchange rate?"
"What is the onshore USD to CNY exchange rate of [Date]?"
Answer Precision
Questions must specify the required precision for numerical answers (e.g.. number decimal places, rounding to nearest integer).
"What is the profit margin?"
"What is the profit margin percentage, rounded to two decimal places?"
General Rules
Unit Specification
Clearly state the unit for the answer (e.g., million, billion, USD, %).
"What the revenue?"
"What the revenue in billions of USD, rounded to the nearest integer?"
B.6 DATA STATISTICS
C DETAILED SCORES ON FINSEARCHCOMP
We show the detailed scores of various models FINSFARCHCOMP in Table 5
17

---

Published as a conference paper at ICLR 2026
T3:
13:
T1: 134
20
TE: 110
106
31
T2: 119
T2: 100
Maci Finuncial Ope sing
Stuck
Mazro Eoonony
Corporate Openting
Ecenonty
Camnay
Seomtary Markis
Otber
Curreny
Sccenday Market
Gihem
Others
Others
(a) Topic distribution of Global subset.
(b) Topic distribution Greater China subset.
Figure 7: Topic distributions in FINSEARCHCOMP.
Global Subset
Greater China Subset
Averge Store
-0-
K
H
Chinos Medels US Medel
ERNEE (oub)
Klsl (eell
Hantiami (Qub)
(mord (ovb)
Gamal (oub)
GPEE (evi)
Oweed (Bub)
Grmad (oul)
Kisu (ont)
Demdeek (ovb)
CPPSS (oud))
Han revi)
Dealher outy)
Destbr
BRaEE 21
Giok
Figure& 8: The performance of web-based products the global subset (left) and the Greater China subset (right) of FINSEARCHCOMP. Note that the performance of human experts 75.0 and 88.3 on the Global and Greater China subsets, respectively.
C.1 RUBRICS WITH VARIOUS ASSET CLASSES
Evaluation conducted during non-trading hours. Only minor discrepancies attributable rounding are permitted.
Mainstream Market Stocks and Indices:
Other Regional Indices: An answer is considered correct if its value falls within the day's high-low price range.
Foreign Exchange (FX) Rates: To account for potential discrepancies across different data providers, the valid range for an answer is defined as the high-low range of the day, augmented by an additional buffer.
D PROMPT
This section primarily describes the judge system prompts on three subtasks.
Judge for Time-Sensitive Data Retrieval
You are strict judge. Your task is to score student's 's response to financial question based on the question itself, the Real-time Authentic Information provide, and the Scoring Criteria. A score of means the student's response meets the requirements, and means it does not. Please provide your analysis first, then give final score. the final score 1, output ["score":1]: it is O, output "["score":0)' You must output strict JSON.
The specific rules follows:
18

---

Published as a conference paper at ICLR 2026
Table 5: Performance of various models and human on FINSEARCHCOMP.
Global
Greater China
Model
Reasoning Search
Avg.
T1
T2
T3
Avg.
T1
T2
T3
Avg.
Human Performance
100.0
73.3
51.4
75.0
100.0
88.1
76.7
88.3
81.6
Web-based products
Grok 4
87.3
68.1
51.2
68.9
64.9
67.0
23.9
51.9
60.4
GPT-5-Thinking
76.9
67.2
47.6
63.9
60.4
63.0
15.9
46.4
55.2
Gemini2.5-pro
56.0
44.5
27.4
42.6
51.9
46.0
12.5
36.8
39.7
DouBao
61.2
33.6
22.6
39.1
88.3
63.0
11.4
54.2
46.7
DouBao-Thinking
34.3
33.6
21.4
29.8
62.2
61.0
23.9
49.0
39.4
YuanBao-Hun Yunn-aa-HuYYunn-- -I-Thinkig
59.0
18.5
11.9
29.8
82.0
58.0
11.5
50.5
40.1
YuanBao- YuanBao-DeepSede Deeppeek-VV3
53.0
24.4
14.3
30.5
81.1
55.0
10.2
48.8
39.7
Deepee
22.7
13.1
29.8
84.7
58.0
14.8
41.2
Kimi k2
30.6
47.1
10.7
29.5
35.1
73.0
6.8
38.3
33.9
60.2
37.8
14.3
37.4
18.1
42.0
5.7
21.9
29.7
DeepSeek-R1
17.9
19.3
14.3
17.2
56.8
51.0
13.6
40.5
28.8
ERNIE-XI
23.9
15.1
10.7
16.6
62.2
49.0
11.4
40.8
28.7
APIs
Gemini2.5-pa (Google Search)
42.5
17.1
25.0
38.2
28.8
52.0
13.6
31.5
34.8
Gemini2.5-pmo
0.0
22.7
13.1
11.9
2.2
12.0
10.2
8.1
10.0
Hunyuan- Ti-latest
15.5
19.8
52.0
21.1
Kimi k2
0.0
23.5
10.7
11.4
0.0
14.0
8.0
7.3
9.4
DeepScek-R1
0.0
14.3
9.5
7.9
0.0
7.0
6.8
4.6
6.3
ERNIE-X1
0.0
10.1
8.3
6.1
0.0
11.0
6.8
5.9
6.0
DouBao
0.0
16.0
4.8
6.9
0.0
6.0
8.0
5.8
DouBao- ThBBai-Thinking
11.9
7.9
0.0
7.0
2.3
5.5
0.0
2.5
1.2
1.2
0.0
0.0
1.1
0.4
0.8
If the Student Answer is empty, score 0.
If the Student Answer is not empty, but the data in the Real-time Authentic Information is empty (contains numbers), output """sc"re"""nul"""
The "Scoring Criteria Required Content" is minimum information the student must provide. If the student's response does not meet the Required Content (e.g.. is missing content), score O. Providing extra information is not error and will not result in deduction. You only need to focus the information itself, not the time. You should not deduct points because the question asks for the "latest price" and student only provides the "price on month x, day x."
The "Real-time Authentic Information" is the **correct response information for the user's question at the time of inquiry, and you should completely trust it. If the student's response meets the Required Content, use the Real-time Authentic Information the ground truth to verify the correctness of the student's answer. Score it according the "Scoring Criteria Accuracy Requirements": score if meets the requirements. and if does not.
the "Scoring Criteria Accuracy Requirements" is "Only rounding errors are allowed." it means difference between the student's number and the real-time information is permissible if it's caused by rounding to different number of decimal places. For example, 12.26 vs. 12.256 considered match, scoring
the "Scoring Criteria Accuracy Requirements" "Any answer between the high and low price is considered correct," " means the student's number is correct if it falls between the maximum and minimum values provided in the Real-time Authentic Information. Rounding errors are also allowed. For example, if the Real-time Authentic Information gives high of 12.256 and low of 12.152, student answer of 12.26 is considered within the correct range and scores while answer 12.27 is not and scores 0.
If the "Scoring Criteria Accuracy Requirements" is "The allowable error range absolute value of +xx," it means the student's 's number is correct if it falls within the range of the Real-time Authentic Information's 's number +xx. For example, if the requirement is "The allowable error range is absolute value =0.01%," any student answer within the range [13.55% 13.57%] is correct and scores 1; otherwise, it scores If the "Scoring Criteria Accuracy Requirements" is "Expand the high-low price range by absolute value of a." the correct range is from "minimum value a" "maximum value a" based on the Real-time Authentic Information.
number in the Real-time Authentic Information 13.56%, then
19

---

Published as a conference paper at ICLR 2026
Note:
You only need to verify numerical values like stock prices/indices, not dates. As long as the numerical values correct, the student's response is considered correct. You should not deduct points the student's response mentions "Iatest/yesterday" while the reference information specifies particular date (e.g. "Month X, Day Y"). nor should you deduct points if dates do not align (due to potential issues with time zones, trading days vs. calendar days, etc.).
The following examples fictional dates and numbers. So do not refer the specific fig- ures.
Example 1:
Question: How is NVDA's stock market performance today?
Scoring Criteria:
Required Content: NVIDIA's latest opening price, high price, low price, and change percentage.
Accuracy Requirements: Only rounding errors are allowed
Real-time Authentic Information:
"data": "symbol": "NVDA".
"open": "112.905",
"high": "114.665",
"low": "112.660".
"price": "113.820".
baatest.tradng__dae "200 "2025042"",
pllsse" "114.5"" "114.500", "change": "-0.680", "change.pereent "-0.59%"
Student Answer:
According the latest of 17:47 Beijing time April 27, the close of the most recent U.S. trading day, NVIDIA's (NVDA) stock price fell by 0.59%.
On that day. NVIDIA's opening price was $112.91, the high was $114.67, the low was $112,66, the volume was 130 million, and the turnover was $15.15 billion. The after-hours price was $1 113.288, down $0.54 from the closing price, decrease of 0.47%.
Your Output:
Analysis:
Does the student's answer meet the Required Content: Yes:
Are the numbers consistent (allowing for rounding errors): The opening, high, and low prices consistent, with only rounding errors; The student's answer about the change percentage (-0.47%) is not consistent with the real-time information's "change-pereent": "-0.59%". Not all requirements are met, final score is O; Final score: ("score":0)
Example
Question: USDICNY onshore exchange rate
Scoring Criteria:
Required Content:
The latest onshore USDICNY exchange rate
Accuracy Requirements: Any answer between the high price considered correct
Real-time Authentic Information:
"currency pair": "USDICNY".
"exchange per": "-0.0403",
"exchange_range" range": "-0.0029"'
Exxchange_rate"" "7.1871",
"exchange_rat" "irtange_rate_h"" _i": "7.1934".
"exchange_rae _rate_ooo_t__o "7.1855",
aopen_exchange.rge r: "7.1900"
"pre_close_ www..clos..exccanee.eee rate": "7.1900",
"radding date": "20250605"
20

---

Published as a conference paper at ICLR 2026
Student Answer:
of 03:00 Beijing time on June 5, 2025, the onshore yuan (CNY) closed at 7.1905 against the US dollar, down 26 points from the previous trading day's night session close, with trading volume of $36.094 billion.
Additionally, on June 5, 2025, central parity rate RMB inter-bank foreign ex- change market was 7.1729 yuan per US dollar, an appreciation of 32 basis points from the previous trading day.
Your Output:
Analysis:
Did the student answer the Required Content: Yes;
Does the number meet the accuracy requirement: The threshold range from the real-time information is [7.1855, 7.1934]. The student's is answer, 7.1905, falls within this range and meets the requirement. Final score: ("score":1)
Example 3:
Question: Latest COMEX gold price
Scoring Criteria:
Required Content: Latest COMEX gold price
Accuracy Requirements: The allowable error range is an absolute value #0.6
Real-time Authentic Information:
"res":
"request": "W.WVq("GC.CM"""" date,, time.rt last"""
"code":
"data":
"GC.CMX":
"RT DATE": 20250619,
"RT TIME": 90628,
"RT LAST": 3383.2000
Student Answer:
According to Hithink RoyalFlush Futures data, of 23:22 on June 18, 2025, the price of New York gold (COMEX Gold Futures) $3382.7 / ounce, down $25.4 from the previous trading day, decrease 0.75%, with a high $3405.2 ounce and low of $3363.6 ounce.
Your Output:
Analysis:
Student's
the Required Content: Yes:
the accuracy requirement: The threshold from the real-time information is 3383.2000 + 0.6, which is [3382.6000, 3383.8000J. The student's answer, 3382.7, falls within this range and meets the requirement.
Does the number
Final score: ("score":1]
-an==EEedd End of examples, this your task
Judge for Single Historical Data Point Retrieval and Complex Data Retrieval
You are intelligent judge and scorer for answers to financial questions. You will receive Question, its Reference Answer, and Student Answer. Some Reference Answers may be supplemented with "Scoring Criteria". You need evaluate the Student Answer and complete the following tasks:
1. Based on the content of the Student Answer, accurately identify its final answer (identifica- tion only, no need output). You can identify the position and content of the final answer by analyzing the Student Answer by searching for keywords, including but not limited "the answer is." "the final result is." "the correct option is." etc. the Student Answer empty, meaning it contains no content, assign score of directly and skip steps 2 and 3 below. 2. Separately list the final answer from the Reference Answer and the final answer you identified from the Student Answer, and compare the two (no need to output the listing and comparison process or results).
21

---

1. You do not need to and should not answer or solve the question yourself. Your sole task is Published as a conference paper at ICLR 2026
Based on result of the comparison any Scoring Criteria that may provided with the Reference Answer, judge whether Student Answer is correct and assign score. The score can only be 1 or O; indicates the Student Answer is correct, and 0 indicates it is incorrect. No scores other than 0 and I are permitted.
No
to judge and score.
The Reference Answer is accurate and correct. You can fully trust it.
3. the Reference Answer contains 2 or more key points, such as a person's name and a number, an item and a time, or multiple parallel locations, the Student Answer can only receive point if it provides all the key points and all of them correct. If only portion of the key points are provided or correct, the score O.
Student Answer is empty or an abnormal enror message. please assign score 5. Numerical values of the same magnitude written in different formats are considered the same, for example. 12.45% vs 0.1245, or 120,400,000 vs 120.4 million. or 2/5 vs 0.4. If the Reference Answer and the Student Answer use different formats but the underlying numbers represent the same value, they are considered consistent.
6. An answer is deemed correct numerical component matches the Reference Answer, even the unit is omitted. For example, if the Question asks for "xx revenue in billions of yuan", a Student Answer of "12.98" is valid match for the Reference Answer "12.98 billion yuan". Similarly, if the required unit specified "%" and the Reference Answer is "12.98%", Student Answer of "12.98"
valid match
"Regarding your output format:** You need to first output the evaluation result, then provide the scoring hasis and the score. Your output format is:
Scoring Basis: (Briefly state your reason for the score. not exceeding 150 words or 150 to- kens)
-SSON:
{" ("answer_score" your score}
###
"Exmmple +*
Question:
What is the average number of years it took for the C9 League university presidents who spoke at the
2024 graduation ceremonies become president of university for the first time after receiving their PhD? (Calculate years, round decimal place) Reference Answer:
27.9 years
Scoring Criteria:
The number in the Student Answer must be strictly identical to the number in the Reference Answer.
Student Answer:
Based on known information, average time current presidents C9 League universities to become a president for the first time after their PhD is approximately 27.3 years. Here is the specific calculation:
1. Peking University: President Gong Qihuang, PhD in 1992, became president in 2022, gap of 30
years.
2. Tsinghua University: President Wang Xiqin, PhD in 1996, became president 2022, gap of
years.
3. Fudan University: President Jin Li, PhD in 1994, became president in 2021, gap of 27 years.
4. Shanghai Jiao Tong University: President Ding Kuiling, PhD 1990, became president in 2021.
gap of 31 years.
5. Nanjing University: President Lü Jian, PhD in 1993, became president in 2018, a gap of 25 years.
6. University of Science and Technology of China: President Bao Xinhe, PhD in 1991, became president in 2017, a gap of 26 years.
Xi' an Jiaotong University: President Wang Shuguo, PhD 1986, became president in 2014, gap of years.
8. Zhejiang University: President Wu Zhaohui, PhD in 1993, became president 2015, gap of 22
years.
22

---

Published as a conference paper at ICLR 2026
Harbin Institute of Technology: President Han Jiecai, PhD 1991, became president 2022, gap of 31 years.
Summing the years:
30 + 26 27 31 25 26 28 + 22 + 31 246
Average time:
246 27.3 years
Therefore, on average, it took these presidents approximately 27.3 years to become a university president for the first time after their PhD.
Scoring Basis: This question has Scoring Criteria, which require the number in the Student Answer number
to strictly identical
Reference Answer. Here, number the Student Answer is not strictly identical to the number in the Reference Answer. Therefore, the Student Answer incorrect, and the score is O. IISON:
(" ("answer score": 0}
###
"Example 2**
Question:
An interesting aspect of the United States is that a state's political center often not its economic center. For example, the capital of California is the relatively lesser-known Sacramento, not the economically developed and famous San Francisco Los Angeles. In contrast, in China, the provincial capital of each province (excluding autonomous regions) often the city with the highest GDP that province Using the 2022 GDP as the standard, please list the provinces in China where the city with the highest GDP within the province is not the provincial capital. Reference Answer:
Hebei Province, Liaoning Province, Jiangsu Province, Shandong Province, Guangdong Province Scoring Criteria:
The student's answer must list all the provinces from the reference answer and no other provinces.
Student Answer:
According to 2022 GDP data, the provinces in China (excluding autonomous regions) where the city with the highest GDP not the provincial capital include:
1. Hebei Province: The city with the highest GDP Tangshan, while the capital Shijiazhuang.
2. Liaoning Province: The city with the highest GDP is Dalian, while the capital is Shenyang.
3. Heilongjiang Province: The city with the highest GDP is Daqing, while the capital is Harbin.
4. Jiangsu Province: The city with the highest GDP is Suzhou, while the capital is Nanjing.
Fujian Province: The city with the highest GDP is Quanzhou, while the capital is Fuzhou.
6. Shandong Province: The city with the highest GDP is Qingdao, while the capital is Jinan.
7. Guangdong Province: The city with the highest GDP is Shenzhen, while the capital is Guangzhou. often concentrated in non-capital cities, reflecting different regional economic development patterns.
These provinces' economic centers
-Scoring Basis: This question has Scoring Criteria, which require the Student Answer to list all the provinces from the Reference Answer and no other provinces. Here, the Student Answer lists all the provinces from the Reference Answer but also includes provinces not in the Reference Answer [[leilnngiiagg Province, Fujian Province). Therefore, the Student Answer is incorrect, and the score
JSSON:
s"ans""r 0}
###
*Example 3+*
Question:
How many Michelin three-star restaurants San Francisco specialize in Asian cuisine, and what are
their names?
Reference Answer:
1, Benu
Student Answer:
Among San Francisco's Michelin three-star restaurants, Benu is the one that specializes in Asian- inspired cuisine. Helmed by renowned chef Corey Lee, Benu offers modern Asian cuisine, fully showcasing Lee's creativity and skill. The dishes blend Korean, Chinese, and other Asian elements
23

---

Published as a conference paper at ICLR 2026
with a high degree innovation refinement. According current information, there Michelin three-star restaurant in San Francisco specializing Asian cuisine.
Scoring Basis: This question does not have Scoring Criteria. Here, the Student Answer states that there is Michelin three-star restaurant specializing Asian cuisine in San Francisco, and its name is Benu. Although the wording different from the Reference Answer, the meaning is the same. Therefore, the Student Answer is correct, the score -JSON:
scnne"" 1}
###
*Example 4+*
Question:
In terms of area, which provincial capital city China (mainland provinces) closest in to Slovenia? What is the difference in square kilometers?
Reference Answer:
It is closest Kunming, with difference of about 740 square kilometers. Student Answer:
The total area of Slovenia is 20,273 square kilometers. Among China's provincial capitals, the one closest in area is Kunming, with an area of 22,100 square kilometers. The difference between the two is: 22,1220.0 20,273 = 1827 square kilometers. Scooring Basis: This question does not have Scoring Criteria. Here, the Student Answer correctlyy identifies Kunming as being closest area to Slovenia. which is consistent with the Reference Answer. However, the area difference of "1827 square kilometers" given in the Student Answer is inconsistent with "about 740 square kilometers" in the Reference Answer. Therefore, the Student Answer is incorrect, and the score is O. -JSON:
(" ("answer score": 0}
###
hope you are competent for this job, as it very important to me. If you perform well, will reward you accordingly. Otherwise, I may penalize you.
E ADDITIONAL ANALYSIS EXPERIMENTS
E DOES REASONING ABILITY ENHANCE PERFORMANCE ON FINSEARCHCOMP?
An evaluation of models within the same series is
Thinking
detailed in Figure 9 We observe an average decline of 7.0 points for TI for reasoning capacity, which is likely due to the low complexity of the task and po- tential overthinking of reasoning models (Aggarwal et al. 2025). For T2 and T3, the change for adding reasoning capacity negligible.
80
0
Doubuo
Yaukuo
Figure 9: Performance changes from the rea- soning capability across models in the same series.
F CASE
In this section, we present typical cases identified during evaluation FINSEARCHCOMP.
24

---

Published as a conference paper at ICLR 2026
web-based product successfully addresses a question of Time-Sensitive Data Fetching in FIN- SEARCHCOMP by leveraging financial plugins.
Question
walmart current stock price
Response
As of the U.S. market close on August 27, 2025, the stock price of Walmart Inc. (Ticker: WMT) was $96.08, marking a daily gain of 0.03% with a trading turnover of $1.241 billion. The price represented slight increase from the previous day's close of $96.05. During the session, the stock reached intraday high of $96.32 and low of $95.60.
In terms of recent performance, Walmart's 's stock has declined by 6.33% over the past five trading days, yet it still maintains a year-to-date (YTD) gain of 7.13%. Analysts note that despite facing short-term pressure from tariff' costs, the company has sustained its market competitiveness through supply chain optimization and its omnichannel strategy.
Search Sources
1. tradersunion.co com
Walmart Inc. (WMT) Stock Price Forecast for 30 days: Today, Tomorrow, Next Week August 27, 2025 The current price of Walmart Inc. (WMT) is US$96.02. The analytical forecast indicates that the price of WMT, expected reach US396. 12 by the end of the week, could riseffall to USS99.35 35 by the end of the month.
2. tradersunion.com com
Walmart Inc. (WMT) Stock Price Forecast for 30 days: Today. Tomorrow, Next Week August 27, 2025 The current price of Walmart Inc. (WMT) is USS96.02. 02. The analytical forecast indicates that the price of WMT expected reach USS96.12 by end of the week, and could riseffall to US$99.35 by the end of the month.
3. in.benzinga.com com
Walmart's Options: A Look at What the Big Money is Thinking Walmart (NYSE:WMT)
Benzinga
August 26, 2025 Whales with a of money to spend have taken noticeably bearish stance on Walmart. Looking at options history for Walmart WMT we detected 15 trades. If we consider the specifics of each trade, it is accurate to state that 33% of the investors opened trades with bullish expectations and 46% with bearish.
Tencent Financial Industry Database
Walmart WMTN
August 28, 2025
Real-time Stock Quote for Walmart (Code: WMTN in US Stocks: Data
Updated: 2025-08-27 16:04:58; Latest Price: $96.08; Previous Close: $96.05; Today's
Open: $95.90; Day's High: 896.32; Day's Low: $95.60; Day's Change: 0.03%; Day's
Amplitude: 0.74%; Turnover Rate: 0.16%; Volume (shares): 12, 12,925,656; Turnover (USD):
1,241,094, www.09948770000 879.00; Volume Ratio: 0.45; 5-Day Change: -6.33%; YTD Change:
7.13%; P/E
Ratio (TTM): 36.26; P/B Ratio: 8.51; Dividend Yield: 0.98%; 20-Day Change: -1.32%;
Total Market Cap (billion USD): 766.76.
tradersunion.co com
Walmart Inc. (WMT) Stock Price Forecast (30 days): Today, Tomorrow, Next Week August 27, 2025 The current price of Walmart Inc. (WMT) is $96.02. According to ana- lytical forecasts, the price of WMT is expected to reach $96. 12 by the end of this week, and could rise/fall $99.35 by the end of this month.
marketbeat.com
Walmart (NYSEWWT)) Price Target Raised to $114.00
August 24, 2025 Walmart (NYSE WMT) Price Target Raised to $114.00 Key Points The Goldman Sachs Group has raised Walmart's price target from $101.00 $114.00, indicat- ing potential upside of 17.25% from its previous close.
Tonghuashun Finance
Walmart's 's turnover on August 27 was $1.241 billion, decrease of 41.91% from the previ- trading day.
August 28, 2025 On August 27, 2025, consumer staples retailer Walmart (WMT) had turnover of $1.241 billion, ranking 44th US stocks for the day. The turnover decreased by 41.91% from the previous day, and the trading volume was 12.9257 million shares.
8. WeChat Official Account CC Finance Walmart, with Unimpressive Revenue and Profits, Has Stock Price Stronger Than the
25

---

Published as a conference paper at ICLR 2026
Magnificent Seven
July 29, 2025 The continuous rise of Walmart's 's stock price (around $95. 67..77e July 2025, market cap of $760 billion, up over 300% from its 2020 low), despite single-digit growth in revenue and profit...
WeChat Official Account Haishi Shibei
Walmart (WMT) Stock Price Under Pressure! Profit Misses Expectations for the First Time in Three Years, Soaring Tariff Costs Drag Down Profit Outlook
August 22, 2025 Transcript from the conference call: "Good morning/afternoon, everyone. Welcome to Walmart's Fiscal Year 2026 Second Quarter Earnings Conference Call. All participants will now be in a listen-only mode..."
A web-based product fails Single Historical Data Retrieval question FINSEARCHCOMP due absence of search utilization.
Question
How much cash was used in investing activities by Apple in the 2021 fiscal year? (Answer in millions of dollars, rounded nearest integer)
Reasoning
The user needs to know the amount of cash used in investing activities by Apple in the 2021 fiscal year, millions of dollars, rounded to the nearest integer. First, need recall Apple's financial reports, specifically the cash flow statement the 2021 fiscal year.
Apple's fiscal year typically ends September, so 2021 fiscal year would September 2021. According to public financial data, in Apple's 2021 cash flow statement, the net cash flow from investing activities is usually negative hecause investment expenditures exceed income. The "cash used" here refers the amount of total cash outflows from investing activities minus inflows, typically "cash used" a question refers the net cash outflow from investing activities. Consulting Apple's ' 2021 10-K report. the investing activities section includes capital expenditures (like PP&E), acquisitions, investment securitics, ctc. From memory, was the net cash outflow from investing activities for Apple's 2021 fiscal year approximately 144,266 million dollars? Or is more accurate data needed.
year ending
Confirming again, in Apple's 2021 fiscal year cash flow statement, the cash flow from investing activities was -144,266 million dollars. This means the cash used for investing activities was 144,266 million dollars. So the answer should be 144266.
Response
144266
Search Sources
No search sources used.
A web-based product successfully solves Complex Historical Investigation question by scoping its search to a reliable source (nasdaq.com) and then cross-verifying retrieved data through reverse searches.
Question
What was the change in opening price compared to the previous closing price (in dollars, unadjusted) on the effective date of Apple Inc.'s (AAPI.) most recent stock split? What was split ratio? (Round to two decimal places)
Reasoning
Now, let's try scarching NASDAQ for "AAPL closing price August 28 2020" to confirm the exact figure.
Searched for "AAPL closing price August 28 2020 sitc:nasdaq.com"
nasdaag.com (x10)
Searched for AAPL opening price August 31 2020 site:nasdaq.com"
naxdaq.com (x10)
26

---

Published as a conference paper at ICLR 2026
Confirming closing price
I'm searching for "AAPL August 28 2020 close 499.23" to verify the exact figure.
Searched for "AAPL August 28 2020 close 499.23"
statmuse.com
stockanalysis.com
investor.apple.com
financechurts.com
fool.com
Searched for "AAPL August 31 2020 open 127.58
Response
The most recent stock split for Apple Inc. (AAPL) was effective on August 31, 2020, with a split ratio of 4-forr-1
On that date, the opening price $127.58, compared the previous closing price of $499.23 August 28, 2020. The change was $3371.65.
27


Reviews:
[{"review_id": "wLJikRXUei", "reviewer": "Reviewer_9TaR", "summary": "This paper presents a benchmark for evaluating LLMs on real-world financial reasoning tasks that involve real-time data retrieval, historical lookup, and multi-source analysis. It includes 635 expert-verified queries, split into 337 Global and 298 Greater China tasks, created and reviewed by 70 finance professionals. The authors evaluate 21 models and show that even top web-enabled systems still fall short of human analysts, especially on time-sensitive or multi-hop reasoning. The main value lies in its holistic setup, as it tests the complete agent process from search and tool use to reasoning and final judgment within realistic financial workflows rather than isolated question answering.", "questions": "1. Was the same LLM judge and prompt used for the global/English and Greater China/Chinese parts, and did you check whether the 0–1 scheme behaves the same across languages or for partly correct long answers?\n\n2. Performance appears to depend on the region of both the model and its attached tools, with international systems performing better on global tasks and Chinese systems doing better on the Greater China subset. How much of these differences reflect model reasoning capability versus disparities in access to region-specific data sources or plugins?\n\n3. Given that 635 expert tasks is already expensive, is there a plan for semi-automatic growth, and how do you see this benchmark sitting next to Finance Agent Benchmark or FinEval?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The useful part of this work is that it evaluates the whole agent process in a financial setting where the answer can actually change over time. The model is expected to go out to a source (web or a financial plugin), pull time aligned data, match it to the task (for example the right fiscal date or the right currency), and then produce an answer that is checked by a rubric. It covers three analyst style tasks: time sensitive queries, point in time historical queries, and longer multi period questions, so it is not only checking whether the model can retrieve a document.\n\nThe task split into time sensitive, point in time historical, and longer multi period questions is sensible and shows where models start to fail. They validated the LLM as judge setup against human labels and got about 95 percent agreement, so the scoring looks trustworthy. They also released the scoring rules and judge prompts, which makes the benchmark easier to reproduce.", "weaknesses": "A limitation of the work is the effective scale of the benchmark. Although the dataset contains 635 tasks in total, the division into two regions and three task types results in relatively small groups; most subsets contain fewer than 150 items, and the most demanding setting (T3) has only about 80 examples. This makes fine-grained comparisons across models, topics, or markets less statistically robust than the headline number suggests. The coverage is also concentrated on U.S. and Greater China markets, with limited representation of other regions, so the current release should be viewed as financially realistic but not yet globally comprehensive.\n\nThe paper is reasonably transparent in terms of evaluation methodology, since the scoring rules and judge prompts are provided. However, reproducibility of the exact numbers remains difficult, because the time-sensitive tasks rely on live market data and some of the evaluated systems are proprietary web products.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vwSNCQkJCk", "reviewer": "Reviewer_f1kM", "summary": "This paper introduces FINSEARCHCOMP, an open-source benchmark designed to evaluate the search and reasoning capabilities of LLM-based agents in the financial domain.  FINSEARCHCOMP comprises 635 expert-curated questions divided into three realistic tasks. The dataset spans Global and Greater China markets and was developed through a rigorous quality-control pipeline involving 70 financial experts. The authors evaluate 21 models , finding that Grok 4 (web) and DouBao (web) lead the Global and Greater China subsets, respectively , though all models significantly trail human expert performance.", "questions": "See weakness above", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The benchmark's construction was overseen by 70 professional financial experts (50 annotators, 20 senior reviewers) from prominent institutions, ensuring high domain relevance and realism\n\n2. The paper provides a broad evaluation of the current landscape by testing a large set of 21 models, which includes 12 web-based products and 9 APIs.", "weaknesses": "1. The primary novelty, financial searching evaluation, is a superficial extension that does not meaningfully advance methodologies for financial retrieval or agent benchmarking. The dataset is modest in scale and task complexity remains shallow, yielding only incremental progress beyond prior financial QA benchmarks.\n\n2. Although the work claims to evaluate agentic capability, it does not assess actual agent frameworks or contrast general-purpose agents with financial-domain agents. Relying solely on API vs. web endpoints provides no evidence that the benchmark measures planning, tool choice, error recovery, or other core agent behaviors.\n\n3. The exclusive use of binary 0/1 accuracy collapses retrieval, reasoning, source selection, and tool-use behavior into a single outcome, preventing attribution of failure sources and limiting diagnostic interpretability.\n\n4. Multisource retrieval is presented as essential to financial tasks, yet the benchmark includes no metrics for source coverage, retrieval depth, or authority resolution. Thus, it remains unclear whether models perform multisource search at all, weakening a central premise of the benchmark.\n\n5. The annotation process depends on senior expert arbitration but reports neither annotator disagreement nor arbitration rates, leaving uncertainty about reliability, transparency, and potential bias in conflict resolution.\n\n6. Market coverage is overstated: the benchmark focuses mainly on Western and Greater China markets and omits major regions (e.g., Japan, India, emerging economies). Coupled with the limited dataset size, this constrains representativeness and community impact.\n\n7. Although real-time correctness is highlighted, the protocol lacks detail on how temporal consistency is enforced between model inference and ground-truth snapshots, especially for volatile assets, creating concerns about reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y5AjElf4q1", "reviewer": "Reviewer_UJjp", "summary": "This paper introduces FinSearchComp, an open source benchmark for evaluating financial data search and reasoning via LLM agents. The benchmark covers 3 regimes of QA tasks: 1. Time sensitive data retrieval (real time quotes); 2. Simple historical lookups (point in time facts from records); 3. Complex historical investigations (multi-hop and multi step analysis across multiple sources, ex: search for best performing period). The authors also compared an extensive array of providers and models with regional origin clearly labeled. Further more, the analysis is ablated on greater China and western markets, with experiments faceted by region and provider. The dataset is 635 questions labeled by experts, and evaluations by LLM judges are grounded in human alignment of 97%.", "questions": "1. For T1, T2, T3 can independent researchers fully reproduce the ground truth using the released artifacts and public sources? Is there any private information that requires subscriptions? Can you also specific what tools and time budget human labelers had during annotation?\n2. Judge robustness: how did the judge alignment differ from the same model vs different families of models?\n3. Time-stamping for T1: how are the authors handling live snapshots for reproducibility? Do authors encode universal time zones in their templates? How can this be reproduced for the original analysis in August vs today?\n4. Could you quantify the error taxonomy in T3? Which error is most common? Misaligned fiscal calendars, units, etc?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Human experts involved in labeling process. Quality controls are included in curation.\n2. Well motivated task breakdown into realtime, search, and aggregate query. Paper provides examples in appendix across different topics. \n3. Coverage and experimental breakdown by regions and providers. 21 providers are experimented, and results highlight regional strengths. This allows for a fair and balanced evaluation of origin bias and current gaps in models for financial tool calling. \n4. Open source commitment and good evaluation of tools. Ablations are conducted in api vs web enabled products. \n5. Highlights clear SOTA gaps vs human annotations, gives a good future direction for development with the dataset (T3). \n6. Handles realtime queries, not just point in time.", "weaknesses": "1. Limited scope to search insights, and the dataset is not focused on reasoning or future looking guidance. Most analysts are focused on providing guidance with research. \n2. Limited 20 day window for search in August. Authors should consider adding different time seasons to encode seasonality in their analysis. \n3. Limited details on data sourcing and licensing. How will the open source benchmark handle possible access issues in the underlying sources, if from different providers or services. \n4. For T3, the scores are low but the paper doesn’t dive deeper into which sub-step fails. Is this a search issue, a tagging issue? Is this a reasoning failure by models? Context length issues?\n5. Human labeling - details missing on how much time each labeled was given, what resources were available during the annotation process (terminals, APIs, fin service access)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pIc63bmGEH", "reviewer": "Reviewer_i6JZ", "summary": "FinSearchComp introduces the first open-source benchmark specifically designed to evaluate LLM agents in realistic financial search and reasoning scenarios. The benchmark replicates real financial analyst workflows through three task types: Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation. Developed with input from professional financial experts and a multi-stage quality assurance process, it contains 635 annotated questions across global and Greater China markets. Overall, FINSEARCHCOMP provides a rigorous, professional-grade testbed for assessing end-to-end financial search and reasoning in LLM-based agents.", "questions": "1. Do the three task types mentioned in the paper (i.e., Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation) fully capture the main information retrieval scenarios encountered by financial analysts in real-world work? Are there any important task types missing, such as forecasting, report generation, or causal reasoning?\n\n2. From Figure 3, models without search capabilities still achieve non-trivial scores on T2 and T3 questions. The paper attribute this to parametric memory from pre-training. However, could these results also indicate that many questions in T2 and T3 do not genuinely require search to answer, thus weakening their effectiveness in evaluating search-augmented reasoning?\n\n\n\n#####", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Considering the critical importance of data timeliness in financial scenarios, the benchmark is designed from this perspective to test the capabilities of LLM agents.\n\n2. The tasks are constructed through expert human validation rather than LLM-generated content, ensuring high-quality and reliability of the benchmark questions.", "weaknesses": "1. In the case study (page 25), one example shows a web-based LLM agent answering purely from memory, without using a search tool. In real scenarios, an LLM might appear confident while fabricating both reasoning and sources. Therefore, attributing errors solely to parametric memory is inaccurate (as noted in line 377), hallucination itself is another major factor contributing to model mistakes.\n\n2. The paper contains a large amount of content, and to save space, the authors often refer readers to the appendix for details. This makes the reading experience somewhat incomplete. If possible, more of these details should be presented in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)