EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements
Authors: Issa Sugiura
Takashi Ishida
Taro Makino
Chieko Tazuke
Takanori Nakagawa
Kosuke Nakago
David Ha
Venue: ICLR 2026 Poster (2026)
Keywords: Financial Large language models
Financial benchmark
Accounting fraud detection
Earnings forecast
Primary area: datasets and benchmarks
TL;DR: Evaluating LLMs on complex financial tasks

Abstract: Large Language Models (LLMs) have made remarkable progress, surpassing human performance on several benchmarks in domains such as mathematics and coding. A key driver of this progress has been the development of benchmark datasets. In contrast, the financial domain poses higher entry barriers due to its demand for specialized expertise, and benchmarks remain relatively scarce compared to those in mathematics or coding.
We introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate LLMs on challenging tasks such as accounting fraud detection, earnings forecasting, and industry classification. EDINET-Bench is constructed from ten years of annual reports filed by Japanese companies. These tasks require models to process entire annual reports and integrate information across multiple tables and textual sections, demanding expert-level reasoning that is challenging even for human professionals.
Our experiments show that even state-of-the-art LLMs struggle in this domain, performing only marginally better than logistic regression in binary classification tasks such as fraud detection and earnings forecasting. Our results show that simply providing reports to LLMs in a straightforward setting is not enough. This highlights the need for benchmark frameworks that better reflect the environments in which financial professionals operate, with richer scaffolding such as realistic simulations and task-specific reasoning support to enable more effective problem solving.
We make our dataset and code publicly available to support future research.

Published as a conference paper at ICLR 2026
EDINET-BENCH:
EVALUATING LLMS ON COMPLEX FINANCIAL TASKS USING JAPANESE FINANCIAL STATEMENTS
Issa Sugiura!., 1.2, Takashi Ishida', Taro Makino', Chieko Tazuke* Takanori Nakagawa', Kosuke Nakago', David Ha' Sakana AI, KKyooo University
ABSTRACT
Large Language Models (LLMs) have made remarkable progress, surpassing human performance several benchmarks in domains such as mathematics and coding. key driver of this progress has been the development of benchmark datasets. In contrast, the financial domain poses higher entry barriers due demand for specialized expertise, and benchmarks remain relatively searce compared those in mathematics coding. We introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate LLMs on challenging tasks such as accounting fraud detection, earnings forecasting, and industry classification. EDINET-Bend is constructed from ten years of annual reports filed by Japanese companies. These tasks require models to process entire annual reports and integrate information across multiple tables and textual sections, demanding expert-level reasoning that is challenging even for human professionals. Our experiments show that even state-of state-of-the-arr art LLMs struggle in this domain, performing only marginally better than logistic regression in binary classification tasks such as fraud detection and earnings forecasting. Our results show that simply providing reports to I.LMs in straightforward setting is not enough. This highlights the need for benchmark frameworks that better reflect the environments in which financial professionals operate, with richer scaffolding such as realistic simulations and task lask-specific reasoning support enable more effective problem solving. We make our dataset and code publicly available to support future research.
https: http::///maaaatteee///////ttt/tt
Bench
Fraud not
Text
EDINET
Earnings change
Table
Industry class
Annual Report
Prediction
Figure 1: EDINET-Bende is a challenging benchmark evaluating LLMs on fraud detection, earnings forecasting, and industry classification from annual reports with text and tables.
INTRODUCTION
Large Language Models (LLMs) have recently demonstrated remarkable capabilities across diverse domains, including general knowledge, mathematics, and coding (OpenAl 2024 Anthropic 2024 Google, 20244) DeepSeck-Al 2025b) Kimi Team! (025).. A key driver of this progress Is the de- velopment of benchmark datasets. By evaluating models benchmarks, researchers can identify

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of financial benchmarks. Expert-level reasoning refers tasks that require synthesizing information across tables and text, including analyzing year-over-year trends, inter- preting relationships between multiple financial tables, and combining these signals with narrative disclosures. Tasks such as fraud detection and earnings forecasting fall into this category, they require comprehensive processing of both tabular and textual evidence together with domain-specific financial and accounting expertise.
Benchmark
Input modality
Expert level reasoning
Language
FinQA (Chen et al.
2021
Text, Table
En
ConvFinQA (Chen el al. 2022b
Text, Table
En
FinanceBench Islam et al. 2023
Text
En
FiTH (Zhao al. 2024)
Text, Table
En
FAMMA Xue at. 2023
Text, Image
En
AuditBench (Wang ct al.
2024)
Text, Table
En
gopanese-lm-tnnhaamee HHiano
2024
Text, Table
Ja
EDINET-Bench (Ours)
Text, Table
Ja
current limitations, refine model architectures, training data, and methods, and ultimately climb the "benchmark hill". As result, many existing benchmarks are now approaching saturation (OpenAT 2025), motivating the creation of new benchmarks that are substantially more difficult and less prone rapid saturation.
Prominent examples include Humanity's Last Exam (Phan et al. 2025 which consists of expert-level QA problems; SWE-Bench variants (Jimenez al., 2024] Chowdhury al. 2024 Deng al. 2025), which require fixing real GitHub issues so that unit tests pass; and WebArena (Zhou et al.
2024p, which evaluates agents operating real web browsers. Such benchmarks push models closer to weal-world usability by mirroring expert-level tasks.
While benchmark research has advanced across many domains, progress in finance has lagged behind (Lee et al., 2025). Existing financial benchmarks primarily focus on relatively simple tasks, such as knowledge-
QA or data extraction from tables (Chen et al. 2021] 2022b; [Islam et al.. 2023). These are valuable for measuring basic domain knowledge but fall short of the kind of complex, high-stakes tasks that, solved, would immediately translate to real-world utility.
To fill this gap, we introduce EDINET-Bench, an open-source financial benchmark constructed from ten years real-world financial reports filed through Japan's 's Electronic Disclosure for Investors' NETwork (EDINET EDINET-Bench presents demanding financial analysis challenges, including accounting fraud detection, earnings forecasting, and industry classification. These tasks require models process complete annual reports, combine information from multiple tables and text sections, and demonstrate expert-level financial reasoning. To the best of our knowledge, this the first open dataset and benchmark for accounting fraud detection.
We evaluate frontier LLMs on EDINET-Bench in zero-shot settings. Our results show that even state-of-the-art models struggle, performing only marginally better than logistic regression binary classification tasks. This suggests that merely providing annual reports to LLMs in simple setting is insufficient. Our findings underscore the need for benchmark frameworks that more closely mirror the environments in which financial professionals operate, incorporating richer scaffolding such as realistic simulations and task specifici reasoning support to facilitate more effective problem solving.
By releasing EDINET-Bench and its construction toolkit, we provide a foundation for studying expert-level financial reasoning with LLMs and for developing more powerful methods that can eventually support real-world financial applications.
2 BACKGROUND AND RELATED WORK
Financial benchmarks. A variety of financial benchmarks have heen proposed in order to evaluate
the financial knowledge and reasoning abilities of LLMs (Chen et al. [202]] 1: Islam et al., 2023
Xic
et al.) 2024 Guo et al. 2025). For example, FinQA (Chen al. 202 and ConvFin (Chen
https://diae hatp:///eecccouuee2.2 www.wa..ee.oooo edi net fsa go. jp/
2

---

Published as a conference paper at ICLR 2026
et al. 2022b are closed numerical reasoning question answering tasks related to financial analysis. Similarly. FinanceBench (Islam et al. (023) is an open-book financial question answering benchmark
dataset, and FinBen (Xie et al. [024) is benchmark-suit that contains 24 tasks. FAMMA (Xue et al.,
2025) uses university textbooks and the CFA exam to construct a multimodal question answering
benchmark. Several benchmarks exist in the Japanese financial domain (see Nakagawa et al.) (2025) for a recent survey), including Japanese-Im-fin-ueee
(Hirano) 2024) and JCPA dataset (Masuda ct al. 2023), which focus on Japanese financial exam questions. Recently, benchmarks such as FinanceMATH (Zhao et al. 2024), which evaluates an LLM's financial knowledge together with its mathematical reasoning, and AuditBench (Wang et al., 2024), which identifies subtle discrepancies between transaction data and financial statements, have been introduced. Despite these advances, existing benchmarks still center on tasks that can be solved with basic financial knowledge and relatively simple computations. As summarized in Table [ most existing benchmarks rely on relatively simple QA settings that do not require expert-level reasoning. In contrast, EDINET-Bench poses more complex challenge: the inputs consist of entire financial reports, and solving tasks such as accounting fraud detection requires integrating information across multiple tables and textual sections, demanding expert-level reasoning.
Accounting fraud detection. Accounting fraud detection involves identifying deliberate misstate- ments, manipulations, or discrepancies in financial reports. These reports are critical for numerous stakcholders, as investors base allocation decisions them, while job seekers evaluate potential employers through their financial disclosures (Soul 2018). Despite preliminary audits, fraudulent activities are frequently discovered only after publication, necessitating subsequent corrections (Japan Institute of Certified Public Accountants| 2024). Accounting fraud detection has been studied for many years, with pioneering work by [Beneish| (1999). After that, traditional machine learning approaches have been applied to accounting fraud detection (Perols) 2011] Dechow et al., 2011 Song et al. [016 West & Bhattacharyal 2016; Kondo et al., 2019), but effectively processing textual information alongside numerical data in annual reports using LLMs remains underexplored.
Earnings forecasting. While carnings forecasting typically involves predicting the numeric mag- nitude of future earnings (Penman & Sougiannis 1998 Monahan (018)), large body of work instead focuses on predicting the sign of the change in earnings (Ou & Penman 1989 Chen et al. 2022a), which remains difficult task for professionals. Kim et al. (2024) investigated if GPT4, when provided with financial statements in which company names and fiscal years are anonymized, and without using any textual information, will outperform human analysts in predicting the direction of earnings for the following year. However, it is based on proprietary data and evaluation code were not made public. On the other hand, we will publish both the evaluation dataset and evaluation code to facilitate future research in this area.
Industry prediction. Industry prediction is a multi-class classification task that aims to predict the industry category based on security reports. While listed firms already are tagged with industry labels, e.g., by Securities Identification Code Committee (SICC), portfolio managers may want to adopt their own industry definitions, e.g., see Kimura & Nakagawa (2022) for a data-driven approach, or anticipate reclassifications as firms' business evolve or mergers reshape their operations. Evaluating whether LLMs can predict industries based on financial information such as balance sheets and profit/loss statements serves as an effective task for measuring LLMS* financial domain knowledge.
Van Der Heijden| 2022); Dolphin et al. (023) used machine learning for this application.
EDINET. Electronic Disclosure for Investors' NETwork (EDINET) is a platform managed by the Financial Services Agency (FSA) of Japan that provides access to disclosure documents such as securities reports. This platform offers both a web interface and an EDINET API to access reports, allowing users to download annual, years. In addition to PDFs, TSV files are also available, containing structured data where each row represents a record of individual attributes that exist in the annual report PDF Detailed information for each disclosure item can be extracted and analyzed by parsing these attributes. EDINET is similar to EDGAR in the United States. We use EDINET as the primary data source for our benchmark.
quarterly, and amended reports for the past ten
https:/ http:s///wwwww..c.o wttp://ccc.gooeeearr/eeeoooe /tt.a///aa//////rrrrr.rrr//////r
3

---

Published as a conference paper at ICLR 2026
Table 2: Number of annual reports per fiscal year.
IT & Services
1634
Retail
409
Trading
381
Construction
363
Start Fiscal Year
Reports
Electronics
334
Materials
315
2014
3,638
Machinery
248
2015
4,047
Industry
Logistics
193
2016
4,065
Real Estate
182
2017
4,111
Food
156
2018
4.120
Finance
154
Banks
129
2019
4.146
Auto
119
2020
4,206
Pharma
93
2021
4,242
Steel/Metal
166
2022
4,263
Energy 154
2023
4,267
500
1000 Count
1500
2024
586
Total
41,691
Figure 2: Number of companies per industry.
CONSTRUCTION OF EDINET-BENCH
In this section, describe the construction method of EDINET-Bench. Overview of our construction pipeline is shown in Figure Generally, EDINET-Bentd Bench is constructed by downloading Japanese listed companies' annual reports from EDINET and assigning labels for each task.
3.1 EDINET2DATASER
We first create edinet2dataset, tool for downloading and parsing financial documents from EDINET. This tool primarily includes a download function for securities reports using the EDINET API and parsing function to extract financial information from the downloaded report files in TSV format. The parsing process leverages Polars (Vink) 202T) to enable high-speed processing. edinet2dataset is inspired by the edgar-crawler from the Edger-Corpus (Loukas et al. (02T)) and shares similar functionality.. However, edinet2dataset is capable of extracting more detailed information. such as the current year's is sales. The information extracted by edinet2dataset can be broadly categorized as follows: Meta: Metadata such as company name and EDINET code. Summary: Key financial indicators. BS: Balance sheet statement. PL: Profit and loss statement. CF: Cash flow statement. Text: Other textual information in the report, such as company history and business risk explanations.
3.2 EDINET-CORPUS
By using edinet2dataset, we collect available annual reports and amended annual reports from EDINET for the period from April 2014 to April 2025, which includes approximately 40,000 documents (4,000 listed companies X years). The number of annual reports obtained from EDINET over the past ten years for each year, as well as the number of companies per industry, are We call this dataset the EDINET-Corpin and it will be publicly available for reproducibility. The EDINET-Corpan serves the primary data source for constructing EDINET-Bench, as described the next section.
shown in Table 2 and Figure 2
3.3
EDINET-BENCH
Using the aforementioned tool and corpus, we construct three challenging financial benchmark tasks: accounting fraud detection, caminns forecasting, and industry prediction.
Accounting fraud detection. We construct a binary classification dataset for accounting fraud detection. We collect both fraudulent and non-fraudulent reports in order to build this dataset. For the fraudulent reports, we first downloaded the past ten years of amended annual reports from EDINET, obtaining total of 6. 7.212 reports. An amended annual report document that discloses amendments to the original annual report. These amendments can related to fraudulent activities con-fraudulent issues, such misreporting the ownership percentage and share count of major
4

---

before January 1, 2020, it is included in Published as a conference paper at ICLR 2026
Table Class distribution in industry prediction task (test split).
Table 3: Class distribution in accounting fraud detection task.
Industry
Count
Banking
Split
Non-fraud
Fraud
Total
Electronics & Precision Instruments
34
Automobiles & Transportation
34
Train
453
412
865
Transportation & Logistics
34
Test 102
122
224
Electricity, Gas Energy Resources
Real Estate
33
Machinery
33
Table 4: Class distribution in carnings forecasting task.
Steel & Nonferrous Metals
33
Materials & Chemicals
32
Finance (Excluding Banking)
Food
31
Split
Decrease
Increase
Total
Construction & Materials
29
Trading Companies & Wholesale
28
Train
254
295
549
Information & Communication Services
Test
157
294
451
Pharmaceuticals
24
Retail
23
Total
476
shareholders, under-reporting executive compensation, or failing to disclose critical information. To identify amendments linked to fraudulent activities, we extract text from amended amnual reports in PDF format using pdfminer (Shinyama] 2016). specialized tool for PDF text extraction. Notably, amended reports explicitly state the reasons for corrections text form, which provides direct evidence of irregularities. Leveraging this property, analyze the extracted text with Claude 3.7 Sonnet, prompting the model to assess whether the stated reasons indicate fraud. This enables reliable and large-scale scale detection of fraudulent cases. The specific prompt used for this classification detailed in Appendix B
As a result, 668 reports were identified as fraudulent and are labeled fraudulent. We manually reviewed all cases labeled as fraudulent by reading the amendment reasons in the reports. Fewer than 5% of the fraud-labeled cases appeared unrelated (e.g.. board member changes), suggesting low error rate and supporting the reliability of the labels. For the non-fraudulent reports, we randomly sample 700 companies from all annual reports submitted over the past ten years, excluding those identified as fraudulent. From each selected company, we randomly choose one annual report from the reports of the company that have been downloaded. We then split the fraudulent and non- con-fraudulent samples into training and test sets, ensuring that data from the same company does not overlap between the two sets. This resulted in dataset with 865 samples for training and 224 samples for testing, totaling 1,089 instances. Due to document parsing errors, the dataset was reduced 534 fraudulent and 555 non-fraudulent samples, smaller than the initial 668 and 700 samples, respectively. The class distribution of the dataset is shown in Table
Earnings forecasting. We construct a binary classification dataset for earnings forecasting to predict whether company's earnings will increase or decrease in the following fiscal year, given its current annual report") To create this dataset, company, generate pairs of consecutive annual reports spanning two fiscal years, such as (2015, 2016), (2016, 2017), .... (2023, 2024). From these pairs, we randomly choose one pair for each company to include in the dataset (e.g., (2016, 2017)). For each sampled pair, extract the value of "Profit Attributable to Owners of the Parent Company from the corresponding TSV files. We then compare the profit value between the two years: if the current year's 's profit exceeded the previous year's s, the instance is labeled as an increase; otherwise, it is labeled as decrease. Additionally, to evaluate baseline performance, we collect labels for earnings changes between as a naive baseline model. The dataset is split into train and test sets based on the fiscal year of the previous report in each pair. If the fiscal year started
randomly select 1,000 companies. For each
and one year prior
Note that with edinet2dataset, is relatively easy to modify the task depending the user's needs, e.g., change a regression task or use other metrics.
"This value represents the portion of the consolidated net profit that is attributable the sharcholders of the parent company.
5

---

Published as a conference paper at ICLR 2026
the train set; otherwise, it is assigned to the test set. This resulted in 549 samples for training and 451 for testing, totaling 1,000 instances. The class distribution for each split is shown in Table
Industry prediction. We construct a multi-class classification dataset for industry prediction to predict a company's 's industry based on its annual report. We use the industry information provided by EDINET for each company. The industry definitions follow the classification established by the SICC, which categorizes listed companies into 33 distinct industries. This classification, known as TOPIX-33, widely recognized among Japanese investors. However, the granularity of 33 industries is considered too fine for our purposes. To enhance interpretability and simplify the prediction task, we consolidate similar industries into 16 broader c This merging process guided by the Japan Standard Industrial Classification published by the Ministry of Internal Affairs and Communications (MIC) and the SICC's official guidelines. We categorize all companies available in the EDINET-Corpus into 16 industry groups and randomly sample approximately 35 companies from each category. For each sampled company, the latest fiscal year's 's securities report is used as input for the industry prediction task. This process yields test split of 496 samples with the class distribution shown in Table 5 In addition, we prepare a train split consisting of 900 examples.
4 EVALUATION
We evaluate the performance of current state-of the-art art LLMs in zero-shot setting, and compare them to classical baselines trained on the train split.
4.1 EVALUATION SETUP
Models. We evaluate several LLMs in a zero-shot setting. For closed-source mod- els, we include GPT-4o 200t-40-2024-1-222 04-mini (o1-miin--2025-04-11)) GPT-5 (gpt-5-2025-08-
Claude 3.5 Haiku
Claude 3.5 Sonnet
and Claude 3.7 Sonnet
(claude-33 3oiiee-37--ooneee 2025c222922299999 (OpenAl 2024, Anthropic/ 2024). For open-source models, we evaluate Kimi-K2 K2, DeepSeek- V3, DeepSeck-RI, and Llama 3.3 70B (Kimi Team
2023 DeepSeek-AI] 2025b Llama Team 2024). In addition, we evaluate Llama-3.2-18 model
fine-tuned on the train split. For classical models, we use Logistic Regression, Random Forest, and
XGBoost (Cox 1958 Breiman 2001: Chen & Guestrin 2016).
Sampling parameters. For inference, the temperature parameter is set to 0.0 and the maximum token length for generation is set to 4096. For GPT-4o, the random seed is fixed at O.
Prompt. We use the system prompt "You are financial analyst" across all tasks. For binary classification tasks (accounting fraud detection and earnings forecasting), models are prompted to output: (1) prediction (0 1), (2) a probability value (0 to I), and (3) reasoning based on securities report information. For multi-class industry prediction, models output the predicted industry class and classification reasoning. The fraud detection prompt is shown in Figure 3 with other task prompts provided in Appendix
Input formats. We extract the following items from the reports for prediction: Balance Sheet (BS), Cash Flow (CF), Profit and Loss (PL), Consolidated Summary (Summary), and Text Block (Text). These items are concatenated, appended to the end of the instruction prompt, and fed into the models in zero-shot manner. To assess how information quantity affects LLM performance, we tested three configurations: Summary only, BS+CF+PL-Summary, and all items CL--C--L--Summr---Teee Note that BS, CF, PL, and Summary consist of tabular data and do not include company names or exact year information, whereas Text contains unstructured textual information.
Cost and processing time analysis. Each annual report in EDINET-Bent Bench contains approximately 30,000 tokens spanning all sections. The average output length around 500 tokens. For Claude 3.7
In TOPIX- 17, the number of cases in the industries of "Energy Resources" and "Electricity & Gas." which are loosely related domains, was smaller compared to other industries. Therefore, these two were merged to create 16 industries.

---

Published as a conference paper at ICLR 2026
Prompt for accounting fraud detection
Please analyze the following information extracted from Japanese company's securities report for any signs of fraudulent activities. Please note that some data may he missing and represented as "_" due to parsing emors. The report has been verified by a certified public accountant, and the numerical values are consistent and correct from calculation perspective. Therefore, please focus your analysis non-numerical inconsistencies logical red flags that could suggest fraud. Respond in the following format: JSON: "jjonn "reasoning". "string", "prob": float (between and probability that the report fraudulent), "prediction"; : (0: No fraud, I: fraud) } "" The extracted securities report is as follows:
Figure 3: Prompt for accounting fraud detection.
Sonnet, which costs $3 million tokens for input and $15 per million tokens for output, the cost. per report is approximately $0.1. Regarding processing time, Claude 3.7 Sonnet can handle each case in about 10 seconds.
Evaluation metrics. For accounting fraud detection and earnings forecasting, we use ROC-AUC (Receiver Operating Characteristic Area Under the Curve) and MCC (Matthews Correlation Coeffi- cient) as evaluation metrics, and for industry prediction, we use accuracy.
4.2 RESULTS
Table the performance of each model on each task in EDINET-Bench. In the following, we discuss the results for each task.
Accounting fraud detection. In the accounting fraud detection task, the random forest achieves the highest performance. All LLMs struggle, with even the sfate-of-the-ard Claude 3.5 Sonnet only marginally outperforming logistic regression in terms of ROC-AUC, Nevertheless, most models benefite substantially from incorporating textual information, suggesting that narrative disclosures provide useful signals for fraud detection. In some cases, model reasoning explicitly referred to auditor reputation (e.g.. "Their financial statements have been audited by reputable firm, Azusa Audit Corporation") as part of the decision-making process, highlighting a potential bias towards well- known auditing corporations. We further examine the feature importance of the logistic regression model, as presented in Table 7 This reveals that total comprehensive income and total assets influential factors in non-fran con-fraud predictions, suggesting a bias towards predicting larger companies as non-fraud and smaller ones fraud.
Earnings forecasting. For the earnings forecasting task, many LLMs struggle to deliver precise predictions. Even GPT-5 achieves only a ROC-AUC of 0.65, aligning with the low forecasting performance reported by Xie et al.] (2024). This limitation likely stems from the challenge of predicting next-year earnings based solely on the current securities reports. Unlike the fraud detection task, incorporating text information do not enhance performance in carnings forecasting, suggesting weaker reliance textual factors such company size and audit firm reputation.
Industry prediction. In the industry prediction task, all models demonstrate predictive performance significantly higher than random guessing. Furthermore, across all models, expanding the input from Summary to Summary+BS+CF+P results in improved performance, Notably, Claude 3.5 Sonnet achieves state-of-the-art performance with accuracy score of 0.41 when provided with These results indicate that industry prediction is a relatively simpler task compared accounting fraud detection and carnings forecasting, with predictions often possible using only securities reports. This likely reflects differences in asset composition, such as varying proportions of cash, loans, and securities across industries.
Summary+BS+CCR

---

Published as a conference paper at ICLR 2026
Table 6: Performance of each model on each task in EDINET-Bench. For LLMs, the scores represent the mean = standard deviation over three runs. Bold indicates the best score. For industry prediction, the score for input formats that include the text section is omitted, as the text section of annual reports often explicitly states the company's industry, and such cases are indicated as "_". denotes models that are trained on the train split.
Fraud Detection
Earnings Forecasting
Industry Prediction
Model
Input Setup
ROC-AUC MCC
ROC-AUC
MCC
Accuracy
Closed-source
Summary
0.61 0.01
0.19 = 0.02
0.41 + 0.01
-002 : 0.02
0.09 0.00
Claude 3.5 Haiku
Summary+B++CF4P
0.60 +
0.45
0.00
0.00 0.05
0.13 0.00
0.67 1 0.00
0.28 = 0.02
0.01 0.44
-0.02 1: 0.01
Summary
0.64 1 0.01
0,05 1: 0.03
0.54
1 0.01
0,08 0.001
0.24 0.01
Claude 3.5 Sonnet
Summary&BS<CF4P
0.63 0.03
0.18
0.55
0:00
0.10 0.02
0.41 0.00
Summary-BS-CF-PLE
0.73
0.32
0.52
0.02
0.08 0.02
Summary
0.59 + 0.01
0.10 = 0.02
0.55
0.01
0.06 0.02
0.24 0.01
Claude 3.7 Somnet
Summary+BS+CT-FPL
0.58 1: 0,02
0.09
0.04
0.58
0.01
0.13
0.01
0.39 0.01
Summary-BS-CFFPP
0.67 +: 0.01
0,25 = 0.02
0.61
+ 0.01
0.16 0.02
Summury
0.59 1: 0.00
0.16 + 0.02
0.40
0.00
-0.17 1: 0.00
0.14 1: 0.01
GPT-4o
Summar+BS+CF4PL
0.61 + 0,01
0.19 = 0.02
0.41
0.00
-0.13
0.19 0.01
Bammaayy-BS-FFFPP--TTa
0.69 1: 0.01
0.29
0.02
0.00 0.41
-0.14 0.00
Summary
0.53
0.08
0.51
0.00
0.05 0.02
0.19 0.01
o4-mini
Summary4BS+CI4F
0.52 1: 0.01
0.04 = 0.05
0.54
1:00
0.13 + 0.04
0.27 1 0.01
Summary-BS--C BS-CF---+-CFFP
0.61 + 0.01
0.10 = 0.05
0.58 + 0.01
0.15 0.01
Summary
0.56 0.00
0.00 = 0.00
0.58 0:00
0.09 0.02
0.21 0.01
GPT-5
Summary+B5+CF-PL
0.62 +
0.62 0.01
0.20 0.02
0,31 0.01
0.67 + 0.01 0.07 = 0.04
0.65 0.01
0.21 + 0.01
Open-weight models
Summary
+:
0.46 0.00
.0.06 0.00
0.14 0.01
Kimi-K2
Summary+BS+CF+P
0.54 + 0.02
0.08 = 0.03
0.53 - 0.00
0.04 =
0.26 2:6==0 0.01
Summay--S--CF-RL BS-CC++L+TT Text
0.59 + 0.00
0.11 = 0.05
0.55
0.02
0.07 0.07
Summury
0.61 1: 0.03
0.21 = 0.02
0.39 1: 0.01
-0.14 1: 0.02
0.12 :: 0.00
DeepSeek-VV V3
Gummary+BS+CT-4
1:
-
1.40::000
-0.13 0.01
0.15 0.01
Summary- PS-CFFPLL
0.55 0.03
0.07 = 0.06
0.39 0.01
-0.15 0.02
Summary
0.54 1: 0.04
0.01 1: 0.08
0.43
0.01
-0.06 0.01
0.11 0.02
DeepSeck-RR R1
Summary+BS+CII
0.56 1: 0,01
0.09 = 0.02
0.46 1 0.00
-0.01 +: 0.01
0.16 :: 0.01
Summary-BBS-CF-ae BS-CF-PL-+Ted
1:
0.01
Summary
0.58 + 0.01
0.11 = 0.01
0.42 0.01
-0.99 0.03
0.10 0.01
Llama 70B
Summary+BS+CTFPP
0.59 1 0.00
0.11 1: 0.01
0.41
0.01
.0.11 0.02
0.14 0.01
Summary-BB- BS-CF+PL
0.56 + 0.04
0.06 = 0.06
0.42 : 0.01
-0.04 + 0.01
Summary
0.61 + 0.01
0,18 = 0.05
0,52 + 0.01
0.03 0.03
0,25 0.01
Llama-3.2-18 SFT!
Summay+BS+CIIPL
0.66 + 0.02
0.17 0.01
0.52 0.01
0.04 0.02
0.26 0.02
Summary BSS
0.54 1 0.02
0.09 0.05
0.00 0.48
0.02 0.01
Classical models
Summary
0.68
0.17
0.56
0.05
0.27
Logistic
Summary+BS+CF-PP
0.59
0.08
-0.01
Summary Somnaary-B--CCC BS-CF+PLTTe
0.57
0.06
0.51
0.03
Summary
0.75
0.38
0.54
0.09
0.35
Random Forest
Summary+BS+CCT
0.75
0.38
0.53
0.01
0.40
Summary-BS BB-CN-------FF---T
0.73
0.31
0.52
0.03
Summary
XGBoost
Summary-B++TT-PL
0.63
0.27
0.53
0.06
0.42
0.61
0.24
0.53
0.05
CONTAMINATION
Since the annual reports EDINET- Bench are publicly available online, there potential risk of test set contamination (Oren et al.) 2024 Golchin & Surdeanu 2024). Several methods have been proposed to detect test set contamination, including prompting models to reproduce verbatim examples from the test set (Golchin & Surdeanu, 2024) and perplexity-bate based approaches (Mattern ict al. 2023 Chan et al. 2025). However, generating verbatim examples is not suitable for assessing whether model has memorized the correspondence between reports and their labels, and perplexity
8

---

Published as a conference paper at ICLR 2026
Table Feature importance of logistic regression on accounting fraud detection, sorted by absolute value of feature importance scores.
Feature
Importance
Abs Importance
Comprehensive Income (CurrentYcar)
-1.165
1.165
Total Assets (Prior4Yess
-1.118
1.118
Total Assets (Prior3Yea)
-1.073
1.073
Comprehensive Income Yerior3Year)
1.066
1.066
Price-to-Farming Ratio (Prior2Ycar)
0.019
0.019
Price-to-Eane Eo---iannng Ratio Year)
-0.014
0.014
Return on Equity (Prior Year) Frice-to-Earning Ratio (Priorl Year)
-0.008
0.008
0.001
0.001
Table 8: Performance of each model on company name prediction. BS+CT+PL-SSuman is used as input. The comparison between the ground truth company names and the predicted names is conducted using GPT-4o judge to account for inconsistencies in name representations.
Claude 3.5 Haiku Claude 3.5 Sonnet Claude 3.7 Sonnet DeepSeck-V. v3 DeepSeck-RI GPT-4o 04-mini
Acc.
0.045
0,050
0.045
0.000
0,005
0.000
0.005
cannot be computed for closed-source models. To evaluate the potential impact of contamination, we instead focus on company name prediction and year-wise performance analysis.
Company name prediction. To confirm whether the models already knew the content of the securities reports used in the evaluation dataset, we measure the performance of task where the model is given tabular data (BS, CF, PL, Summary) from the test split of the accounting fraud detection dataset and asked predict the company names. The prompt is shown in Figure 7 For evaluation, we accepte minor variations in company names, such as the inclusion or omission of terms like 'Co., Ltd.", using the LLM--asa-juudee approach (Zheng et al., (023). Table shows the result. For all models, the accuracy 0.05 or lower, indicating the difficulty for models to associate tabular data with company names.
shows
Year-wise performance analysis. To check for potential test set contamination, we analyzed performance by fiscal year on the accounting fraud detection task. Since models may have seen older data during training due their knowledge cutoff dates, contamination would manifest as higher performance on older years compared to recent years. We split the test set by fiscal start year and plotted yearly performance in Figure The results show declining trend for recent years, indicating that test set contamination is unlikely to be affecting our evaluation.
Although the above results suggest that contamination has only limited impact the experiments, we cannot conclude that the dataset is entirely free of contamination. Achieving a fully contamination- free evaluation would require using annual reports released after the model's 's cutoff date. This is possible with our toolkit, which iteratively updates datasets with the latest annual reports.
CONCLUSION
In this work, we introduced EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate the performance of LLMSs on challenging financial tasks including accounting fraud detection, earnings forecasting, and industry prediction. We leveraged Japan's 's Electronic Disclosure for Investors' NETwork (EDINET) to construct EDINET-Bench and can easily incorporate future reports using our toolkit, edinet2dataset. Our experiments reveal that even state-of-the-an LLMs perform only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting. This suggests that simply providing reports to LLMs in straightforward setting insufficient. The results underscore the need for benchmark frameworks that better emulate the environments in which financial professionals operate, incorporating richer scaffolding, realistic

---

Published as a conference paper at ICLR 2026
ROC AUC
MCC
RDC AUC
NCC
0.6
0.4
Score
6.0
(eccal
Year
(a) GPT 40
(b) Claude 3.5 Sonnet
Figure 4: Performance per fiscal first year accounting fraud detection.
simulations, and task-specific reasoning support to enable more effective problem solving. We hope our benchmark and toolkit will accelerate advancements in applying large language models to complex financial analysis tasks.
FUTURE WORK
Building on this study, several promising directions remain for future exploration.
Multilingual dataset. EDINET-Bench is constructed from Japanese annual reports. Extending the benchmark to multiple languages would allow for cross-lingual evaluation of financial document understanding and better reflect the global nature of financial markets. For instance, the U.S. EDGAR system provides functionality analogous to EDINET, suggesting that a comparable dataset construction pipeline could be applied to build multilingual extension of the benchmark.
Agentic framework. In this work, we evaluated LLMs a simple setting where only a single annual report was provided as input. In practice, however, financial professionals rely on much broader range of information, such as internal records including receipts and meeting minutes, reports from peer companies, and even real-time economic news. Future research could therefore explore benchmark task designs that enable models to incorporate information beyond the annual report, for example by leveraging agentic capabilities such web browsing for up-to-date financial news, local file access for company records, or a Python execution environment for quantitative analysis (Wang al. 2025 Yang et al., (2024). In addition, rather than restricting evaluations binary classification, benchmarks could adopt rubric-based evaluation, where model outputs are assessed according to
multiple criteria tailored to each specific question (Starace al. 2025 Arora al. 2025).
ETHICS STATEMENT
We constructed EDINET- Bench using publicly available data from EDINET, which contains private information, ensuring compliance with privacy regulations. By releasing EDINET-Bene Bench and edinet2dataset. aim contribute significantly the development of LLMs in real-world financial domain applications. However, since the data represents real companies, there is a risk of misuse, such damaging company's reputation. This benchmark dataset is intended purely for advancing LLM applications in practical tasks and should never be used to target or harm actual companies. There also a possibility that this dataset could be misused make fraud more difficult for LLMs detect. We strongly discourage any such attempts, especially in real-world applications.
REPRODUCIBILITY STATEMENT
In this work, we constructed a benchmark dataset and evaluate models using it. To ensure repro- ducibility and facilitate further research, publicly release both the dataset and the code used for its construction, allowing others to rebuild and directly use the dataset. For the evaluation, we also release the evaluation code that the reported results can be fully reproduced.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like to thank Ryuichi Maruyama, Shimpei Fukagai, Shengran Hu, and Robert Tjarko Lange for helpful discussions.
REFERENCES
Anthropic. Claude 3.5 sonnet, 2024. URL http:://ww..nttrroocc...cooo/e claude- 3-5- sonnett Accessed: 2026-02-20.
Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quinonero-Candela. Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025. URL nittps://arxi... 2505. 08775
Messod D. Beneish. The detection of earnings manipulation. Financial Analysts Jourmal, 55(5): 24-36, 1999. doi: 10.2469ffaj..55..552222 URL http:////6oooooo.0.22//////////
n5. 2296
Leo Breiman. Random Forests. Mach. Learn., 45(1):5-32, 2001. ISSN 0885-6125. doi: 10.1023/A: 1010933404324.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLE- bench: Evaluating machine learning agents on machine learning engineering. ICLR, 2025. URL httpps://ppenrete net hwtp:/nnetooorrteeeeeee
Forum? 7 F
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. SIGKDD, 2016.
ISBN 978145034232.. doi: 10. 00.11455/93967222999773
7939672.2939753
Xi Chen, Yang Ha Cho, Yiwei Dou, and Baruch Lev. Predicting future carnings changes using machine learning and detailed financial data. Journal of Accounting Research, 60(2):467-515, 2022a.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Han Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In EMNLP, 2021. URL https://
aclanthology. wwt. emmlp main.330
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. In EMNLP, 2022b.
4217
Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench verified, 2024. URL https: htttps://openas //tp///ppaaa..//////////// wwt.ccoominnn.e itteeeiiiittrrte induuint swe bww.ceehccheeeeeeeeeeeeeeeeeeeee verified: Accessed: 2026-02-20.
David R Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statistical Methodology, 20(2):215-232, 1958.
Patricia M Dechow, Weili Ge, Chad R Larson, and Richard G Sloan. Predicting material accounting misstatements. Contemporary accounting research, 28(1 28(1): 201
DeepSeek-AL DeepSeek-Rl R1 incentivizes reasoning in Ilms through reinforcement learning. Nature, 645 1))633 638, 2025a. ISSN 1476-4687, doi: 10.0099/441586-0255-0092.. URL http: 025 09422-2
7/dx. doi www.doo...rrg...oo 1088

---

Published as a conference paper at ICLR 2026
DeepSeek-AL DeepSeek- V3 technical report. arXiv preprint arXiv:2412. 19437, 2025b. URL https: http://www....a/////////.2.2....
Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj. Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang. Noah Jacobson, Bing Liu, and Brad Kenstler. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?, 2025. URL https: /ttpai///arxvvvvve httpp://aax....rgg//aa.... 2509 16941
/
Rian Dolphin, Barry Smyth, and Ruihai Dong- A machine learning approach to industry classification in financial markets. In Artificial Intelligence and Cognitive Science, pp. 81-94. Springer Nature Switzerland, 2023. ISBN 178-3-031-26438-2
Shahriar Golchin and Mihai Surdeanu. Time travel in LLMs: Tracing data contamination in large language models. In ICLR, 2024. URL httpS///ppenreeveew.eeee//eermr
2Rwo6c3tVv
Google. Our next-generation model: Gemini 1.5, 2024. URL httpp::///lo...0/00
Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang. Zhiqiang Liu, Sizhe Wang. Jinyi Niu, Chuqi Wang. Yanhui Wang. Xiaolong Liang. Xiaoming Huang, Bing Zhu, Zhongyu Wei. Yun Chen, Weining Shen, and Liwen Zhang. FinEval: A Chinese financial domain knowledge evaluation benchmark for large language models. In NAACI, 2025. ISBN 979-8-89176-188.6 ogy 2025 naac long 318/
URL
Masanori Hirano. Construction of Japanese Financial Benchmark for Large Language Models. In Joint Workshop FinNLP, KDF, and ECONLP, pp. 1-9, 2024. doi: 0t113//ssm.47799222 URL
finn1p-1.1
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Fi- nanceBench: A new benchmark for financial question answering, arXiv preprint arXiv:2311.1199 2023. URL URL UL/tttss//axii...rr/////1/1111 11944
Japan Institute of Certified Public Accountants. Trends in accounting fraud among listed companies, 2024.
URL
jp/specialized httpeicialieeeeieeeeeeeeeeeeee Accessed: 2026-02-20.
2-3- 10000-22-2407-200 pdf
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In ICLR, 2024.
URL htt.:httsss//ppeeeeveee
net nwt/fooummmm ddvTF & NQM66
Alex Kim, Maximilian Muhn, and Valeri Nikolaev. Financial statement analysis with large language models. arXiv preprint arXiv:2407.17866
2024. URL https::/aaxiv..rrttt Paper withdrawn by authors on February 20, 2025.
17866
Kimi Team. Kimi K2: Open agentic intelligence, arXiv preprint arXiv:2507.20534 2025. URL
https://arxiv org/ abts 20534
Yuya Kimura and Kei Nakagawa. Industry momentum strategy based on text mining the japanese stock market. In IIAI-AAI, pp. 420-423, 2022. doi: A.I10/IIAAAA1531122222.000
Satoshi Kondo, Daisuke Miyakawa, Kengo Shiraki, Miki Suga, and Teppei Usuki. Using machine learning detect and forecast accounting fraud, 2019.
Jean Lee, Nicholas Stevens, and Soyeon Caren Han. Large language models in finance (finllms).
Neural Computing and Applicaations, 2025. ISSN 1433-3058. doi: -20077/000021--4-040055 URL http: doi httt:::/dx..di.org///D.... 1007 00000//0002200 100...00000011111000000000100-00
Llama Team. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https:
//axxiv.o //aaxiv..rg/aaabbiia wtw.v.orrg//bs/2200..2222
12

---

Published as a conference paper at ICLR 2026
Lefteris Loukas, Manos Fergadiotis, Ion Androutsopoulos, and Prodromos Malakasiotis. EDGAR-
CORPUS: Billions of tokens make the world go round. In ECONLP, 2021. URL https
//aclanchology.orgg wwt.hhowwooo....///oooo/o 2021. econ1p-1.27
Tatsuki Masuda, Kei Nakagawa, and Takahiro Hoshino. Can ChatGPT pass the JCPA exam? Challenge for the short-answer method test on auditing. In JSAI, 2023.
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Kirk-Kiikpatrick. Membership inference attacks against language models via neighbourhood comparison. In ACL, 2023. URL httppss/acclatihooooo.ooo///////
Findings-acl. 7197|
Steven J. Monahan. Financial Statement Analysis and Famings Forecasting. volume 12, pp. 105-215. 2018. doi: 00.1561/1400000336 URL httpt:ttt///eeeee httppp/..r///o///////eo/t/////ae http://oowfttcc/c/c//c//// 1400000036.h2 h
Kei Nakagawa, Masanori Hirano, and Kaito Takano. Survey and perspectives large language
models in Japanese financial sector. 2025. URL http:t:s/////i..ss....o
php/
jxiv/preprint/vveww 12668
OpenAl. GPT-4 Technical Report. arXiv preprint arXiv:2303.00777, URL httph:t/ttxiiiiii
org/abs/2300 2203/333..000 08774
OpenAl. Introducing o3 and 04-mini, 2025. introducing and 04 miini/
URL https::/openai.com/
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto.
Proving test set contamination black-box language models. In ICLR, 2024. URL https
//openeeiiew net fow.mmid---uannaiiieair
Jane A Ou and Stephen H Penman. Financial statement analysis and the prediction of stock returns.
Journal of accounting and economics, 11(4):295-329, 1989.
Stephen H Penman and Theodore Sougiannis. A comparison of dividend, cash flow, and earnings
approaches equity valuation. Contemporary accounting research, 15(3):343-383 1998.
Johan Perols. Financial statement fraud detection: An analysis of statistical and machine learning algorithms. Auditing: A Journal of Practice & Theory, 30(2):19-50, 2011.
Long Phan, Alice Gatti, Ziwen Han, Nathamiel Li, et al. Humamity's ' Last Exam. arXiv preprint (5fi::2501.12449 2025. 1tthhttt.e///arrr...rr/////////.
Yusuke Shinyama, pdfminer. htttesss/pppo..oor///////////t/p 2026-02-20.
2016. Accessed:
Mingzi Song, Naoto Oshiro, and Akinobu Shuto. Predicting accounting fraud: Evidence from japan. The Japanese Accounting Review, 672016):1-663, 2016.
Akiko Sou. Research on early detection of inappropriate accounting in response recent
changes in economic environment, 2018.
go. httpppp//teeeee ht/////teekkkk
$iiccussion20117 2110011111P111110 6- pdf Accessed: 2026-02-20.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating Al's ability replicate Al research. In ICML, 2025. URL httpps://opareviiie httpeeewwwwwwwweeeeeeeeeeeeeeeee / wow.oormmmddxSPPrPi.....
Hans Van Der Heijden. Predicting industry sectors from financial statements: An illustration of machine learning in accounting research. The British Accounting Review, 54(5): 101099, 2022.
Ritchie Vink. Polars. httpss///pppl..rr///r//tttttt/ot
2021. Accessed: 2026-02-20.
Rushi Wang, Jiateng Liu, Weijie Zhao, Shenglan Li, and Denghui Zhang. Automating financial statement audits with large language models. In 2nd Al4Research Workshop, 2024. hthpss
7/operrevi ew net//forumm MM
13

---

Published as a conference paper at ICLR 2026
Xingyao Wang. Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for Al software developers generalist agents. In ICLR, 2025.
/ttp///pp//vviieeeeeee ew net. forum?
id-OJdBayDDoF
Jarrod West and Maumita Bhattacharya. Intelligent financial fraud detection: A comprehensive review. Computers Security, 57:47-66, 2016. ISSN 0167-4048. doi: https://doi.or///0.0066
j.cose.015509.0005 URL https://www wttp://weweeeeeeee...oooooo////e
011/0167704411001025
Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang. Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang. Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, Jiajia Huang. Xiao Yang Liu, Alejandro Lopez-Lira, Benyou Wang, Yanzhao Lai, Hao Wang, Min Peng, Sophia Ananiadou, and Jimin Huang. FinBen: A holistic financial benchmark for large language models. In NeurIPS, 2024. wtw.:eeeedinnns..eeiiieeeeee wwwrieeurp.... oc wttp:ccppppppeeeeeee files pp2/p/////////1le Datasets and Benchmarks
URL
adb 1o01118888666776220033322bbb-323
Track. pdf
Siqiao Xue, Xiaojing Li, Fan Zhou, Qingyang Dai, Zhixuan Chu, and Hongyuan Mei. FAMMA: A benchmark for financial domain multilingual multimodal question answering, arXiv preprint arXiv:2410.04526, 2025. URLpattss://axxi wwt.:/wwwww..g.r/////...aa
04526
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In NeurIPS, 2024. URL M
net. wtt.:eeoooommmdddddooooooooooeo.
Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. FinanceMATH: Knowledge Knowledge-intensivy intensive math reasoning in finance domains. ACL, 2024. doi: 10.2844 o2g/2024 acl -oong. 693/
acl-long.693. URL
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2023. URL httpe:://proceddingg...errrrrrrp wttp:eurrrs.cccpppppppeeeeee wwwppppperrfeeppppeppeppeeeeeeee http:///pppppeille////rrrr/////p http:/paap//22//1111// and Benchmarks.
pdf
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tranyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In ICLR, 2024. URL net/foorm?idd=oke Lc
A
INSTRUCTION PROMPT FOR EACH TASK
Figures 5 and 6 show the prompts used in the evaluation of the respective tasks.
Prompt for earnings forecasting
Please whether the "g 42 & R) (Net incame attrihutable
parent) in fiscal year's socurities
report will incresse compured to the current fiscal year. based on the informatio available the current year's securities report. The input extracted from a Jepunese company's is securisses report Some information may be missing and represented is "L" due punsig ernors. Some attributes are missing und the total does not probsiiility profit incresse), "preictioon" Decrense Increase) "" current year's extracted securites report follows:
the Respond the Collowing format: JSON:""joo ''reasoning'; "string". "prob": Boat (between 0 and
Figure 5: Prompt for earnings forecasting.
14

---

Published as a conference paper at ICLR 2026
Prompt for industry prediciton
Taased on the following fimncial report, classify the compuny into une of the Japunese industry calegories You must classity the compuny into esaclly one caregories: d" nWt
A z. A-4 4---WWW: U... "dhat 9t6 "Bed (155% "Bx 88.8 "E135 M8386 "Mtdl Ji5t" "BeMe" "Muli fitt"". "Iil btiRf 7 2 D "Mllk 4ot"". "RRAL 30bte" "dlvls" "SHF7" "Gilhh 8842)" ">F""""" "] input extracted from a Japanese company' 's secaritess report. Some information may missing due parsing emun. Respund in the following formar: ISON: "Joon "reasoning": sur ireasening for the classiiiation), "prediction": :: sir (predicsed industry category in Japanese) The cureent year's estracted securities report follows:
Figure 6: Prompt for industry prediciton.
Prompt for company name prediciton
Please predict the name of company of the securilies repor, based on the information availablee in the current year's securities report The input is from Jupanese company's securities repert, Some information be missing represented "." due equal
attributes "json "reassoing" "stringg "peediction" "suring"
missing and
parts, Respond following The current year's extected securilies report follows
Figure 7: Prompt for company name prediciton.
B ACCOUNTING FRAUD DETECTION
Figure 8 shows the prompt used to assess whether amendment reasons are related to accounting fraud violations. Figure and 10 show examples of model responses.
Prompt to detect the reason of the amendment related to accounting fraud
AEB All A
AiE KEE 2 MH 21
2 C
SE S L L CRE &
I 1 C--AABBAkk &EE D Di dit [No] t
(E DJUSONNZ & fi SSONNN
"ioon { "scaccomming frandd boul Irue or false "explanation: str Muil 1ill "company name" str d:
# 5 03f AlE 20 tht Vy ISONIE 18 "jo""junn" CIEA BEA KKRSV TEEii & Au MM0
Figure 8: Prompt to detect if the reason of the amendment is related to accounting fraud.
C EARNINGS FORECASTING
Figure and 12 shows examples of model response.
D INDUSTRY PREDICTION
Table shows the list of the 33 and 17 class labels we use for industry prediction. Figure 13 and 14 show examples of model response.
E LLM USAGE
We used LLMs assist with labeling during the dataset construction for the accounting fraud detection task. Additionally, they were employed to help improve the readability of the manuscript during the writing process.
15

---

Published as a conference paper at ICLR 2026
Table 9: Mapping of SICC 33-sector classification the TOPIX-17 Series.
No. 33 Sectors
TOPIX-17 Series
Fishery, Agriculture & Forestry
Foods
2
Foods
3
Mining
Energy Resources
4
Oil and Coal Products
5
Construction
Construction & Materials
6
Metal Products
7
Glass and Ceramics Products
8
Textiles and Apparels
Raw Materials & Chemicals
9
Pulp and Paper
10
Chemicals
11
Pharmaceutical
Pharmaceutical
12
Rubber Products
Automobiles & Transpoortatinn Equipment
13
Transportation Equipment
14
Iron and Steel
Steel & Nonferrous Metals
15
Nonferrous Metals
16
Machinery
Machinery
17
Electric Appliances
Electric Appliances & Precision Instruments
18
Precision Instruments
19
Other Products
IT Services, Others
20
Information & Communication
21
Services
22
Electric Power and Gas
Electric Power Gas
23
Land Transportation
24
Marine Transportation
Transportation & Logistics
25
Air Transportation
26
Warchousing and Harbor Transportation
27
Wholesale Trade
Commercial Wholesale Trade
28
Retail Trade
Retail Trade
29
Banks
Banks
30
Securities and Commodities Futures
Financials Banks)
31
Insurance
32
Other Financing Business
33
Real Estate
Real Estate
16

---

Published as a conference paper at ICLR 2026
Accounting fraud detection analysis with Claude 3.7 Sonnet
attenion: There's sigmificntt jump parent company net profit (1222 11 I 88R2 5 Siiie fl (2) from 4,431 million yen in the prior year 7,465 million yen the carren year (-68.5%) despiie uperating prodit (St RUF du decreasing from 8,693 million yea 5, 5,392 million yea (:35, 7%6 and ordinary peofit (#f.8 Rld decreusing from 8, 1.5 million yen The sourve this profi
analyzing securities report, I've identified several unusual patterns
6,773 million yen (-16.7%). This
profit puteer requires explanation. million yen to 4,078 million yen. This one-time gain being used to mask deteriorating openational
appears be from estment secuniies
sales (89 (1/55 15 which jumped performaace. 3. Cash nlw patterns show inconsiteencis:: W hile the company reports incecused profiss. openating cash flow decressed from 8,400 million yen to 6,042 million yen. Meanwhill,, invessing cash Bow turned positive frrm 5,073 million yen to +2.25 million yen), primarily due to proceeds from investment securities sales Bt flll it 00:11
B. of 5,551 million yen. 4. The company meations these sales suggests they may be liquddaing ussets to artificially boost
risk
that they sold
anvestment securiaes sccure liquidity profits during a downtum. S. The company delayed ed publishing their medium temm management plan twice, citing COVID- 19 unvertinty. This could be legitimate resson, but it also provides a convenient excuse to avoid making specific commutments that mught be difficult achieve given their operational challenges. The company has high debt ratio 31.9 total these ohereations definitivlly prove frand, they suggest the company may ongaging in cornnns managentent using ase-time gains from asset sales mask deteriorating operational performance. This approoch is unsustainable and rasses concerts about the company" is transpareccy regarding its uue financial condition.
COVID 19 concerns, but the
borrowings), which creates
maintain proficbilitt metrics. While
Figure 9: Accounting fraud detection analysis with Claude 3.7 Sonnet. BS+CF4+L-SSumarr+TTo Ground truth: non-frat non-fraud.
Input:
Accounting fraud detection analysis with GPT-40
The securities report appears to consistent tems of numerical and celcalations, shatel. However, there are several non-mmericaal inconsistencies and logical red flags that could suggest potzsral finddalent octivities or at least wamnet further investiaaion: 1. F'Unsuull Increase in Parent Company Net Income** * The ponent compuny's 's not income ( ('R2 88 1: $9 && 11 incressed significumily compared
the current (F7,468 million)
prior year (V4,43) million), despite dealine in (316 (1A))) openating profi S $ $2") explained by large gain on the sale of lnvestment securitiss (74,078 million), but the reliance on non-cgenating income for peofit growth a lock of sustainable operational performanoe. 2. **iinnfficant Changes in Compechensiee Income**; Comprehersive income (' Ft (E") shows dramoric increase from nese-tero figure the prior yeur (6-60 million) to W18, 543 mi million fhe current year This primarily driven by gains in other such magnitude of the shift warranis serutiny. *aah Flow from Investing Activitios
discrejancy partially
indicate
valuation differences and retiement benelit adjustments, While cush flow from investing activilies tumoel posilive (V2 mi williss) in the currest your. prinurly due the sile of investment socaiiiies This reliance on ssset sales for cash flow could indicate liqudity issues or an allempt buoe, financial metrics artiiiially. 4, e* U>unuul Acconnting Pracices The inberently frandieett,
comprehessive
changes may legitimate, the
menions the of land revalaation' and deferred bedge
accounting. which comples accomning tncatmenss While these mimber temporary employess (11 1009 A has decrossed significantly over year, from 255 the prior four yeur to 215 in the current year, despiie relatively stable tocal employee nimbers This could indicate cost -uttiittinn messures -* "Related Party Transactions* The report mentions significant transsctions and conditions
can he to manipulate fisancial reults not
applied coneenly. S. "Faconsistent Employee Trend
operational downsizing, which are not clearly explained in the affilintes While these disclosed, the
reluted parties, such subsidiaries rumm for potential manipulstim. *CCompics Curporate Sumucume The company has large number of subsidiaries and affiliutes. including many overaens emities. This complesity can make it casier to obsarme financia imqulaiitis. 8. **Unsuual Guins, and I nvves*. The report inclskes significani one-time gains (e investment sccuritiss sales) and losses (o noset empairment, While these be legiiimate, the tiining and magnitude of these would be used to smouth comings meet financial targets. **Lack of Tramsprency in Certain Areas" Some sections the report, details the '2025 Vision' and the use funds from nest sales, lack sufficiont detail to fully sosess their impact on the company" 's financial health. While these issus do not conclusively indicase frand, they highlight areas of concern that should be investigaed further.
detailed,
Figure 10: Accounting fraud detection analysis with GPT-4o. Input: BS+CF++L-Summary-tnn Ground truth: non-fraud.
Earnings forecasting analysis with Claude 3.7 Sonnet
Bused in the financial dan, can observe several positive trends that suggest continued profit growth: 1) The compuny has shown consistent growth in nei income over the past years (160,483,00 176 454451.000
3,002030000- - 285, 490,0000, with significant 31.4% incresse the most
2)
growth the current year. 3) Oyenting profit incressed by 17.6% (327 476 385 065 (00)) and ordinury prodit incressed by 20.0% (330,591,000 - 397,228.000 20,0000 the current yeur. 4) The company has a healthy finuncial puditim with incressing net assets (1.974, 469 .000 - 2,335,901 (00) and a strong eash position (2. (2.04,7200.0 240.720.000) 5) The company is in the IT soluions industyy which confinues to sce demiand for digital tranffemmtion, cload services, and other technology solutions. 6) The company has been expunding its workforce (605
have heen steadily incressing,
previous years' higher growth). Operating eash decressed signi ficuntly (273,903 000 - 157.958.000 9500000 The company mentions potential impacis from COVID-19 the next discal year's is performance. 4) There are increasing investments and expenses related to business expansion. Despite these concerns, the overall trend profiability improvement, the 's strong fimaniil position, and the continued demand the IT industry sugpest that profi growth is likely continue, albeit possibly
637 employees) to suppun growth. However, there some potential concerne:
growth rate sales slowed ()
moderste pace than the current year's exceptional 31.4% growth.
Figure 11: Earnings forecasting analysis with Claude 3.7 Sonnet. Input: BS+CC-PL--Summary--lcc Ground truth: Increase.
17

---

Published as a conference paper at ICLR 2026
Earnings forecasting analysis with GPT-4o
The company's financaal perfomumce shown concistent growth key metrics over the past yeirs, Revenue C'
incressed steadily
1,961,794,000 yen three years ago to 6,233,729,00 000 yen in the current year. Openaing profit ( X (t)) and ordinary profit ('86% fl have also significuntly with onlinury profit from 241, 174,000 ihree
to 267.228.000 000 160,483000 ven three years ago
the current
Net incume altributable
ownees parest Bl current year. Additionally. the company bas a strong finuncial positio. with increusing net assets ( "t.kkk))))) and high eqult raio &t While there are some risks, such potential eomomic flnctaations und challenges in securing humam resources, the compony i stralegic focus on IT solutions, including emerging technologies like and cloud services, posiions well comiiued growth. Based on these factors, highly probable parent increase nest liscal year.
similarly incressed
yen
the
Figure 12: Earnings forecasting analysis with GPT-4o. Input: BS+CF-PL-Summar--Ter Ground truth: Increase.
Industry prediction analysis with Claude 3.7 Sonnet
Hased the financial datn, this appears finaccial services company, specifically act bank Key indicatars inclade: 1) Vary large total neeets (3.39 tillion yen) compared
large sheet-err berowings (708.5 billion yeui,
capital intensive financial hillion yen), band issunces (393.5 hillion yon). Negative opensting
Significaat borrowings lending
hillion yen), loep-term beerowings cash flow (-112.1 billion yeat with postive finascing eash flow (131.8 billion yes). typical of lending businesses. 4) Low self-capital msio (12.795),
common financial institutions. 5) The balance sheet stracture shows chameterisicc of financial services related
with lirge assets and lisbs lifies financial
non-bank activities. The company appears involved lending leosing operations
making likely
Figure 13: Industry prediction analysis with Claude 3.7 Sonnet. Input: BS+CF+PL+Summary Ground truth: non-bank financial institution.
Industry prediction analysis with GPT-4o
The financial repoct indicates company with signifleant revense und profit figures, as well as substantial neeets and liabilitiss The company also has high level of cash flow sctivity, inclading barowing and bend issuance, which charcctristic of trading company. Addinonally, presence supports classification trading company. These chometeristis
securities acrivitios related dividends interest align Iiitt WDG (rading and wholeule) industry categary Japan.
Figure 14: Industry prediction analysis with GPT-4o. Input: BS+CT-PL+Summary Ground truth: non-bank financial institution.
18


Reviews:
[{"review_id": "yCK2zyHvxF", "reviewer": "Reviewer_tbHW", "summary": "1. The manuscript discusses the EDINET-Bench: an Open-source Japanese financial benchmark evaluating LLMs on accounting fraud detection, earnings forecasting, and industry classification.\n\n2. The benchmark is very in depth and built from 10 years of EDINET annual reports (~40K docs). The manuscript is worked very comprehensive and seems like a good work\n\n3. Tasks demand integration of text and tables  which is resulting into expert-level financial reasoning. The financial model is well comprehensive.\n\n4. Even top LLMs barely outperform logistic regression. The regression is also well connected and shows some good work done.", "questions": "1. How would fine-tuned domain-specific small language LLMs perform when compared with zero-shot learning?\n2. Could multilingual or cross-market benchmarks improve robustness, that will be good to explore?\n3. What scaffolding or reasoning frameworks could help ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This would be a large-scale Japanese financial benchmark requiring expert reasoning. So the model looks like a good work delivered.\n2. Covers diverse, realistic financial tasks. The real world scenario based tasks being financial in nature would be good. \n3. Includes reproducible toolkit and dataset which have other applications as well", "weaknesses": "1. LLMs show limited performance gains over traditional models. SLMs could also be explored in depth for the tasks. Even MCP route with more sophisticated approaches can be taken into consideration\n\n2. Some of the Evaluation limited to zero-shot; no fine-tuning comparisons. Zero shot learnings in some cases can be a good way to go but cannot be relied on all the time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i1Kq2Ikh66", "reviewer": "Reviewer_rxpy", "summary": "The paper presents EDINET-Bench, an open-source benchmark built from ~10 years of Japanese securities (annual) reports from the FSA’s EDINET system to evaluate LLMs on three expert-level financial tasks: accounting fraud detection, next-year earnings direction forecasting, and industry prediction. The authors release a tool (edinet2dataset) to harvest/parse EDINET TSV/PDFs and construct both a large corpus and three task datasets. Zero-shot evaluations of several closed/open LLMs versus classical baselines show that LLMs perform only marginally better than logistic regression and are outperformed by random forests on fraud detection; earnings forecasting is hard for all models; industry prediction is relatively easier from tables+summary. The paper argues that “just feed the report” is insufficient and calls for richer, more realistic scaffolding.", "questions": "Label audit: How many fraud-flagged amendments did two independent human annotators review? What was Cohen’s κ? Can you release a gold subset with multi-rater consensus?\n\nNegative controls: Did you try Beneish-M-Score or other ratio-based baselines (or hybrid RF + text signals) for fraud detection as stronger non-LLM baselines?\n\nTool-use baselines: Could you include RAG over earlier filings + programmatic ratio extraction (few carefully designed tools/skills) to test the “richer scaffolding” claim within this paper?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear gap & relevance. Financial NLP benchmarks often emphasize QA/extraction (e.g., FinQA/ConvFinQA/FinanceBench; largely English), whereas this work targets end-to-end expert reasoning over full reports in Japanese—a meaningful and under-served setting. \n\nNontrivial tasks and inputs. Whole-report inputs (tables + text) and tasks like fraud detection are practically impactful and distinct from prior QA settings and multimodal finance QA (e.g., FAMMA). \n\nOpen tooling & dataset construction details. The pipeline (EDINET API + TSV parsing via Polars), corpus statistics, class balances, and prompts are described with reasonable transparency.", "weaknesses": "Fraud labels rely on LLM-assisted screening of amended reports. Although later manually checked, the pipeline first classifies “amended report reasons” with Claude and then claims <5% label errors. This creates potential circularity (LLM both builds and is evaluated on the dataset domain) and possible systematic biases in what counts as fraud (e.g., wording styles of amendments). A larger human validation and inter-rater agreement would strengthen validity.\n\nDefinition of “fraud” is ambiguous. Amendments can be due to material misstatements or benign issues (e.g., shareholder counts). The paper filters for fraud phrases but the ground-truth notion of fraud vs. error could be better anchored (e.g., to JICPA/FSA definitions or enforcement actions). See classic fraud detection literature (e.g., Beneish M-Score) for framing and baselines.\n\nEvaluation setting is narrow. The main headline is that “LLMs struggle in zero-shot long-input.” But practitioner pipelines typically use retrieval over filings plus structured features (ratios, trends) and programmatic reasoning. Without strong RAG/agentic or tool-use baselines (tables→ratios→rules, fraud heuristics, or Beneish-style features), the conclusion may over-generalize. (The paper suggests this as future work, but including such baselines would be very informative.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uL4fLmpyJc", "reviewer": "Reviewer_rFcH", "summary": "EDINET-Bench is an open, reproducible benchmark built from 10 years of Japanese EDINET annual reports to evaluate LLMs on expert-level financial tasks: accounting fraud detection, earnings direction forecasting, and industry classification. The authors release edinet2dataset and the EDINET-Corpus, construct labels , and test frontier closed/open LLMs in zero-shot across multiple input configurations (tables + text). Results show state-of-the-art LLMs struggle and often only match simple classical baselines, highlighting the need for richer scaffolding and agentic settings to better reflect real-world financial analysis.", "questions": "- For each individual sub-task in the paper, how does the work establish novelty? For instance, in the case of fraud detection, there are already existing benchmarks such as:\n  Wang, R., Liu, J., Zhao, W., et al. *AuditBench: A Benchmark for Large Language Models in Financial Statement Auditing* [C] // International Workshop on AI for Transportation. Springer, Singapore, 2025: 59-81.\n\n- Why not evaluate stronger tabular models like XGBoost, LightGBM, or CatBoost to provide a stronger baseline in Table 6?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Real-world, long-context financial documents (tables + text) with open data, code, and tooling for reproducibility.\n- Clear task definitions and splits; systematic evaluation across models and input modalities, plus contamination checks.\n- Practical insights (e.g., text helps fraud detection but not earnings forecasting) that inform future benchmark and agentic framework design.", "weaknesses": "- The benchmark is limited to the Japanese financial market. But I believe the methodology of the benchmark is applicable to other financial markets. Results may be different and interesting in other markets (I guess the industry prediction task results may vary).\n\n- The industry prediction task, while straightforward to evaluate, does not closely reflect real-world use cases. In practice, industry labels are readily available from official sources, so predicting them from financial statements has limited practical value. As such, its significance for assessing LLM capabilities in realistic financial analysis scenarios is questionable.*\n\n- The earnings forecasting task does not incorporate any macroeconomic, industry-wide, or peer company data, despite earnings being strongly influenced by such factors in real-world scenarios. As a result, the task may undervalue the importance of external context and limit the realism of its predictive setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsvmRo1jzC", "reviewer": "Reviewer_epuV", "summary": "This paper introduces EDINET-Bench, an open-source benchmark designed to evaluate Large Language Models (LLMs) on complex financial tasks using Japanese financial statements. The authors highlight the current gap in financial benchmarks, which often focus on simpler tasks, unlike the specialized expertise required in real-world finance.\nEDINET-Bench comprises three challenging tasks: Accounting Fraud Detection, Earnings Forecasting and Industry Classification.\nThe evaluation setup involves testing various closed-source and open-source LLMs in a zero-shot setting, comparing their performance against classical baselines like Logistic Regression and Random Forest. Evaluation metrics include ROC-AUC and MCC for fraud detection and earnings forecasting, and accuracy for industry classification.\nKey findings indicate that even advanced LLMs struggle with these tasks, performing only marginally better than logistic regression in binary classification tasks. The results suggest that simply providing raw annual reports to LLMs is insufficient, underscoring the need for benchmark frameworks that better simulate real-world financial environments with richer scaffolding and task-specific reasoning support.", "questions": "1. The authors utilized Claude 3.7 Sonnet to generate labels for the accounting fraud detection task. Could you please elaborate on the rationale behind selecting this specific model over other potentially more powerful alternatives available at the time? Furthermore, as this approach relies on a single model for judgment, could you discuss whether and how this might introduce model-specific biases into the dataset, and what impact this could have on the overall label quality and benchmark reliability?\n2. How to rigorously demonstrate that the questions in proposed dataset genuinely require expert-level reasoning? Is it because models achieve lower accuracy on this dataset compared to existing benchmarks (a comparison notably absent in the paper)? Is it because the questions necessitate more reasoning steps? Or is it due to a requirement for external, specialized knowledge—a claim that likewise lacks supporting case analysis?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Authenticity and Challenging Nature of the Dataset: The benchmark is constructed from a large volume of real-world annual reports. The tasks, such as fraud detection and earnings forecasting, are highly challenging and effectively test the capabilities of large language models in complex, expert-level reasoning.\n2. Data Quality and Integrity: The risk of test set contamination was convincingly demonstrated to be low through rigorous analysis, such as company name prediction and temporal performance checks. This significantly enhances the credibility and reliability of the evaluation results.\n3. Commitment to Reproducibility: The authors clearly commit to open-sourcing the complete dataset (including both the EDINET-Corpus and EDINET-Bench) alongside the data construction toolkit (edinet2dataset). This ensures full transparency and greatly facilitates future research and replication of the reported results.", "weaknesses": "1. Limitations in Task Formulation and Evaluation Setup: The benchmark simplifies all three core tasks into binary classification problems. While this formulation facilitates straightforward evaluation, it may not fully capture the continuous and probabilistic nature of professional financial judgment. In practice, for instance, accounting fraud risk assessment often involves estimating a probability or risk score rather than making a binary determination, and earnings forecasting typically focuses on predicting specific numerical values or the magnitude of change.\n2. The definition of Expert-level reasoning in the paper is not rigorous enough. It is stated that\n\"Expert-level reasoning refers to tasks that require integrating information across tables and text, rather than mere extraction or simple computation.\"\nHowever, on the one hand, multi-table reasoning datasets already exist in the financial domain, such as FinMath[1]. On the other hand, the authors neither provide a quantitative explanation nor cite any references to clarify what constitutes \"simple computation,\" which undermines the clarity and rigor of the definition.\n3. Incomprehensive Experimental Evaluation: The selection of LLMs for evaluation, both open-source and closed-source, does not represent the most recent or state-of-the-art models available, even when considering the submission timeline of the paper and the inherent time required for experimentation. This limits the scope of the performance analysis and its relevance to the current frontier of LLM capabilities.\n\n[1]Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024. FinanceMATH: Knowledge-Intensive Math Reasoning in Finance Domains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12841–12858, Bangkok, Thailand. Association for Computational Linguistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)