Title: FINDVER: Explainable Claim Verification over Long and Hybrid-Content Financial Documents
Benchmark: FINDVER
Summary: A comprehensive benchmark designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FINDVER contains 2,400 expertannotated examples, divided into three subsets: information extraction, numerical reasoning, and knowledge-intensive reasoning—each addressing common scenarios encountered in realworld financial contexts.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Claim verification over long financial documents
Phenomenon defined by authors: Yes
Phenomenon definition: Given a financial document and a claim, a model is expected to provide a label of whether the claim is refuted or entailed based on the evidence in the document, followed by a rationale explanation of its prediction. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Consider a single financial document d, containing textual data P and tabular data T, associated with a claim c that requires verification. The task is defined as follows:

1. Entailment Classification: The language model must determine the entailment label ℓ ∈ L = {“entailed”, “refuted”}, based on the hybrid-content financial document (P and T).

2. Reasoning-Process Explanation Generation: The model must generate a natural language explanation e, which articulates the reasoning process behind the validity of the claim c, relying solely on the textual (P) and tabular (T) content of the document d.
Task item definition: A financial document, a claim, label i.e. refutes or entails
Task source: Real task examples (e.g. GitHub issues), Domain expert annotators
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1700
Dataset size notes: 700
Dataset metadata detail: subset task e.g. FDV-IE, FDV-MATH, FDV-KNOW, relevant context, report
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Metrics across each subset task e.g. FDV-IE, FDV-MATH, FDV-KNOW
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No comparisons made
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Somehwat
Author validity detail: The authors engage with domain experts during dataset design "To identify the common reasoning-intensive scenarios in claim verification based on financial documents, we engage with domain experts and conducted a preliminary study. This helped us determine three key types of scenarios that frequently arise in realworld settings: information extraction, numerical reasoning, and knowledge-intensive reasoning"
Taxonomy: Finance / 
New taxonomy: Specific Application / Profesisonal Domains