Title: WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained
Language Model for Financial Domain
Benchmark: FLUE
Summary:  the Financial Language Understanding
Evaluation (FLUE), an open-source comprehensive
suite of benchmarks for the financial
domain. These include new benchmarks across
5 NLP tasks in financial domain as well as common
benchmarks used in the previous research.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: natural language understanding in the financial domain
Phenomenon defined by authors: Yes
Phenomenon definition: The ability of LLMs to perform across 5 financial tasks such as financial sentiment analysis, news headline classification, named entity recognition, structure boundary detection, and question answering.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task is defined as evaluating language models on a suite of five financial domain NLP tasks: financial sentiment analysis, news headline classification, named entity recognition, structure boundary detection, and question answering.
Task item definition: N/A, for every task there will be a respective item
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 969, 234, 2282, 302, 131, 333
Dataset size notes: for all 5 tasks: 19,367 and 2,674
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean: F1 scores and accuracy. MSE. nDCG and MRR. Perplexity
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Finance / 
New taxonomy: Specific Application / Profesisonal Domains