Omkar2118/datasets huggingface dataset:
FinanceBench

Certainly! **FinanceBench** is a groundbreaking benchmark designed for evaluating the performance of large language models (LLMs) in the domain of financial question answering (QA). Here are the key details about FinanceBench:

1. **What is FinanceBench?**
   - **FinanceBench** is a **first-of-its-kind test suite** specifically tailored for assessing LLMs' capabilities in answering financial questions.
   - It focuses on **open book financial QA** and comprises a collection of **10,231 questions** related to publicly traded companies.
   - Each question comes with corresponding answers and evidence strings.

2. **Why FinanceBench Matters:**
   - The questions in FinanceBench are **ecologically valid**, covering a diverse set of scenarios.
   - They are intentionally designed to be **clear-cut and straightforward**, serving as a minimum performance standard.
   - FinanceBench aims to evaluate how well LLMs handle financial queries, especially those related to publicly traded companies.

3. **Model Evaluation:**
   - Researchers tested **16 state-of-the-art model configurations**, including **GPT-4-Turbo**, **Llama2**, and **Claude2**.
   - The evaluation involved a sample of **150 cases** from FinanceBench, with **manual review of answers** (totaling **2,400**).
   - Notably, **existing LLMs have limitations** for financial QA. For instance:
     - **GPT-4-Turbo**, when used with a retrieval system, **incorrectly answered or refused to answer 81% of questions**.
     - Augmentation techniques (such as longer context windows) improved performance but are **unrealistic for enterprise settings** due to increased latency and inability to handle larger financial documents.
   - All examined models exhibit weaknesses, such as **hallucinations**, which limit their suitability for enterprise use.

4. **Availability:**
   - The **FinanceBench cases** are available **open-source** for further exploration and research.

¹: Islam, P., Kannappan, A., Kiela, D., Qian, R., Scherrer, N., & Vidgen, B. (2023). FinanceBench: A New Benchmark for Financial Question Answering. arXiv preprint arXiv:2311.11944.
²: [Link to the official paper](https://arxiv.org/abs/2311.11944)
³: [Papers with Code - FinanceBench](https://paperswithcode.com/paper/financebench-a-new-benchmark-for-financial)

Source: Conversation with Bing, 3/16/2024
(1) Papers with Code - FinanceBench: A New Benchmark for Financial Question .... https://paperswithcode.com/paper/financebench-a-new-benchmark-for-financial.
(2) FinanceBench: A New Benchmark for Financial Question Answering. https://arxiv.org/abs/2311.11944.
(3) Papers with Code - Paper tables with annotated results for FinanceBench .... https://paperswithcode.com/paper/financebench-a-new-benchmark-for-financial/review/.

Introducing paper: FinanceBench: A New Benchmark for Financial Question Answering
Earnings-22

**Earnings-22** is a practical benchmark designed to evaluate **automatic speech recognition (ASR)** systems' performance on real-world, accented audio. Let me provide you with more details:

1. **Corpus Description**:
   - Earnings-22 consists of **125 audio files** totaling **119 hours** of English-language **earnings calls**. These calls were gathered from **global companies**.
   - Unlike many existing corpora, Earnings-22 focuses on **speech in the wild**, representing real-world scenarios where accents and environmental conditions vary.

2. **Purpose and Significance**:
   - ASR systems have achieved impressive performance on common corpora but often struggle with real-world speech.
   - Earnings-22 aims to bridge this gap by providing a **free-to-use benchmark** that includes **accented audio**.
   - Researchers and industry professionals can use Earnings-22 to evaluate and improve ASR models' robustness.

3. **Comparison and Insights**:
   - The benchmark involves **four commercial ASR models**, and their performance is compared.
   - By considering the **country of origin**, the study reveals variations in ASR accuracy.
   - **Individual Word Error Rate (IWER)** analysis highlights how certain accents impact model performance more than others.

4. **Academic and Industrial Impact**:
   - Earnings-22 serves as a valuable resource for both **academic research** and **industrial applications**.
   - It provides a realistic dataset for evaluating ASR systems' effectiveness in handling diverse accents.

Source: Conversation with Bing, 3/16/2024
(1) Earnings-22: A Practical Benchmark for Accents in the Wild. https://arxiv.org/abs/2203.15591.
(2) Earnings-22: A Practical Benchmark for Accents in the Wild. https://deepai.org/publication/earnings-22-a-practical-benchmark-for-accents-in-the-wild.
(3) arXiv:2203.15591v1 [cs.CL] 29 Mar 2022. https://arxiv.org/pdf/2203.15591.pdf.
(4) undefined. https://doi.org/10.48550/arXiv.2203.15591.

Introducing paper: Earnings-22: A Practical Benchmark for Accents in the Wild