$\texttt{M$^3$FinMeeting}$: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset
Authors: Jie Zhu, Junhui Li, yalong wen, Xiandong Li, Lifan Guo, Feng Chen
Accepted at: acl25

M^3FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset

Jie Zhu1,2, Junhui Li1*, Yalong Wen2, Xiandong Li3, Lifan Guo2, Feng Chen2
1School of Computer Science and Technology, Soochow University
2Qwen DianJin Team, Alibaba Cloud Computing
3Nanjing University
zhujie951121@gmail.com, lijunhui@suda.edu.cn
{lifan.lg, betterman.chenf}@alibaba-inc.com

Abstract
Recent breakthroughs in large language models (LLMs) have led to the development of new benchmarks for evaluating their performance in the financial domain. However, current financial benchmarks often rely on news articles, earnings reports, or announcements, making it challenging to capture the real-world dynamics of financial meetings. To address this gap, we propose a novel benchmark called M^3FinMeeting, which is a multilingual, multi-sector, and multi-task dataset designed for financial meeting understanding. First, M^3FinMeeting supports English, Chinese, and Japanese, enhancing comprehension of financial discussions in diverse linguistic contexts. Second, it encompasses various industry sectors defined by the Global Industry Classification Standard (GICS), ensuring that the benchmark spans a broad range of financial activities. Finally, M^3FinMeeting includes three tasks: summarization, question-answer (QA) pair extraction, and question answering, facilitating a more realistic and comprehensive evaluation of understanding. Experimental results with seven popular LLMs reveal that even the most advanced long-context models have significant room for improvement, demonstrating the effectiveness of M^3FinMeeting as a benchmark for assessing LLMs’ financial meeting comprehension skills.1

1 Introduction
Financial meetings, whether in person or virtual, serve as critical venues for decision-making, negotiation, and strategy formulation among various participants. Although large language models (LLMs) have demonstrated impressive performance across multiple natural language processing (NLP) tasks (OpenAI, 2023), it remains uncertain how they can effectively understand and process lengthy speech texts to help financial professionals expedite their work. Key capabilities, such as summarizing crucial points, responding to inquiries, and extracting question-answer pairs, are particularly beneficial for enhancing productivity and facilitating informed discussions in this context.

In the financial domain, there are various benchmarks available in different languages, such as FinQA (Chen et al., 2021) and ConvFinQA (Chen et al., 2022) in English, as well as CFLUE (Zhu et al., 2024) and the CCKS series shared tasks in Chinese (Tianchi, 2019, 2020, 2021, 2022). However, these datasets are primarily sourced from financial news and earnings reports, lacking content from real-world financial meetings. Additionally, they are monolingual, limited to English or Chinese. To address this gap, we introduce a new dataset called M^3FinMeeting, designed for multilingual and multi-sector evaluation of financial meeting understanding, featuring multiple tasks. First, M^3FinMeeting supports multiple languages, including English, Chinese and Japanese, enhancing the understanding of financial discussions across different linguistic contexts. Second, it encompasses all 11 industry sectors defined by Global Industry Classification Standard (GICS). Finally, M^3FinMeeting includes multiple classical NLP tasks such as summarization, question answering, and question-answer pair extraction. Importantly, since financial meetings typically last one to two hours, M^3FinMeeting provides long-context data, which is essential for assessing the ability of LLMs to handle complex tasks. Additionally, the tasks within M^3FinMeeting require LLMs to generate long responses, allowing for a thorough assessment of their capabilities in producing coherent and relevant outputs in challenging scenarios.

Based on M^3FinMeeting, we assess the effectiveness of seven representative LLMs including two OpenAI GPTs and five open-sourced LLMs. The experimental results indicate that Qwen2.5-72B-
Instruct (Yang et al., 2024b) significantly outperforms other large language models (LLMs), achieving overall scores above 70 when evaluated by GPT-4 (OpenAI, 2023). However, this also suggests that even the most advanced LLMs currently available struggle with the tasks in M^3FinMeeting, revealing substantial rooms for performance improvement.

Our main contributions can be summarized as follows:

• M^3FinMeeting introduces a novel evaluation benchmark specifically designed for financial meetings, addressing the lack of real-world financial meeting data in existing benchmarks.

• M^3FinMeeting supports multilingual evaluation in English, Chinese, and Japanese, spans the 11 industry sectors defined by GICS, and includes three key NLP tasks: summarization, question-answer pair extraction, and question answering. All documents are carefully annotated by financial analysts to ensure high-quality and accurate evaluation.

• Through extensive experiments and detailed analyses, we thoroughly assess the performance of state-of-the-art long-context LLMs, offering valuable insights into their limitations and potential improvements for better long-context modeling in financial scenarios.

2 Related Work

2.1 Financial Evaluation Benchmarks
Financial NLP has become a key application area for LLMs, attracting increasing attention for its benchmarks. Table 1 summarizes recent benchmarks in this field. In English, FINQA by Chen et al. (2021) contains 8,281 question-answering pairs, with their numerical reasoning, from the earnings reports of S&P 500 companies. ECTSum by (Mukherjee et al., 2022) contains 2,425 transcripts of earnings calls, paired with short telegram-style bullet point summaries. FLUE by (Shah et al., 2022) and FLARE by Xie et al. (2024) offer heterogeneous benchmarks with various financial NLP tasks from existing datasets, including financial sentiment detection (Malo et al., 2014), named entity recognition (Alvarado et al., 2015), news headline classification (Sinha and Khandait, 2020), question answering (Maia et al., 2018), boundary detection tasks (Au et al., 2021), text summarization (Zhou et al., 2021; Mukherjee et al., 2022), and stock movement prediction (Xu and Cohen, 2018; Wu et al., 2018; Soun et al., 2022). FinTextQA by Chen et al. (2024) includes 1,262 long-form QA pairs from finance textbooks and government websites. BizBench by Krumdick et al. (2024) features eight quantitative reasoning tasks from professional exams, earnings reports, and other financial sources. FINANCEBENCH by Islam et al. (2023) includes 10,231 question-answer-evidence triplets from public filings and earnings reports.

In Chinese, the CCKS series has released several datasets specifically designed for various event extraction tasks (Tianchi, 2019, 2020, 2021, 2022). Additionally, significant effort has gone into creating evaluation datasets for document-level extraction from financial announcements, judgments, and news articles. These include DCFEE (Yang et al., 2018), DuEE-Fin (Han et al., 2022), Doc2EDAG (Zheng et al., 2019), and CFinDEE (Zhang et al., 2024a). Moreover, both BBT-CFLEB dataset (Lu et al., 2023) and CFLUE dataset (Zhu et al., 2024) serve as heterogeneous benchmarks that cover a wide range of NLP tasks, including multiple-choice question answering and reasoning, news and text classification, summarization, relation extraction, sentiment classification, question answering, and reading comprehension.

In addition to the benchmarks mentioned above, several datasets have been developed in the financial domain to support the training of financial LLMs, such as FLANG (Shah et al., 2022), Pixiu (Xie et al., 2024), InvestLM (Yang et al., 2023b), FinGPT (Yang et al., 2023a), and DianJinR1 (Zhu et al., 2025). However, it is important to note that most of these datasets primarily rely on sources such as financial news, announcements, filings, and earnings reports. In contrast, our focus is on financial meetings, which provide unique insights and discussions that are often absent from existing benchmarks.

2.2 Summarization and Question-Answering Benchmarks

Summarization Benchmarks. Recent studies (Goyal et al., 2022; Zhang et al., 2024b; Pu et al., 2023) indicate that human preferences strongly favor summaries produced by LLMs over those generated by fine-tuned models or even reference summaries. This highlights the need to create new datasets that can comprehensively evaluate the summarization abilities of LLMs.
<table>
  <tr>
    <th>Language</th>
    <th>Dataset</th>
    <th>Sources</th>
    <th>Tasks</th>
  </tr>
  <tr>
    <td rowspan="8">English</td>
    <td>FinQA (<a href="https://arxiv.org/abs/2106.07453">Chen et al., 2021</a>)</td>
    <td>Earnings reports</td>
    <td>Question answering & reasoning</td>
  </tr>
  <tr>
    <td>ConvFinQA (<a href="https://arxiv.org/abs/2205.09262">Chen et al., 2022</a>)</td>
    <td>Reports</td>
    <td>Question answering & reasoning</td>
  </tr>
  <tr>
    <td>FLUE (<a href="https://arxiv.org/abs/2205.09262">Shah et al., 2022</a>)</td>
    <td>Multiple</td>
    <td>5 tasks</td>
  </tr>
  <tr>
    <td>ECTSum (<a href="https://arxiv.org/abs/2205.09262">Mukherjee et al., 2022</a>)</td>
    <td>Call transcripts</td>
    <td>Summarization</td>
  </tr>
  <tr>
    <td>FLARE (<a href="https://arxiv.org/abs/2401.07272">Xie et al., 2024</a>)</td>
    <td>Multiple</td>
    <td>8 tasks</td>
  </tr>
  <tr>
    <td>FinTextQA (<a href="https://arxiv.org/abs/2401.07272">Chen et al., 2024</a>)</td>
    <td>Long reports</td>
    <td>Question answering</td>
  </tr>
  <tr>
    <td>BizBench (<a href="https://arxiv.org/abs/2401.07272">Krumdick et al., 2024</a>)</td>
    <td>Multiple</td>
    <td>8 quantitative reasoning tasks</td>
  </tr>
  <tr>
    <td>FinanceBench (<a href="https://arxiv.org/abs/2309.13472">Islam et al., 2023</a>)</td>
    <td>Public filings, Earnings reports</td>
    <td>Question answering</td>
  </tr>
  <tr>
    <td rowspan="6">Chinese</td>
    <td>CCKS Tianchi (2019, 2020, 2021, 2022)</td>
    <td>Multiple</td>
    <td>Multiple event-related tasks</td>
  </tr>
  <tr>
    <td>DCFEE (<a href="https://arxiv.org/abs/1808.01764">Yang et al., 2018</a>)</td>
    <td>Announcements</td>
    <td>Document-level event extraction</td>
  </tr>
  <tr>
    <td>Doc2EDAG (<a href="https://arxiv.org/abs/1905.07777">Zheng et al., 2019</a>)</td>
    <td>Announcements</td>
    <td>Document-level event extraction</td>
  </tr>
  <tr>
    <td>DuEE-Fin (<a href="https://arxiv.org/abs/2205.09262">Han et al., 2022</a>)</td>
    <td>Multiple</td>
    <td>Document-level event extraction</td>
  </tr>
  <tr>
    <td>CFinDEE (<a href="https://arxiv.org/abs/2401.07272">Zhang et al., 2024a</a>)</td>
    <td>News articles</td>
    <td>Document-level event extraction</td>
  </tr>
  <tr>
    <td>BBT-CFLEB (<a href="https://arxiv.org/abs/2309.13472">Lu et al., 2023</a>)</td>
    <td>Multiple</td>
    <td>6 tasks</td>
  </tr>
  <tr>
    <td>CFLUE (<a href="https://arxiv.org/abs/2401.07272">Zhu et al., 2024</a>)</td>
    <td>Multiple</td>
    <td>Multiple-choice Question Answering & Reasoning and 5 tasks</td>
  </tr>
  <tr>
    <td>English, Chinese, Japanese</td>
    <td>M^3FinMeeting (Ours)</td>
    <td>Meetings</td>
    <td>Summarization, Question answering, QA pair extraction</td>
  </tr>
</table>

Table 1: Summary of recent financial benchmarks.

For instance, SumSurvey by <a href="https://arxiv.org/abs/2401.07272">Liu et al. (2024a)</a> is a new dataset focused on summarizing long scientific survey papers. REFINESUMM by <a href="https://arxiv.org/abs/2401.07272">Patil et al. (2024)</a> is tailored for image-text multimodal summarization. MovieSum by <a href="https://arxiv.org/abs/2401.07272">Saxena and Keller (2024)</a> comprises 2,200 pairs of movie screenplays and their corresponding summaries. Additionally, HeSum by <a href="https://arxiv.org/abs/2401.07272">Paz-Argaman et al. (2024)</a> is a novel benchmark dataset specifically designed for the low-resource Hebrew language. Our research also emphasizes long context, but we concentrate on meetings as our primary source, allowing us to analyze communication dynamics within this particular area. Several meeting speech summarization benchmarks are available, such as AMI (<a href="https://arxiv.org/abs/0506001">Carletta et al., 2005</a>) and ICSI (<a href="https://arxiv.org/abs/0306001">Janin et al., 2003</a>), which consist of English-language video (or audio) recordings of meetings, typically spanning tens of hours.

Question-Answering Benchmarks. Question Answering (QA) is a key task in NLP, with increasingly challenging benchmarks being developed, including those focused on the financial domain. For example, several benchmarks have been introduced to assess the long context understanding capabilities of LLMs (<a href="https://arxiv.org/abs/2401.07272">Li et al., 2024; Wang et al., 2024; Bai et al., 2024b</a>). Additionally, many multimodal QA benchmarks have emerged to assess LLMs’ capabilities in detecting and retrieving relevant information from various inputs, such as images, videos, texts, tables, and meetings (<a href="https://arxiv.org/abs/2309.13472">Li et al., 2023; Fu et al., 2023; Jin et al., 2024; Prasad et al., 2023</a>).

3 M^3FinMeeting: A Financial Meeting Benchmark

3.1 Overview

A formal financial meeting typically involves a few participants and lasts one to two hours. It promotes discussions among key participants, allowing for decision-making and strategic planning based on real-time information and reports. Participants express their opinions verbally, which distinguishes both the content and style of financial meetings from financial news, earnings reports, or announcements. The M^3FinMeeting benchmark, based on hundreds real financial meetings, includes three NLP tasks: summarization, question answering, and QA pair extraction. To reflect real-world sce-
<table>
  <tr>
    <th colspan="4">Overview</th>
  </tr>
  <tr>
    <th>Language</th>
    <th>#Meeting</th>
    <th>Avg hour</th>
    <th>Avg token</th>
  </tr>
  <tr>
    <td>EN</td>
    <td>100</td>
    <td>0.96</td>
    <td>10,086</td>
  </tr>
  <tr>
    <td>ZH</td>
    <td>400</td>
    <td>1.15</td>
    <td>11,740</td>
  </tr>
  <tr>
    <td>JA</td>
    <td>100</td>
    <td>1.01</td>
    <td>13,284</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="4">GICS Sector</th>
  </tr>
  <tr>
    <th>Sector</th>
    <th>Language</th>
    <th>#Meeting</th>
    <th>Avg token</th>
  </tr>
  <tr>
    <td>Com. Services</td>
    <td>EN, ZH, JA</td>
    <td>36</td>
    <td>11,240</td>
  </tr>
  <tr>
    <td>Con. Dis.</td>
    <td>EN, ZH, JA</td>
    <td>122</td>
    <td>11,104</td>
  </tr>
  <tr>
    <td>Con. Staples</td>
    <td>EN, ZH, JA</td>
    <td>52</td>
    <td>11,699</td>
  </tr>
  <tr>
    <td>Energy</td>
    <td>EN, ZH, JA</td>
    <td>32</td>
    <td>15,841</td>
  </tr>
  <tr>
    <td>Financials</td>
    <td>EN, ZH, JA</td>
    <td>49</td>
    <td>12,835</td>
  </tr>
  <tr>
    <td>Healthcare</td>
    <td>EN, ZH, JA</td>
    <td>47</td>
    <td>13,937</td>
  </tr>
  <tr>
    <td>Industrials</td>
    <td>EN, ZH, JA</td>
    <td>111</td>
    <td>12,062</td>
  </tr>
  <tr>
    <td>IT</td>
    <td>EN, ZH, JA</td>
    <td>98</td>
    <td>13,430</td>
  </tr>
  <tr>
    <td>Materials</td>
    <td>EN, ZH, JA</td>
    <td>32</td>
    <td>11,393</td>
  </tr>
  <tr>
    <td>RealEstate</td>
    <td>EN, ZH, JA</td>
    <td>13</td>
    <td>15,270</td>
  </tr>
  <tr>
    <td>Utilities</td>
    <td>EN, ZH, JA</td>
    <td>8</td>
    <td>17,290</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="4">Length Set</th>
  </tr>
  <tr>
    <th>Length</th>
    <th>Language</th>
    <th>#Meeting</th>
    <th>Avg token</th>
  </tr>
  <tr>
    <td>Set1 (0-5K)</td>
    <td>EN, ZH, JA</td>
    <td>59</td>
    <td>3,546</td>
  </tr>
  <tr>
    <td>Set2 (5-10K)</td>
    <td>EN, ZH, JA</td>
    <td>164</td>
    <td>7,509</td>
  </tr>
  <tr>
    <td>Set3 (10-15K)</td>
    <td>EN, ZH, JA</td>
    <td>195</td>
    <td>12,476</td>
  </tr>
  <tr>
    <td>Set4 (15-20K)</td>
    <td>EN, ZH, JA</td>
    <td>124</td>
    <td>17,419</td>
  </tr>
  <tr>
    <td>Set5 (>20K)</td>
    <td>EN, ZH, JA</td>
    <td>58</td>
    <td>25,281</td>
  </tr>
</table>

Table 2: Data statistics of M3FinMeeting benchmark.

<table>
  <tr>
    <th>Language</th>
    <th>AST</th>
    <th># SS</th>
    <th>C<sub>token</sub></th>
    <th>C<sub>sent</sub></th>
  </tr>
  <tr>
    <td>EN</td>
    <td>927</td>
    <td>9.20</td>
    <td>10.88</td>
    <td>10.49</td>
  </tr>
  <tr>
    <td>ZH</td>
    <td>2,524</td>
    <td>15.17</td>
    <td>4.65</td>
    <td>3.62</td>
  </tr>
  <tr>
    <td>JA</td>
    <td>1,149</td>
    <td>8.24</td>
    <td>11.56</td>
    <td>11.92</td>
  </tr>
</table>

Table 3: Statistics for the summarization task. Here, AST is for average summary token, #SS is for averaged number of section summary, C<sub>token</sub> and C<sub>sent</sub> for the compression ratio at token-level and sentence-level, respectively.

transcribed speech documents can be structured into sections based on discussion topics, with each section having its own summary, which we refer to section summary. We concatenate these section summaries sequentially to create a summary of the entire document. LLMs must implicitly identify and segment the document into various sections and then extract key point from each. Given transcribed speech documents and their reference summaries, we follow Koh et al. (2022) to compute the compression ratio of a source document length against its reference summary length at both token-level and sentence-level. As shown in Table 3, on average an English meeting contains 9.20 section summaries totaling 927 tokens.

3.2.2 QA Pair Extraction

The task of question-answer (QA) pair extraction involves identifying and extracting relevant QA pairs from transcribed financial meetings. This is crucial for analyzing discussions and making key insights readily accessible. To successfully perform this task, LLMs must recognize various types of questions posed during the meeting and accurately locate their corresponding answers. For example, questions like What were we just talking about? should be disregarded as they lack meaningful information. Additionally, participants may ask multiple questions at once, while responses may address them sequentially. This complexity requires LLMs to discern the structure of the dialogue, correctly pair each question with its answer. By employing natural language processing techniques, this task enhances information retrieval efficiency and facilitates subsequent analyses, providing structured and contextualized insights from the meeting. Table 4 shows statistics of QA pairs.

3.2.3 Question Answering

The question answering (QA) task evaluates the ability of LLMs to localize knowledge, which is essential for effective long-context processing (Wang
<table>
  <tr>
    <th>Lang.</th>
    <th>QA<sub>token</sub></th>
    <th>#QA</th>
    <th>Q<sub>token</sub></th>
    <th>A<sub>token</sub></th>
  </tr>
  <tr>
    <td>EN</td>
    <td>2.239</td>
    <td>17.23</td>
    <td>17.62</td>
    <td>110.19</td>
  </tr>
  <tr>
    <td>ZH</td>
    <td>2.852</td>
    <td>16.10</td>
    <td>36.44</td>
    <td>148.95</td>
  </tr>
  <tr>
    <td>JA</td>
    <td>2.934</td>
    <td>10.84</td>
    <td>34.55</td>
    <td>178.99</td>
  </tr>
</table>

Table 4: Statistics for both QA Pair Extraction and Question Answering Tasks. \( QA_{token} \) is the average token length of all QA pairs per meeting, #QA is the average number of QA pairs per meeting, and \( Q_{token} \) and \( A_{token} \) denote the average token lengths of a question and answer, respectively.

et al., 2024). For simplicity, we use the QA pairs described above for this task. As mentioned, the transcribed speech text can be divided into multiple sections, the QA task tests the LLMs’ capability to find evidence within that designated section, while other sections with similar but unrelated content act as noise. This setup ensures a focused assessment of the models’ information retrieval skills.

3.3 Benchmark Construction

3.3.1 Data Collection
We have established four criteria for the manual collection of audio files from financial meetings, covering a range of events such as public roadshows, brokerage strategy meetings, industry exchanges, and earnings presentations: (1) Timeliness: Most meetings should be from recent years; (2) Length: Preference is given to longer audio files; (3) Categorizability: The audio files must align with categories defined in the GICS; (4) Authoritativeness: All audio files are sourced from our financial firm partners and are protected by our copyright.

All audio files are transcribed into text using ASR toolkit Whisper,\footnote{https://openai.com/index/whisper/} followed by a thorough manual correction process. Strict measures are used to ensure that no sensitive or personally identifiable information is included in the transcripts.

3.3.2 Annotation Process and Quality Control
The datasets for manual annotation are drawn from various projects that recruit experienced analysts fluent in different languages. These annotators follow the annotation guidelines (See Figure 10 in Appendix F), undergo comprehensive training, and have access to onboarding and guidance materials. Additionally, other analysts review the annotators’ work and provide ongoing feedback.

• Transcribed Speech Correction: Annotators correct ASR-generated transcriptions with original audio files available for reference during the correction process.

• Summarization: The annotators segment each document into sections based on distinct topics, ensuring that only sections with clear boundaries are selected for summarization. For each valid section, annotators are encouraged to use simple sentences in the summary. Prior to the annotation process, annotators undergo extensive training and discussions to achieve a high level of consensus. In the annotation process, the original audio files are available for reference.

• QA Pair Extraction: With corrected speech documents, annotators manually extract financially relevant questions from the text. For each identified question, they search for corresponding answers in the subsequent content. Only questions with valid answers are retained for further analysis. This meticulous process guarantees the quality and relevance of the extracted QA pairs, significantly enhancing the dataset’s value for financial insights.

Figure 4 in Appendix A shows a screenshot of an annotated example. For additional complete examples, please refer to the attached file.

4 Experimentation

4.1 Experimental Settings

Models. We evaluate seven advanced long-context LLMs with context windows ranging from 16K to 1000K, including two API-based LLMs: GPT-4o-2024-08-06-128K (OpenAI, 2023) and GPT-3.5.turbo-0125-16K, as well as five open-source LLMs: GLM4-9B-Chat-1000K (Zeng et al., 2022), Llama3.1-8b-Instruct-128K (Dubey et al., 2024), Qwen2-7B-chat-128K (Yang et al., 2024a), Qwen2-72B-Instruct-128K, and Qwen2.5-72B-Instruct-128K (Yang et al., 2024b). All models support the languages in M³FinMeeting.

Prompts. We evaluate LLMs in a zero-shot setting. For summarization, we prompt the LLMs to implicitly identify document sections and generate individual summaries, which are then combined into a final document summary. For QA pair extraction, we first prompt the LLMs to extract all questions, then provide answers for each sequentially. For question answering, instead of addressing one
question at a time, we combine related questions into a single prompt, allowing the LLM to produce a comprehensive response that includes all the answers. This better aligns with real-world tasks, like writing reviews or reports, and reduces API calls. Prompt examples are provided in Appendix C.

Metrics. For summarization, as shown in Figure 1, the final summary consists of multiple section summaries. We evaluate this using precision, recall, and F1 scores rather than traditional metrics like BLEU and ROUGE, as they better reflect how well the generated section summaries align with the reference (gold) summaries. Specifically, we align the automatic and gold section summaries using a cosine similarity score above 0.75, calculated with the OpenAI Embedding model4. Appendix B gives details of the three metrics. Meanwhile, recent studies (Wang et al., 2023; Zhang et al., 2023; Liu et al., 2024b) show that GPT-4 (OpenAI, 2023) aligns closely with human evaluations. Therefore, we use GPT-4 (gpt-4-turbo-2024-04-09) as the judge (GPT-4-Judge) to evaluate document-level summaries based on five criteria: Coverage, Redundancy, Readability, Accuracy, and Consistency, with scores ranging from 0 to 100. We report the average GPT-4-Judge score, based on the prompt in Appendix C.

For QA pair extraction, we assess the quality of generated questions using precision, recall, and F1 scores, comparing them to the reference questions, similar to the summarization evaluation. Additionally, GPT-4-Judge evaluates the generated QA pairs against the gold pairs using the same five criteria, and we report the average score.

For question answering, we group all questions together and prompt the LLMs to answer them sequentially, repeating each question before generating its corresponding answer. Therefore, we evaluate performance using the same metrics as in QA pair extraction, assessing both the quality of repeated questions and the overall performance of the generated QA pairs.

For the three tasks, Appendix D presents performance metrics using BLEU (Papineni et al., 2002) and ROUGE (Lin and Hovy, 2002). Appendix E includes detailed performance across languages, lengths, and GICS sectors. To address potential self-bias in LLM evaluations (e.g., GPT-4-Judge favoring GPT-4-generated answers), we follow Bai et al. (2024a) and use Qwen-plus5 as an alternative judge model. Results, shown in Section 4.3, indicate minimal bias, as the performance trend from GPT-4-Judge and Qwen-plus-Judge are very consistent. Moreover, we perform a human evaluation and calculate Fleiss’ Kappa (Scott, 1995) to measure agreement between GPT-4-Judge and human annotators, further supporting our conclusions. The results are presented in Section 4.4.

4.2 Results

Main Results. Table 5 shows the main results over all meetings.6 From it, we have the following observations:

• Overall performance: The seven LLMs can be divided into three groups. Group 1 consists of Qwen2.5-72B-Instruct, Qwen2-72B-Instruct, and GPT-4o, all achieving an overall GPT-4-Judge score near or above 70.0. Among these, Qwen2.5-72B-Instruct performs best, followed by GPT-4o and Qwen2-72B-Instruct, which deliver comparable results. Group 2 includes Qwen2-7B-Instruct and GLM4-9B-Chat, both scoring around 60.0. Group 3 consists of GPT-3.5-turbo and LLaMA3.1-8B-Instruct, with LLaMA3.1-8B-Instruct outperforming GPT-3.5-turbo.

• Summarization: The precision, recall, and F1 scores for section-level summaries are all below 30%, indicating poor alignment between generated and gold section-level summaries. These low scores suggest that LLMs struggle both with semantic accuracy and document segmentation.

• QA Pair Extraction: The low precision, recall, and F1 scores suggest poor alignment between generated and gold questions.

4https://platform.openai.com/docs/models/embeddings (text-embedding-3-small)

5https://help.aliyun.com/zh/model-studio/developer-reference/what-is-qwen-llm

6The total cost for OpenAI API calls was about $2,500, while experiments with other LLMs use eight NVIDIA A100/80G GPUs.

Figure 1: Illustration of the summarization evaluation.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Summarization</th>
    <th colspan="4">QA Pair Extraction</th>
    <th colspan="4">Question Answering</th>
    <th colspan="4">Overall</th>
  </tr>
  <tr>
    <th>P.</th><th>R.</th><th>F1</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>F1</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>F1</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>F1</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>F1</th><th>GPT-4</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>27.82</td><td>12.07</td><td>16.83</td><td>73.61</td><td>23.33</td><td>41.98</td><td>29.99</td><td>66.85</td><td>93.93</td><td>93.16</td><td>93.55</td><td>71.79</td><td>70.66</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td><td>17.13</td><td>9.39</td><td>12.13</td><td>44.56</td><td>9.66</td><td>25.07</td><td>13.95</td><td>31.13</td><td>84.16</td><td>92.60</td><td>88.18</td><td>42.78</td><td>39.55</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>10.30</td><td>11.26</td><td>10.76</td><td>67.71</td><td>15.08</td><td>5.22</td><td>7.76</td><td>46.06</td><td>93.37</td><td>92.20</td><td>92.78</td><td>67.72</td><td>60.76</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>6.24</td><td>8.34</td><td>7.14</td><td>52.01</td><td>13.05</td><td>7.30</td><td>9.37</td><td>44.64</td><td>57.21</td><td>39.97</td><td>47.06</td><td>40.01</td><td>45.76</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>28.67</td><td>19.39</td><td>23.14</td><td>73.59</td><td>11.98</td><td>16.40</td><td>13.85</td><td>37.33</td><td>89.83</td><td>93.01</td><td>91.40</td><td>69.99</td><td>60.71</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td><b>29.59</b></td><td><b>20.18</b></td><td><b>23.99</b></td><td><b>74.17</b></td><td>22.43</td><td>28.82</td><td>25.22</td><td>60.85</td><td>93.27</td><td><b>93.58</b></td><td><b>93.42</b></td><td><b>73.50</b></td><td><b>69.66</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td><b>28.98</b></td><td>15.56</td><td>20.25</td><td><b>74.51</b></td><td><b>32.61</b></td><td><b>45.65</b></td><td><b>38.41</b></td><td><b>68.03</b></td><td><b>93.75</b></td><td><b>93.59</b></td><td><b>93.66</b></td><td><b>74.81</b></td><td><b>72.54</b></td>
  </tr>
</table>

Table 5: Performance of LLMs on three evaluation tasks. The overall GPT-4-Judge score is the micro-average of its scores across all three tasks. Scores in bold/underline denote the top/second-best performances.

![Bar charts showing GPT-4-Judge scores across languages, GICS sectors, and input lengths](page_184_420_1224_420.png)

Figure 2: Performance based on GPT-4-Judge scores across languages (a), GICS sectors (b), and input lengths (c).

For example, even the best-performing LLM, Qwen2.5-72B-Instruct, achieves only 45.65% recall, missing more than half of the gold questions. This highlights significant room for improvement in extracting relevant QA pairs.

• Question Answering: The performance of all LLMs—measured by precision, recall, F1, and GPT-4-Judge scores—is significantly higher than for QA pair extraction.7 This discrepancy is not surprising, as in the question answering task, the questions are explicitly provided in the prompt. High F1 scores (over 90%) show that most LLMs can follow instructions well and properly repeat questions.

Effect over Different Languages. Figure 2 (a) shows the GPT-4-Judge scores across three languages. Most models perform best in Japanese, but there is no clear advantage in either Chinese or English. A closer look reveals that LLMs are more

7One exception is LLaMA3.1-8B-Instruct, which has difficulty following instructions and often fails to repeat the questions, resulting in lower GPT-4-Judge scores for the question answering task.
consistent in Japanese, likely because they adhere to the instructions more effectively in this language. The three Qwen models perform similarly in the Summarization task across all languages, followed by the Question Answering task. However, their performance varies most in the QA pair extraction task, where Qwen2.5-72B-Instruct obtains the highest scores, outperforming Qwen2-7B-Instruct with GPT-4-Judge scores of 72.76, 63.59, and 83.45 for English, Chinese, and Japanese, respectively.

Effect over Different Sectors. Figure 2 (b) compares model performance across sectors. Communication Services, Consumer Discretionary, and IT generally achieve higher GPT-4-Judge scores in summarization and question answering. However, the performance trend becomes more complex for the QA pair extraction task, showing increased variability across sectors. Overall, the performance gaps among sectors for GPT-4o, Qwen2-72B-Instruct, and Qwen2.5-72B-Instruct are much smaller compared to GPT-3.5-turbo and LLaMA3.1-8B-Instruct.8 This suggests that the former models are less affected by sector differences.

Effect over Different Input Lengths. Figure 2 (c) compares the performance across varying input lengths. A key observation is the sharp drop in GPT-3.5-turbo’s performance when the input exceeds 15K tokens, due to its 16K token context limit. In contrast, both Qwen2.5-72B-Instruct and GPT-4o demonstrate stable and competent performance across the three tasks, particularly excelling in handling longer contexts exceeding 15K tokens. On the other hand, Qwen2-72B-Instruct shows a declining trend in the QA pair extraction task as input length increases, indicating a reduced capability to maintain performance with longer inputs in this specific task. Future research could explore structured modeling, as outlined by (Zhu et al., 2019), to improve handling of long input contexts.

Effect of RAG-based Question Answering. Instead of prompting the LLMs to answer a list of questions in a single response, we also explore RAG-based question answering, where the LLM answers questions individually based on retrieved document chunks. Following Wang et al. (2024), we divide the document into 1,024-token chunks. For embedding, we utilize the OpenAI Embedding model. Figure 3 compares the performance of

![Line chart comparing performance (GPT-4-Judge score) across different input lengths before or after adding RAG module.](page_1012_120_388_282.png)

Figure 3: Performance (in GPT-4-Judge score) across different input lengths before or after adding RAG module. Baseline 1 refers to prompting the LLM to answer a list of questions, while Baseline 2 refers to answering one question per response.

Qwen2.5-72B-Instruct before and after the integration of the RAG module. This comparison is based on a random selection of 10 meetings from each length set, resulting in a total of 50 meetings. The results indicate that for documents exceeding 15K tokens, answering all questions in a single response (Baseline 1) outperforms all other variants that answer questions one at a time. Additionally, for variants that respond to one question at a time in documents longer than 10K tokens, we observe that a larger context leads to better performance, specifically: Baseline 2 > RAG (top 5) > RAG (top 3) > RAG (top 1). Notably, RAG (with top 5) only surpasses the non-RAG variants for documents shorter than 10K tokens.

8GPT-3.5-turbo fails for the Utilities sector due to input length exceeding its 16K token limit.

4.3 Performance Evaluated by Qwen-plus-Judge

In addition to the performance assessed by the GPT-4-Judge, as presented in Table 5, Table 6 shows the performance evaluated by the Qwen-Plus-Judge. The prompt templates used for the Qwen-Plus-Judge are the same as those for GPT-4-Judge, as illustrated in Figure 8 and Figure 9. The performance trends in Table 6 align closely with those in Table 5, where Qwen2.5-72B-Instruct remains the top-performing model, followed by GPT-4o and Qwen2-72B-Instruct, which show similar performance. Next are Qwen2-7B-Instruct and GLM4-9B-Chat, both of which also exhibit comparable performance. This consistency further demonstrates that Qwen-Plus-Judge is a reliable alternative evaluator.
<table>
  <tr>
    <th>Model</th>
    <th>Summarization</th>
    <th>QA Pair Extraction</th>
    <th>Question Answering</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>76.01</td>
    <td>67.02</td>
    <td>69.84</td>
    <td>71.12</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td>
    <td>45.46</td>
    <td>31.43</td>
    <td>37.88</td>
    <td>38.48</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td>
    <td>68.20</td>
    <td>47.06</td>
    <td>60.39</td>
    <td>58.86</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td>
    <td>51.46</td>
    <td>43.90</td>
    <td>32.05</td>
    <td>42.75</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td>
    <td>75.39</td>
    <td>42.65</td>
    <td>67.87</td>
    <td>62.38</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td>
    <td>76.22</td>
    <td>65.58</td>
    <td>70.73</td>
    <td>71.01</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td><b>76.99</b></td>
    <td><b>70.25</b></td>
    <td><b>74.01</b></td>
    <td><b>73.85</b></td>
  </tr>
</table>

Table 6: The results using Qwen-plus as the judge model.

<table>
  <tr>
    <th>Evaluation Method</th>
    <th>Summarization</th>
    <th>QA Pair Extraction</th>
    <th>Question Answering</th>
  </tr>
  <tr>
    <td>GPT-4-Judge</td>
    <td>72.83</td>
    <td>60.76</td>
    <td>66.44</td>
  </tr>
  <tr>
    <td>Human Annotators</td>
    <td>3.68</td>
    <td>3.13</td>
    <td>3.36</td>
  </tr>
</table>

Table 7: Comparison of average performance ratings between GPT-4-Judge and human annotators. Note that GPT-4-Judge uses a 1-100 scale, while human annotators use a 1-5 scale.

<table>
  <tr>
    <th>Agreement</th>
    <th>Kappa</th>
  </tr>
  <tr>
    <td>GPT-4-Judge & Human Annotators</td>
    <td>0.701</td>
  </tr>
  <tr>
    <td>Human Annotators</td>
    <td>0.650</td>
  </tr>
</table>

Table 8: Fleiss’ Kappa score between GPT-4-Judge and human annotators.

4.4 Human Evaluation and Fleiss’ Kappa Agreement Between GPT-4-Judge and Human Evaluators

We randomly select 100 meetings and recruit five expert human annotators to assess the overall quality of GPT-4o’s responses, including summarization, QA pair extraction, and question answering. Each response is rated on a scale from 1 to 5, based on the same five criteria used to prompt GPT-4-Judge. The final score for each response is the average rating across the five annotators. Table 7 compares the performance assessed by GPT-4-Judge and human annotators.

Moreover, to assess the agreement among the six evaluators (five human annotators and GPT-4-Judge), we compute Fleiss’ Kappa score, a metric for inter-annotator agreement. Specifically, we calculate two types of Kappa scores: 1) the agreement among all six evaluators, where GPT-4-Judge’s ratings (on a 1-100 scale) are converted to a 1-5 scale by dividing by 20; and 2) the agreement among the five human annotators. As shown in Table 8, the agreement among GPT-4-Judge and human annotators is still higher than that among humans.

4.5 Performance in BLEU and ROUGE

The analysis of model performances, based on BLEU and ROUGE metrics, highlights notable differences across various tasks. Specifically, Qwen2-72B-Instruct excels in the summarization task, consistently generating concise and coherent summaries that underscore its strength in synthesizing information. Meanwhile, Qwen2.5-72B-Instruct leads in both QA pair extraction and question answering, demonstrating its adeptness at understanding queries and providing precise responses. This comparison underscores the unique advantages and task-specific expertise of each model, offering a more comprehensive insight into their capabilities. For detailed data and specific scores, please refer to the full table located in the appendix D.

5 Conclusion

In this paper, we have introduced M^3FinMeeting, a novel multilingual, multi-sector, and multi-task benchmark specifically designed to evaluate financial meeting understanding in large language models (LLMs). By incorporating real-world dialogue from financial meetings, our dataset fills significant gaps in existing benchmarks, which often rely on static sources like news articles and earnings reports. Supporting English, Chinese, and Japanese, and encompassing 11 industry sectors defined by GICS, M^3FinMeeting enables a comprehensive evaluation of LLMs through tasks such as summarization, question-answer pair extraction, and question answering. Experimental results with seven representative LLMs, including GPT-4o and Qwen2.5-72B-Instruct, demonstrate notable performance challenges, highlighting significant room for improvement and establishing M^3FinMeeting as a valuable resource for advancing research in financial language processing.
Acknowledgments

The authors express their heartfelt gratitude to the anonymous reviewers for their invaluable feedback and insightful comments, which greatly enhanced the quality of this work. We also offer our sincere appreciation to the members of the Qwen DianJin Team for their exceptional contributions, dedication, and hard work, which were instrumental in the success of this project. This work was supported by the Alibaba Innovative Research Program.

Limitations

Our work has several limitations.

• High Annotation Costs and Challenges: Both summarization and question-answer pair extraction require annotators to summarize content and extract question-answer pairs from audio recordings lasting 1-2 hours and text exceeding 10K tokens. This process relies heavily on the annotators’ professional expertise. Summarization, in particular, demands a high level of skill from professional analysts and often involves open-ended responses. Consequently, the annotation process necessitates a significant investment of time and effort.

• Limited Coverage for Question Answering: In the question-answering task, we focus exclusively from the extracted QA pairs. This limits our ability to evaluate the LLMs’ capacity to search for evidence within the document beyond the provided questions. As a result, the dataset may not fully capture the models’ potential in more complex reasoning scenarios that require deeper comprehension of the content.

• Evaluation Issues: The performance of summary alignment in summarization and question alignment in both question-answer pair extraction and question answering relies on the embeddings used. Due to budget constraints, we have not conducted a more extensive evaluation, which may affect the robustness of our findings.

Ethics

Our dataset is sourced from publicly mandated disclosures, such as earnings calls and roadshows. While this information is publicly available, we anonymize it to respect corporate preferences for dissemination. This ensures both the academic utility and ethical integrity of our benchmark. Specifically, we remove company identifiers and sensitive details using GPT-4, followed by manual verification, to maintain privacy without compromising research value. All data annotators are part of the funded projects, ensuring consistent and responsible data handling. The dataset, excluding original audio files, will be available online.

References

Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Technology Association Workshop, pages 84–90.

Willy Au, Abderrahim Ait-Azzi, and Juyeon Kang. 2021. Finsbd-2021: The 3rd shared task on structure boundary detection in unstructured text in the financial domain. In Companion Proceedings of WWW, page 276–279.

Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024a. MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of ACL, pages 7421–7454.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024b. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of ACL, pages 3119–3137.

Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaikos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. 2005. The ami meeting corpus: a pre-announcement. In Proceedings of MLMI, page 28–39.

Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024. FinTextQA: A dataset for long-form financial question answering. In Proceedings of ACL, pages 6025–6047.

Zhiyu Chen, Wenhui Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of EMNLP, pages 3697–3711.

Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the chain of numerical reasoning
in conversational finance question answering. In Proceedings of EMNLP, pages 6279–6292.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. Computing Research Repository, arXiv:2407.21783.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mine: A comprehensive evaluation benchmark for multimodal large language models. Computing Research Repository, arXiv:2306.13339.

Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. Computing Research Repository, arXiv:2209.12356.

Cuiyun Han, Jinchuan Zhang, Xinyu Li, Guojin Xu, Weihua Peng, and Zengfeng Zeng. 2022. Duee-fin: A large-scale dataset for document-level event extraction. In Proceedings of NLPC, pages 172–183.

Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. Computing Research Repository, arXiv:2311.11944.

Adam Janin, Don Baaron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, and Chunk Wooters. 2003. The icsi meeting corpus. In Proceedings of ICASSP, pages 364–367.

Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum, and Tianmin Shu. 2024. MMToM-QA: Multimodal theory of mind question answering. In Proceedings of ACL, pages 16077–16102.

Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022. An empirical survey on long document summarization: Datasets, models, and metrics. ACM Comput. Surv., 55(8).

Michael Krumdick, Rik Koncel-Kedziorski, Viet Lai, Varshini Reddy, Charles Lovering, and Chris Tanner. 2024. BizBench: A quantitative reasoning benchmark for business and finance. In Proceedings of ACL, pages 8309–8332.

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024. LooGLE: Can long-context language models understand long contexts? In Proceedings of ACL, pages 16304–16333.

Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. 2023. M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. Computing Research Repository, arXiv:2306.04387.

Chin-Yew Lin and Eduard Hovy. 2002. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of ACL, page 311–318.

Ran Liu, Ming Liu, Min Yu, He Zhang, Jianguo Jiang, Gang Li, and Weiqing Huang. 2024a. SumSurvey: An abstractive dataset of scientific survey papers for long document summarization. In Findings of ACL, pages 9632–9651.

Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024b. Calibrating llm-based evaluator. In Proceedings of LREC-COLING, pages 2638–2656.

Dakuan Lu, Hengkui Wu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng, Mengkun Han, Yingsi Xin, and Yanghua Xiao. 2023. Bbt-fin: Comprehensive construction of chinese financial domain pre-trained language model, corpus and benchmark. Computing Research Repository, arXiv:2302.09432.

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: Financial opinion mining and question answering. In Companion Proceedings of WWW, page 1941–1942.

Pekka Malo, Ankur Sinha, Pyry Takala, Pekka Korhonen, and Jyrki Wallenius. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the American Society for Information Science and Technology, 65:782–796.

Rajdeep Mukherjee, Abhinav Bohra, Akash Banerjee, Soumya Sharma, Manjunath Hegde, Afreen Shaikh, Shivani Shrivastava, Koustuv Dasgupta, Niloy Ganguly, Saptarshi Ghosh, and Pawan Goyal. 2022. ECT-Sum: A new benchmark dataset for bullet point summarization of long earnings call transcripts. In Proceedings of EMNLP, pages 10893–10906.

OpenAI. 2023. Gpt-4 technical report. Computing Research Repository, arXiv:2303.08774.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL, page 311–318.

Vaidehi Patil, Leonardo Ribeiro, Mengwen Liu, Mohit Bansal, and Markus Dreyer. 2024. REFINESUMM: Self-refining MLLM for generating a multimodal summarization dataset. In Proceedings of ACL, pages 13773–13786.

Tzuf Paz-Argaman, Itai Mondshine, Asaf Achi Mordechai, and Reut Tsarfaty. 2024. HeSum: a novel dataset for abstractive text summarization in Hebrew. In Findings of ACL, pages 6378–6388.

Archiki Prasad, Trung Bui, Seunghyun Yoon, Hanieh Deilamsalehy, Franck Dernoncourt, and Mohit Bansal. 2023. MeetingQA: Extractive question-answering on meeting transcripts. In Proceedings of ACL, pages 15000–15025.
Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. Computing Research Repository, arXiv:2309.09558.

Rohit Saxena and Frank Keller. 2024. MovieSum: An abstractive summarization dataset for movie screenplays. In Findings of ACL, pages 4043–4050.

William A. Scott. 1995. Reliability of content analysis: The case of nominal scale coding. The Public Opinion Quarterly, 19:321–325.

Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charsee Smiley, Jiaao Chen, and Diyi Yang. 2022. When flue meets flang: Benchmarks and large pretrained language model for financial domain. In Proceedings of EMNLP, pages 2322–2335.

Ankur Sinha and Tanmay Khandait. 2020. Impact of news on the commodity market: Dataset and results. Computing Research Repository, arXiv:2009.04202.

Yejun Soun, Jaemin Yoo, Minyoung Cho, Jihyeong Jeon, , and U Kang. 2022. Accurate stock movement prediction with self-supervised learning from sparse noisy tweets. In Proceedings of CKIM, pages 1691–1700.

Tianchi. 2019. Ccks2019 financial domain document-level event entity extraction dataset. https://tianchi.aliyun.com/dataset/111237.

Tianchi. 2020. Ccks2020 financial domain document-level event entity extraction dataset. https://tianchi.aliyun.com/dataset/111209.

Tianchi. 2021. Ccks2021 financial domain event causal relationship extraction dataset. https://tianchi.aliyun.com/dataset/110901.

Tianchi. 2022. Ccks2022 financial domain few-shot event extraction dataset. https://tianchi.aliyun.com/dataset/136800.

Cuxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. 2023. Evaluating open-qa evaluation. In Proceedings of NeurIPS, page 36.

Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. Computing Research Repository, arXiv:2406.17419.

Huizhe Wu, Wei Zhang, Weiwei Shen, and Jun Wang. 2018. Hybrid deep sequential modeling for social text-driven stock prediction. In Proceedings of CKIM, pages 1627–1630.

Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2024. Pixiu: a large language model, instruction data and evaluation benchmark for finance. In Proceedings of NerIPS, pages 33469–33484.

Yumo Xu and Shay B. Cohen. 2018. Stock movement prediction from tweets and historical prices. In Proceedings of ACL, pages 1970–1979.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. 2024a. Qwen2 technical report. Computing Research Repository, arXiv:2407.10671.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. 2024b. Qwen2.5 technical report. Computing Research Repository, arXiv:2412.15115.

Hang Yang, Yubo Chen, Kang Liu, Yang Xiao, and Jun Zhao. 2018. DCFEE: A document-level Chinese financial event extraction system based on automatically labeled training data. In Proceedings of ACL: System Demonstrations, pages 50–55.

Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023a. Fingpt: Open-source financial large language models. Computing Research Repository, arXiv:2306.06031.

Yi Yang, Yixuan Tang, and Kar Yan Tam. 2023b. Investlm: A large language model for investment using financial domain instruction tuning. Computing Research Repository, arXiv:2309.13064.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glim-130b: An open bilingual pre-trained model. Computing Research Repository, arXiv:2210.02414.

Tian Zhang, Maofu Liu, and Bingying Zhou. 2024a. Chindee: A chinese fine-grained financial dataset for document-level event extraction. In Companion Proceedings of the ACM Web Conference, pages 1511–1520.

Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024b. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57.

Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023. Wider and deeper llm networks are fairer llm evaluators. Computing Research Repository, arXiv:2308.01862.

Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019. Doc2EDAG: An end-to-end document-level framework for Chinese financial event extraction. In Proceedings of EMNLP-IJCNLP, pages 337–346.

Zhihan Zhou, Liqian Ma, and Han Liu. 2021. Trade the event: Corporate events detection for news-based event-driven trading. In Findings of ACL-IJCNLP, pages 2114–2124.

Jie Zhu, Qian Chen, Huaixia Dou, Junhui Li, Lifan Guo, Feng Chen, and Chi Zhang. 2025. Dianjin-r1: Evaluating and enhancing financial reasoning in large language models. arXiv preprint arXiv:2504.15716.
Jie Zhu, Junhui Li, Yalong Wen, and Lifan Guo. 2024. Benchmarking large language models on CFLUE - a Chinese financial language understanding evaluation dataset. In Findings of ACL, pages 5673–5693.

Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019. Modeling graph structure in transformer for better AMR-to-text generation. In Proceedings of EMNLP, pages 5459–5468, Hong Kong, China.

A Examples of M^3FinMeeting

For brevity, Figure 4 presents a screenshot of an annotated M^3FinMeeting example. Complete examples are available in the attached file.

B Details of Precision, Recall and F1 Metrics

For summarization, as shown in Figure 1, we align the generated and gold section summaries based on a cosine similarity score above 0.75. Let us assume that there are \( m \) section summaries in the generated summarization and \( n \) section summaries in the reference (gold) summarization. After aligning the section summaries between the two, let \( m_a \) represent the number of generated section summaries aligned, and \( n_a \) represent the number of gold section summarized aligned. Note that \( m_a \) and \( n_a \) do not need to be equal, as one section summary from one side can be aligned to multiple section summaries on the other side. We compute the Precision, Recall, and F1 as follows:

\[
\text{Precision} = \frac{m_a}{m},
\]
(1)

\[
\text{Recall} = \frac{n_a}{n},
\]
(2)

\[
\text{F1} = \frac{2 * \text{Precision} * \text{Recall}}{\text{Precision} + \text{Recall}}.
\]
(3)

The Precision, Recall, and F1 scores for QA pair extraction and question answering are computed similarly, with the generated and gold section summaries replaced by generated and gold questions.

C Prompt Examples for Tasks in M^3FinMeeting

We use the same prompt templates, written in English, for English, Japanese, and Chinese. In these templates, we clearly specify that the output language must align with the meeting content.

Figure 5 and Figure 6 show the prompt template for the summarization task and the QA pair extraction task, respectively. Figure 7 displays the prompt template for the question answering task. In this template, all questions from the document are listed, and the LLMs are instructed to answer them in a single response.

When evaluating with GPT-4-Judge, Figure 8 presents the prompt template for the summarization task. Meanwhile, Figure 9 displays the prompt template for both the QA pair extraction task and the question-answering task.

D Performance in BLEU and ROUGE

Table 9 shows the overall performance in BLEU and ROUGE. It shows that Qwen2-72B-Instruct achieves the best performance in summarization while Qwen2.5-72B-Instruct leads in both QA pair extraction and question answering.

E Detailed Performance Across Languages, Lengths, and GICS Sectors

Table 10, Table 11, Table 12 show the performance of LLMs with respect to the languages, the length sets, and the GICS sectors, respectively.

F Annotation Guidelines

Figure 10 shows the annotation guideline of M^3FinMeeting.
{
    "Meeting_ID": "M_EN_001",
    "GICS_Industry": "Industrials",
    "Language": "EN",
    "ASR_Text": {
        "content": "Honorable Investors, good evening. Thank you for attending the [Company1] 2022 Annual Results Conference. I am [Role1]."
    },
    "Structured_Summary": {
        "content": [
            {
                "text": "In 2022, [Company1] achieved revenue of 267.5 billion, with a 29% year-on-year growth, and a net profit of 6.17 bil"
            },
            {
                "text": "Consumer business growth was 17%, accounting for 52%, driven primarily by sectors like apparel, footwear, beauty, a"
            },
            {
                "text": "Local business significantly reduced losses, improving net profit by 8.2 percentage points. International business"
            }
        ]
    },
    "QA_List": {
        "content": [
            {
                "question": "I am [Role1] from [Company1], thankful for leaders' time and insights on the time-sensitive express business. I"
                "answer": "Respected questioner, thank you for your focus. As outlined earlier, from both personal and company perspectives,"
            },
            {
                "question": "I am [Role3] from [Company2], expressing gratitude to [Company1] executives [Role4], [Role5], [Role6], and [Rol"
                "answer": "For this inquiry, let's have [Role9] respond. Firstly, regarding the overarching industry landscape, mergers like"
            },
            {
                "question": "Dear [Company1] leaders, good evening. I'm [Role18] from [Company6]. I'd like to ask about [Company1] and [Comp"
                "answer": "At a strategic level, business core positioning aims not to align cycles to maintain group steadiness, as reflect"
            },
            {
                "question": "Could the company provide us with guidance over this year's performance as a reference? Specifically, during Q1"
                "answer": "Thank you for the inquiry. The question pertains to yearly and first-quarter directionality, addressed separately"
            }
        ]
    }
},

Figure 4: Example of annotated M^3FinMeeting.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Summarization</th>
    <th colspan="4">QA Pair Extraction</th>
    <th colspan="4">Question Answering</th>
  </tr>
  <tr>
    <th>B-4</th>
    <th>R-1</th>
    <th>R-2</th>
    <th>R-L</th>
    <th>B-4</th>
    <th>R-1</th>
    <th>R-2</th>
    <th>R-L</th>
    <th>B-4</th>
    <th>R-1</th>
    <th>R-2</th>
    <th>R-L</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>9.47</td>
    <td>49.32</td>
    <td>19.02</td>
    <td>29.23</td>
    <td>1.58</td>
    <td>8.40</td>
    <td>3.57</td>
    <td>5.94</td>
    <td>5.21</td>
    <td>27.53</td>
    <td>11.50</td>
    <td>19.65</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td>
    <td>6.03</td>
    <td>35.02</td>
    <td>12.87</td>
    <td>20.05</td>
    <td>0.45</td>
    <td>2.38</td>
    <td>1.02</td>
    <td>1.71</td>
    <td>2.84</td>
    <td>16.55</td>
    <td>6.27</td>
    <td>11.72</td>
  </tr>
  <tr>
    <td>GLM-9B-Chat</td>
    <td>9.27</td>
    <td>47.35</td>
    <td>18.89</td>
    <td>29.59</td>
    <td>1.78</td>
    <td>5.26</td>
    <td>2.59</td>
    <td>3.80</td>
    <td>5.80</td>
    <td>28.01</td>
    <td>11.66</td>
    <td>20.27</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td>
    <td>4.77</td>
    <td>36.37</td>
    <td>11.82</td>
    <td>21.38</td>
    <td>1.16</td>
    <td>3.97</td>
    <td>1.93</td>
    <td>2.98</td>
    <td>2.76</td>
    <td>14.20</td>
    <td>5.98</td>
    <td>10.58</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td>
    <td>14.72</td>
    <td>55.63</td>
    <td>24.12</td>
    <td>35.98</td>
    <td>0.34</td>
    <td>2.00</td>
    <td>0.75</td>
    <td>1.36</td>
    <td>6.67</td>
    <td>29.78</td>
    <td>11.87</td>
    <td>20.04</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td>
    <td><b>14.89</b></td>
    <td><b>56.35</b></td>
    <td><b>24.39</b></td>
    <td><b>36.37</b></td>
    <td>1.65</td>
    <td>6.66</td>
    <td>3.04</td>
    <td>4.81</td>
    <td>6.90</td>
    <td>30.39</td>
    <td>12.22</td>
    <td>21.15</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>11.71</td>
    <td>51.92</td>
    <td>20.84</td>
    <td>32.76</td>
    <td><b>2.71</b></td>
    <td><b>11.32</b></td>
    <td><b>5.04</b></td>
    <td><b>7.91</b></td>
    <td><b>7.44</b></td>
    <td><b>32.35</b></td>
    <td><b>13.65</b></td>
    <td><b>22.43</b></td>
  </tr>
</table>

Table 9: Performance (in BLEU and ROUGE) of LLMs on three evaluation tasks. Here B-4 is for BLEU-4, R-1/2/L for ROUGE-1/2/L.
## Role:
You are a meeting content analyst responsible for summarizing the core information from meetings. Please analyze based on the following requirements.

## Skill:
- Content Focus: Concentrate on substantive discussions, decisions, and key data from the meeting, ignoring procedural content, opening introductions, and host speeches.
- Information Extraction: Extract important financial metrics, market dynamics, management strategies, and interactive content from the provided meeting text. Digital information should be quoted accurately from the original text without any unauthorized additions.
- Formatted Output: Present the summary results in JSON format, ensuring that each summary element is complete and informative.
- Level of Detail: Ensure that the summary content is rich, and consolidate similar themes into a single paragraph to cover as many key insights as possible.

## Constraints:
- Ignore procedural content, opening introductions, and host remarks that lack substantial significance, and focus on substantive discussions and decisions.
- The content should be as thorough as possible, merging similar themes for clarity.
- The output language should match the input language.

## Meeting Text:
{meeting_note}

## OutputFormat:
- The output should follow the specified JSON structure:
[
    {
        "text": "Extracted key summary paragraph"
    },
    ...
]

Figure 5: Prompt template used for the summarization task.
### Skills:
- Precise Information Extraction: Distinguishing and extracting each specific question and answer from extensive meeting records.
- Content Rewriting Ability: Transforming colloquial meeting dialogue into clear, written expression.
- Structured Data Output: Strictly organizing and outputting all Q&A pairs according to JSON format standards, satisfying detailed format requirements.

## Constraints:
- Maintain the integrity and accuracy of the information, omitting no questions or answers.
- Rewrite content to streamline questions while ensuring the richness of the answers.
- Must adhere to the specified JSON output format.
- Output language must be consistent with the input meeting notes.

## Workflow:
- First, thoroughly review the meeting content to identify and separate each independent question and response paragraph.
- Next, rewrite each paragraph to ensure questions are concise and answers are detailed and formal.
- Finally, consolidate the processed information into JSON format, forming an organized list of Q&A pairs.

## Output Format:
- The output should be a JSON array containing multiple Q&A objects, each with "question" and "answer" sections, for example:
```json
[
    {
        "question": "",
        "answer": ""
    },
    {
        # second Q&A pair
    }
]
```

## Meeting Notes
{meeting_notes}

Figure 6: Prompt template used for the QA pair extraction task.
## Role:
Meeting Content Analysis Expert: Possesses the ability to deeply understand lengthy meeting texts and can accurately extract information to answer a list of questions related to the meeting.

## Skill:
-- Efficient Text Summarization: Quickly distills key information and details from meeting texts.
-- Question Response Matching: Precisely identifies relevant answers in the meeting content for each question on the list.

## Constraints:
-- Input Limitation: Long meeting text and question list have been provided.
-- Output Requirement: Answers to each question in the list formatted in JSON.
-- Output Language: Output language must remain consistent with the language of the meeting text.

## Workflow:
-- First, conduct a comprehensive analysis of the meeting text to extract core points.
-- Then, meticulously compare the content of the meeting against each question in the question list to find corresponding answers.
-- Finally, organize the answers to ensure they are direct and accurate for each question.

## Meeting Text:
{meeting_note}

## Question List:
{question_list}

## Output Format:
-- The output language must be consistent with that of the meeting text.
-- The output should follow the JSON structure below:
[
    {
        "question": "First question",
        "answer": "Answer to the first question"
    },
    ...
]
]

Figure 7: Prompt template used for the question answering task. All questions from a document are listed, and the LLMs are prompted to answer them in a single response.
## Role:
Evaluation Expert: A professionally competent AI assistant evaluation specialist, proficient in accurately assessing the quality of AI assistant responses based on established criteria, ensuring the accuracy and objectivity of the evaluation.

## Gold Answer:
{gold_answer}

## Assistant’s Predicted Answer:
{predicted_answer}

## Skill:
- Precision Matching: Accurately identify semantic consistency, numerical and sequential accuracy, and the presence of hallucinations in the AI assistant's response based on a gold standard answer.
- Key Point Extraction: Quickly identify and verify whether the response includes all necessary key information to meet the completeness requirements of the question answered.
- Unbiased Evaluation: Provide comprehensive and impartial evaluation explanations, strictly following scoring criteria to deliver a fair score.

## Constraints:
- Evaluation Criteria: The assessment is limited to the following core aspects:
  1. Coverage: The summary should include key information from the source document, ensuring a comprehensive reflection of the original content.
  2. Redundancy: The summary should avoid unnecessary repetition or lengthy expressions, such as repeated sentences or overused noun phrases.
  3. Readability: The summary should remain fluent and understandable, with clear logic and well-organized information, avoiding ambiguities.
  4. Accuracy: The truthfulness and correctness of the information must be guaranteed, ensuring that numbers, facts, and descriptions are consistent with the source document.
  5. Consistency: The information in the summary should be logically consistent throughout, avoiding contradictions or conflicting information.

## Workflow:
- Analyze and compare the assistant’s response with the gold standard answer.
- Determine the relevant compliance level according to the established evaluation criteria.
- Provide the final score, adhering to the specified format for output.

## Output Format:
- Scores should be integers from 1 to 100.
- Strictly follow the specified JSON format without any additional explanations.
{
    "coverage": "",   // Coverage score
    "redundancy": "", // Redundancy score
    "readability": "", // Readability score
    "accuracy": "",   // Accuracy score
    "consistency": "", // Consistency score
    "overall": ""     // Overall score
}
## Role:
- Evaluation Expert: A professionally competent AI assistant evaluation specialist, proficient in accurately assessing the quality of Q&A pairs extracted by the AI assistant from meeting texts based on established criteria, ensuring the accuracy and objectivity of the evaluation.

## Gold Answer:
{gold_answer}

## Assistant’s Predicted Answer:
{predicted_answer}

## Skill:
- Precision Matching: Accurately identify semantic consistency, numerical and sequential accuracy, and the presence of hallucinations in the Q&A pairs extracted by the AI assistant based on the Gold Answer.
- Key Point Extraction: Quickly identify and verify whether the Q&A pairs contain all necessary information and context to meet the completeness requirements of the question answered.
- Unbiased Evaluation: Provide comprehensive and impartial evaluation explanations, strictly following scoring criteria to deliver a fair score.

## Constraints:
- Evaluation Criteria: The assessment is limited to the following core aspects:
  1. Coverage: The Q&A pairs should include key information present in the Gold Answer, maintaining the same quantity and ensuring a comprehensive reflection of the original content.
  2. Redundancy: The Q&A pairs should avoid unnecessary repetition, lengthy expressions, or content exceeding that of the Gold Answer, remaining concise and clear.
  3. Readability: The Q&A pairs should be fluent and understandable, logically clear, with a well-structured information layout, facilitating comprehension.
  4. Accuracy: The truthfulness and correctness of the information must be guaranteed, ensuring that the facts and descriptions in the Q&A pairs are consistent with the Gold Answer.
  5. Consistency: The question and answer in the Q&A pairs should be logically consistent throughout, avoiding contradictions or conflicting information, and align in language with the Gold Answer.

## Workflow:
- Analyze and compare the Q&A pairs extracted by the assistant with the gold standard Q&A pairs.
- Determine the relevant compliance level according to the established evaluation criteria.
- Provide the final score, adhering to the specified format for output.

## OutputFormat:
- Scores should be integers from 1 to 100.
- Strictly follow the specified JSON format without any additional explanations.
{
    "coverage": "",   // Coverage score
    "redundancy": "",  // Redundancy score
    "readability": "",  // Readability score
    "accuracy": "",   // Accuracy score
    "consistency": "",  // Consistency score
    "overall": ""     // Overall score
}

Figure 9: Prompt template utilized for assessing both QA pair extraction and question answering with GPT-4-Judge.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Summarization</th>
    <th colspan="4">QA Pair Extraction</th>
    <th colspan="4">Question Answering</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>P.</th><th>R.</th><th>FI</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>FI</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>FI</th><th>GPT-4</th>
    <th>GPT-4</th>
  </tr>
  <tr>
    <th colspan="15">English</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>5.58</td><td>3.62</td><td>4.39</td><td>72.83</td><td>40.12</td><td>19.21</td><td>25.98</td><td>68.37</td><td>98.25</td><td><b>98.25</b></td><td>98.25</td><td>67.02</td><td>69.41</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td><td>6.84</td><td>4.17</td><td>5.18</td><td>33.68</td><td>19.32</td><td>3.31</td><td>5.65</td><td>26.70</td><td><b>98.58</b></td><td>36.27</td><td>53.03</td><td>35.68</td><td>32.02</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>0.63</td><td>3.29</td><td>1.06</td><td>60.45</td><td>2.54</td><td>7.31</td><td>3.77</td><td>39.69</td><td>98.31</td><td>97.62</td><td>97.96</td><td>69.76</td><td>56.63</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>1.37</td><td>1.65</td><td>1.49</td><td>45.98</td><td>5.78</td><td>9.05</td><td>7.06</td><td>45.53</td><td>87.09</td><td>85.02</td><td>86.04</td><td>52.75</td><td>48.04</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>13.15</td><td>12.84</td><td>12.99</td><td>72.71</td><td>3.91</td><td>2.03</td><td>2.67</td><td>29.85</td><td>97.75</td><td>86.01</td><td>91.50</td><td>71.95</td><td>58.16</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td><b>15.43</b></td><td><b>16.02</b></td><td><b>15.72</b></td><td>71.92</td><td>15.26</td><td>10.38</td><td>12.36</td><td>54.85</td><td>98.31</td><td>94.89</td><td>96.57</td><td>75.03</td><td>67.26</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>6.40</td><td>4.72</td><td>5.51</td><td><b>73.94</b></td><td>39.81</td><td><b>22.57</b></td><td><b>28.81</b></td><td><b>72.76</b></td><td>98.22</td><td>96.45</td><td>97.33</td><td><b>76.76</b></td><td><b>74.48</b></td>
  </tr>
  <tr>
    <th colspan="15">Chinese</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>31.49</td><td>11.55</td><td>16.90</td><td>72.03</td><td>40.76</td><td>23.22</td><td>29.58</td><td>63.43</td><td>91.49</td><td>91.86</td><td>91.68</td><td>69.33</td><td>68.36</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td><td>18.34</td><td>4.52</td><td>7.24</td><td>48.42</td><td>25.11</td><td>6.64</td><td>10.51</td><td>31.41</td><td>92.20</td><td>59.67</td><td>72.45</td><td>47.42</td><td>42.87</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>2.14</td><td>10.49</td><td>14.31</td><td>67.90</td><td>3.31</td><td>15.18</td><td>7.86</td><td>43.45</td><td>90.32</td><td>91.61</td><td>90.96</td><td>64.14</td><td>58.81</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>2.03</td><td>8.35</td><td>8.23</td><td>66.61</td><td>4.20</td><td>12.80</td><td>8.06</td><td>30.00</td><td>90.93</td><td>91.93</td><td>92.34</td><td>67.07</td><td>59.38</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>28.69</td><td>16.22</td><td>20.72</td><td>71.68</td><td>20.89</td><td>12.74</td><td>15.83</td><td>40.47</td><td>91.66</td><td>88.38</td><td>89.99</td><td>66.75</td><td>59.95</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>30.21</td><td><b>16.99</b></td><td><b>21.75</b></td><td>72.48</td><td>32.20</td><td>25.97</td><td>28.75</td><td>58.85</td><td>92.02</td><td>91.83</td><td>91.92</td><td>70.57</td><td>67.43</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td><b>32.38</b></td><td>14.63</td><td>20.16</td><td><b>72.74</b></td><td><b>45.82</b></td><td><b>34.47</b></td><td><b>39.34</b></td><td><b>63.59</b></td><td><b>92.22</b></td><td><b>92.48</b></td><td><b>92.35</b></td><td><b>71.88</b></td><td><b>69.49</b></td>
  </tr>
  <tr>
    <th colspan="15">Japanese</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>37.03</td><td>25.03</td><td>29.87</td><td>80.06</td><td>61.04</td><td>35.51</td><td>44.89</td><td>81.50</td><td>90.95</td><td><b>90.86</b></td><td><b>90.91</b></td><td>85.88</td><td>82.62</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td><td>24.14</td><td>16.77</td><td>19.79</td><td>39.75</td><td>41.95</td><td>5.53</td><td>9.77</td><td>35.55</td><td><b>91.98</b></td><td>17.98</td><td>30.09</td><td>27.36</td><td>34.68</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>33.65</td><td>22.15</td><td>26.71</td><td>74.32</td><td>19.85</td><td>29.33</td><td>23.67</td><td>66.88</td><td>90.56</td><td>90.31</td><td>90.43</td><td>82.43</td><td>74.52</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>3.12</td><td>6.51</td><td>4.22</td><td>44.90</td><td>39.34</td><td>26.56</td><td>31.71</td><td>66.66</td><td>91.07</td><td>90.40</td><td>90.74</td><td>52.30</td><td>53.79</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td><b>43.32</b></td><td><b>50.81</b></td><td><b>46.77</b></td><td>82.42</td><td>13.92</td><td>6.64</td><td>8.99</td><td>31.82</td><td>91.07</td><td>90.40</td><td>90.74</td><td>83.17</td><td>67.20</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>41.41</td><td>48.93</td><td>44.86</td><td><b>83.49</b></td><td>36.87</td><td>25.00</td><td>29.79</td><td>78.46</td><td>90.89</td><td>90.22</td><td>90.55</td><td>85.78</td><td>82.65</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>35.67</td><td>34.91</td><td>35.29</td><td>82.40</td><td><b>67.74</b></td><td><b>46.49</b></td><td><b>55.14</b></td><td><b>83.45</b></td><td>89.75</td><td>89.66</td><td>89.70</td><td><b>86.95</b></td><td><b>84.10</b></td>
  </tr>
</table>

Table 10: Performance of LLMs on three evaluation tasks with different languages.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Summarization</th>
    <th colspan="4">QA Pair Extraction</th>
    <th colspan="4">Question Answering</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>P.</th><th>R.</th><th>FI</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>FI</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>FI</th><th>GPT-4</th>
    <th>GPT-4</th>
  </tr>
  <tr>
    <th colspan="15">Set1 (0-5K)</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>25.00</td><td>28.52</td><td>26.64</td><td>69.75</td><td>36.90</td><td>24.29</td><td>29.30</td><td>69.55</td><td>93.70</td><td>93.70</td><td>93.70</td><td>64.33</td><td>65.25</td>
  </tr>
  <tr>
    <td>GPT3.5-turbo</td><td>18.25</td><td>28.52</td><td>22.26</td><td>58.76</td><td>27.06</td><td>15.29</td><td>19.54</td><td>44.38</td><td>93.70</td><td>93.70</td><td>93.70</td><td>62.67</td><td>55.89</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>23.10</td><td>26.17</td><td>24.54</td><td>66.76</td><td>14.27</td><td>16.19</td><td>15.17</td><td>39.20</td><td>93.77</td><td>90.26</td><td>91.98</td><td>57.33</td><td>56.24</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>10.63</td><td>19.14</td><td>13.67</td><td>54.35</td><td>16.94</td><td>17.24</td><td>17.09</td><td>39.90</td><td>15.84</td><td>38.23</td><td>22.40</td><td>31.81</td><td>42.96</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>25.25</td><td>39.84</td><td>30.91</td><td>67.18</td><td>26.33</td><td>17.84</td><td>21.27</td><td>50.95</td><td>93.54</td><td>93.40</td><td>93.47</td><td>65.62</td><td>61.93</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>23.82</td><td>39.45</td><td>29.71</td><td>70.29</td><td>25.78</td><td>23.69</td><td>24.69</td><td>56.48</td><td>93.55</td><td>93.55</td><td>93.55</td><td>66.24</td><td>64.74</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>28.02</td><td>37.11</td><td>31.93</td><td>70.40</td><td>41.60</td><td>33.43</td><td>37.07</td><td>60.93</td><td>93.70</td><td>93.70</td><td>93.70</td><td>67.07</td><td>66.74</td>
  </tr>
  <tr>
    <th colspan="15">Set2 (5-10K)</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>26.01</td><td>17.01</td><td>20.57</td><td>73.51</td><td>44.78</td><td>26.76</td><td>33.50</td><td>65.37</td><td>91.97</td><td>92.31</td><td>92.14</td><td>69.10</td><td>69.42</td>
  </tr>
  <tr>
    <td>GPT3.5-turbo</td><td>16.22</td><td>11.67</td><td>13.57</td><td>62.07</td><td>27.47</td><td>9.10</td><td>13.68</td><td>45.45</td><td>91.98</td><td>79.26</td><td>85.15</td><td>64.69</td><td>57.71</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>16.34</td><td>13.75</td><td>14.93</td><td>68.46</td><td>17.11</td><td>15.05</td><td>16.01</td><td>45.31</td><td>90.55</td><td>90.75</td><td>90.65</td><td>63.73</td><td>59.10</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>9.90</td><td>12.22</td><td>10.94</td><td>56.25</td><td>4.63</td><td>13.27</td><td>6.87</td><td>41.96</td><td>25.73</td><td>42.29</td><td>31.99</td><td>30.91</td><td>43.95</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>25.59</td><td>26.94</td><td>26.25</td><td>71.97</td><td>21.98</td><td>14.79</td><td>17.68</td><td>43.90</td><td>91.90</td><td>88.48</td><td>90.16</td><td>66.61</td><td>61.47</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>25.41</td><td>27.01</td><td>26.19</td><td>73.39</td><td>37.96</td><td>30.70</td><td>33.93</td><td>61.36</td><td>92.20</td><td>92.20</td><td>92.20</td><td>70.09</td><td>68.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>27.60</td><td>22.29</td><td>24.66</td><td>72.87</td><td>48.25</td><td>35.38</td><td>40.82</td><td>65.91</td><td>92.27</td><td>92.27</td><td>92.27</td><td>71.93</td><td>70.59</td>
  </tr>
  <tr>
    <th colspan="15">Set3 (10-15K)</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>29.80</td><td>11.30</td><td>16.39</td><td>73.54</td><td>43.45</td><td>24.04</td><td>30.96</td><td>67.56</td><td>91.90</td><td>93.48</td><td>92.68</td><td>72.54</td><td>70.49</td>
  </tr>
  <tr>
    <td>GPT3.5-turbo</td><td>17.76</td><td>6.59</td><td>9.61</td><td>61.43</td><td>22.18</td><td>6.31</td><td>9.83</td><td>38.86</td><td>93.90</td><td>63.30</td><td>77.03</td><td>56.94</td><td>53.22</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>16.05</td><td>9.92</td><td>10.03</td><td>68.80</td><td>3.01</td><td>17.21</td><td>5.45</td><td>48.28</td><td>11.86</td><td>94.18</td><td>92.67</td><td>68.44</td><td>61.42</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>9.93</td><td>8.80</td><td>8.07</td><td>51.80</td><td>3.85</td><td>14.01</td><td>10.85</td><td>38.14</td><td>38.14</td><td>52.85</td><td>52.85</td><td>52.85</td><td>44.48</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>28.48</td><td>17.65</td><td>17.79</td><td>74.21</td><td>14.55</td><td>10.11</td><td>11.93</td><td>36.04</td><td>92.26</td><td>88.86</td><td>90.53</td><td>70.89</td><td>60.06</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>30.68</td><td>18.97</td><td>23.44</td><td>73.43</td><td>32.47</td><td>25.39</td><td>28.49</td><td>61.37</td><td>92.62</td><td>91.72</td><td>92.17</td><td>74.14</td><td>69.66</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>29.57</td><td>13.97</td><td>18.98</td><td>74.41</td><td>47.71</td><td>35.19</td><td>40.50</td><td>67.44</td><td>92.30</td><td>91.83</td><td>92.06</td><td>75.89</td><td>72.69</td>
  </tr>
  <tr>
    <th colspan="15">Set4 (15-20K)</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>23.84</td><td>8.96</td><td>13.02</td><td>74.62</td><td>42.40</td><td>23.14</td><td>29.94</td><td>71.32</td><td>94.76</td><td>94.96</td><td>94.86</td><td>75.26</td><td>74.54</td>
  </tr>
  <tr>
    <td>GPT3.5-turbo</td><td>15.31</td><td>0.79</td><td>1.51</td><td>19.00</td><td>31.25</td><td>0.79</td><td>1.54</td><td>14.36</td><td>100.00</td><td>0.68</td><td>1.36</td><td>11.16</td><td>6.19</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>15.27</td><td>10.92</td><td>12.73</td><td>67.41</td><td>8.49</td><td>15.01</td><td>10.85</td><td>50.09</td><td>93.82</td><td>94.81</td><td>94.31</td><td>71.86</td><td>64.54</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>7.13</td><td>6.31</td><td>6.69</td><td>45.04</td><td>18.48</td><td>13.27</td><td>15.45</td><td>48.97</td><td>70.59</td><td>74.30</td><td>52.89</td><td>50.52</td><td>41.83</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>32.26</td><td>18.60</td><td>23.00</td><td>70.08</td><td>11.14</td><td>4.77</td><td>6.08</td><td>34.71</td><td>94.41</td><td>87.72</td><td>90.94</td><td>72.81</td><td>61.83</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>34.69</td><td>19.87</td><td>25.27</td><td>76.69</td><td>29.87</td><td>16.11</td><td>18.18</td><td>62.49</td><td>94.90</td><td>92.65</td><td>94.27</td><td>76.97</td><td>72.91</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>27.47</td><td>12.98</td><td>17.63</td><td>76.67</td><td>48.23</td><td>33.58</td><td>39.59</td><td>73.37</td><td>94.97</td><td>92.02</td><td>94.99</td><td>76.65</td><td>76.73</td>
  </tr>
  <tr>
    <th colspan="15">Set5 (>20K)</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>37.81</td><td>9.24</td><td>14.86</td><td>75.01</td><td>44.14</td><td>17.68</td><td>25.25</td><td>65.42</td><td>92.36</td><td>89.04</td><td>90.67</td><td>69.99</td><td>70.91</td>
  </tr>
  <tr>
    <td>GPT3.5-turbo</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td><td>3.90</td><td>8.02</td><td>5.25</td><td>65.84</td><td>6.18</td><td>11.42</td><td>8.02</td><td>41.12</td><td>90.19</td><td>92.27</td><td>91.22</td><td>70.89</td><td>58.63</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B-Instruct</td><td>1.89</td><td>5.73</td><td>2.84</td><td>52.82</td><td>9.45</td><td>10.68</td><td>10.03</td><td>45.14</td><td>77.85</td><td>77.35</td><td>77.60</td><td>47.42</td><td>47.13</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td>34.06</td><td>11.99</td><td>17.74</td><td>75.52</td><td>10.93</td><td>2.49</td><td>4.05</td><td>31.36</td><td>91.87</td><td>83.24</td><td>87.34</td><td>71.71</td><td>57.21</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>32.30</td><td>11.92</td><td>17.41</td><td>73.70</td><td>16.67</td><td>8.84</td><td>11.55</td><td>59.15</td><td>92.63</td><td>90.34</td><td>91.42</td><td>76.33</td><td>69.83</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>34.93</td><td>11.15</td><td>16.91</td><td>76.79</td><td>46.97</td><td>25.69</td><td>33.21</td><td>68.59</td><td>93.09</td><td>93.09</td><td>93.09</td><td>78.75</td><td>73.28</td>
  </tr>
</table>

Table 11: Performance of LLMs on three evaluation tasks with different length sets.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Summarization</th>
    <th colspan="4">QA Pair Extraction</th>
    <th colspan="4">Question Answering</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>P.</th><th>R.</th><th>F1</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>F1</th><th>GPT-4</th>
    <th>P.</th><th>R.</th><th>F1</th><th>GPT-4</th>
    <th>GPT-4</th>
  </tr>
  <tr>
    <td colspan="15">Communication Services</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>23.72</td><td>11.02</td><td>15.05</td><td>73.78</td><td>37.23</td><td>24.70</td><td>29.70</td><td>68.71</td><td>94.02</td><td>96.23</td><td>95.11</td><td>71.35</td><td>71.41</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>23.05</td><td><b>18.63</b></td><td><b>20.61</b></td><td>74.94</td><td>30.57</td><td>26.11</td><td>28.17</td><td>62.55</td><td><b>95.29</b></td><td><b>95.29</b></td><td><b>95.29</b></td><td>73.45</td><td>70.55</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>24.13</td><td>14.69</td><td>18.27</td><td>75.25</td><td>40.46</td><td>28.47</td><td>33.42</td><td>70.06</td><td>95.29</td><td>95.29</td><td>95.29</td><td>76.65</td><td>73.10</td>
  </tr>
  <tr>
    <td colspan="15">Consumer Discretionary</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>23.82</td><td>10.85</td><td>14.91</td><td>72.24</td><td>41.86</td><td>22.16</td><td>28.98</td><td>64.63</td><td>93.04</td><td>94.49</td><td>93.76</td><td>69.22</td><td>68.74</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td><b>25.04</b></td><td><b>17.08</b></td><td><b>20.31</b></td><td>72.61</td><td>29.95</td><td>23.38</td><td>26.26</td><td>60.16</td><td><b>93.68</b></td><td><b>93.55</b></td><td><b>93.61</b></td><td>71.03</td><td>68.00</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>22.06</td><td>12.34</td><td>15.83</td><td>72.62</td><td><b>44.60</b></td><td><b>30.92</b></td><td><b>36.52</b></td><td><b>67.19</b></td><td>93.17</td><td>93.83</td><td>93.51</td><td><b>72.84</b></td><td><b>73.10</b></td>
  </tr>
  <tr>
    <td colspan="15">Consumer Staples</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td><b>27.43</b></td><td>10.90</td><td>15.60</td><td>71.77</td><td>41.91</td><td>18.51</td><td>25.68</td><td>65.98</td><td>91.27</td><td><b>92.38</b></td><td>91.82</td><td>70.58</td><td>69.47</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>26.63</td><td><b>16.92</b></td><td><b>20.70</b></td><td><b>73.83</b></td><td>32.41</td><td>22.08</td><td>26.27</td><td>60.14</td><td>91.76</td><td>91.07</td><td>91.41</td><td>71.71</td><td>68.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>24.34</td><td>13.05</td><td>17.23</td><td>73.75</td><td><b>44.79</b></td><td><b>29.13</b></td><td><b>35.30</b></td><td><b>64.36</b></td><td><b>92.20</b></td><td>92.29</td><td><b>92.24</b></td><td>73.00</td><td>70.41</td>
  </tr>
  <tr>
    <td colspan="15">Energy</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>48.60</td><td>29.09</td><td>36.40</td><td>80.13</td><td><b>55.80</b></td><td>38.54</td><td>45.59</td><td><b>74.14</b></td><td><b>95.40</b></td><td><b>95.03</b></td><td><b>95.21</b></td><td>79.14</td><td>78.12</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>50.53</td><td><b>47.49</b></td><td><b>48.96</b></td><td>81.26</td><td>26.93</td><td>25.19</td><td>26.03</td><td>58.57</td><td>95.38</td><td>94.65</td><td>95.01</td><td>82.81</td><td>75.17</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>53.81</td><td>40.13</td><td>45.97</td><td>81.59</td><td>52.88</td><td><b>45.41</b></td><td><b>48.87</b></td><td>73.90</td><td><b>95.40</b></td><td><b>95.03</b></td><td><b>95.21</b></td><td><b>85.10</b></td><td><b>80.30</b></td>
  </tr>
  <tr>
    <td colspan="15">Financials</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>24.13</td><td>8.68</td><td>12.77</td><td>73.81</td><td>41.34</td><td>29.62</td><td>34.52</td><td>68.50</td><td>95.49</td><td>95.49</td><td>95.49</td><td>68.59</td><td>70.37</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td><b>29.62</b></td><td><b>16.87</b></td><td><b>21.50</b></td><td>71.37</td><td>28.38</td><td>24.31</td><td>26.19</td><td>61.41</td><td>94.47</td><td>90.82</td><td>92.61</td><td>72.61</td><td>68.52</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>26.92</td><td>12.15</td><td>16.75</td><td>75.02</td><td><b>46.91</b></td><td><b>36.71</b></td><td><b>41.19</b></td><td>68.43</td><td>95.41</td><td>90.49</td><td>92.89</td><td><b>72.70</b></td><td><b>72.11</b></td>
  </tr>
  <tr>
    <td colspan="15">Healthcare</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>24.00</td><td>9.25</td><td>13.36</td><td>73.28</td><td>41.51</td><td>25.72</td><td>31.76</td><td>67.57</td><td><b>93.52</b></td><td><b>93.52</b></td><td><b>93.52</b></td><td>74.07</td><td>71.63</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>28.67</td><td><b>16.97</b></td><td><b>21.32</b></td><td><b>73.72</b></td><td>22.22</td><td>20.71</td><td>21.44</td><td>58.41</td><td>93.41</td><td>89.48</td><td>91.40</td><td>75.59</td><td>69.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>29.07</td><td>12.76</td><td>17.73</td><td>73.37</td><td><b>44.52</b></td><td><b>37.54</b></td><td><b>40.73</b></td><td>68.43</td><td>93.37</td><td>93.52</td><td>93.45</td><td>76.43</td><td><b>72.74</b></td>
  </tr>
  <tr>
    <td colspan="15">Industrials</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>27.15</td><td>10.56</td><td>15.20</td><td>72.13</td><td>43.81</td><td>23.08</td><td>30.23</td><td>63.86</td><td>92.22</td><td>89.85</td><td>91.02</td><td>68.74</td><td>68.28</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>28.93</td><td><b>18.15</b></td><td><b>22.30</b></td><td>73.33</td><td>31.95</td><td>23.60</td><td>27.15</td><td>60.25</td><td>92.52</td><td>92.37</td><td>92.44</td><td>71.75</td><td>68.30</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>29.35</td><td>13.53</td><td>18.52</td><td>73.94</td><td><b>50.84</b></td><td><b>36.27</b></td><td><b>42.34</b></td><td><b>65.46</b></td><td><b>92.53</b></td><td><b>92.53</b></td><td><b>92.53</b></td><td><b>73.67</b></td><td><b>71.05</b></td>
  </tr>
  <tr>
    <td colspan="15">Information Technology (IT)</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>23.82</td><td>11.02</td><td>15.07</td><td>75.28</td><td>46.21</td><td>23.66</td><td>31.30</td><td>70.25</td><td>92.25</td><td>93.07</td><td>92.66</td><td>73.47</td><td>73.05</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td><b>28.78</b></td><td><b>21.13</b></td><td><b>24.37</b></td><td><b>75.58</b></td><td>29.32</td><td>20.02</td><td>23.79</td><td>62.55</td><td>92.59</td><td>91.72</td><td>92.15</td><td>75.30</td><td>71.25</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>27.38</td><td>16.29</td><td>20.42</td><td>75.44</td><td><b>49.45</b></td><td><b>32.00</b></td><td><b>38.85</b></td><td><b>72.26</b></td><td><b>92.66</b></td><td>92.77</td><td><b>92.72</b></td><td><b>76.67</b></td><td><b>74.80</b></td>
  </tr>
  <tr>
    <td colspan="15">Materials</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>30.17</td><td>13.71</td><td>18.85</td><td>70.63</td><td>42.80</td><td>27.95</td><td>33.81</td><td>64.58</td><td>91.34</td><td>91.56</td><td>91.45</td><td>73.04</td><td>69.5</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td><b>31.63</b></td><td><b>25.00</b></td><td><b>27.92</b></td><td><b>71.47</b></td><td>38.03</td><td>29.87</td><td>33.46</td><td>61.42</td><td><b>91.58</b></td><td><b>91.80</b></td><td><b>91.69</b></td><td>75.19</td><td>69.51</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>29.36</td><td>18.54</td><td>22.73</td><td>70.53</td><td><b>49.67</b></td><td><b>36.86</b></td><td><b>42.32</b></td><td><b>65.96</b></td><td>91.56</td><td>91.56</td><td>91.56</td><td><b>76.42</b></td><td><b>70.94</b></td>
  </tr>
  <tr>
    <td colspan="15">Real Estate</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td><b>66.66</b></td><td>38.16</td><td>48.54</td><td>82.23</td><td>38.80</td><td>40</td><td>39.39</td><td>74.50</td><td>89.70</td><td>93.84</td><td>91.72</td><td>78.88</td><td>79.17</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>48.00</td><td>36.64</td><td>41.55</td><td>82.85</td><td>16.92</td><td>16.92</td><td>16.92</td><td>66.50</td><td>93.84</td><td>93.84</td><td>93.84</td><td>82.75</td><td>77.35</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>61.29</td><td><b>43.51</b></td><td><b>50.89</b></td><td><b>83.54</b></td><td><b>48.27</b></td><td><b>43.07</b></td><td><b>45.52</b></td><td><b>67.25</b></td><td>93.84</td><td>93.84</td><td>93.84</td><td>83.75</td><td>79.10</td>
  </tr>
  <tr>
    <td colspan="15">Utilities</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td><b>57.77</b></td><td>43.33</td><td>49.52</td><td>79.25</td><td>39.28</td><td>22.44</td><td>28.57</td><td>63.33</td><td>83.67</td><td>83.67</td><td>83.67</td><td>82.00</td><td>76.42</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td><td>48.64</td><td><b>60.00</b></td><td><b>53.73</b></td><td><b>80.25</b></td><td>22.72</td><td>20.40</td><td>21.50</td><td>62.33</td><td>83.67</td><td>83.67</td><td>83.67</td><td>85.67</td><td>77.35</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>53.70</td><td>48.33</td><td>50.87</td><td>79.50</td><td><b>65.95</b></td><td><b>63.26</b></td><td><b>64.58</b></td><td><b>69.33</b></td><td>83.67</td><td>83.67</td><td>83.67</td><td><b>86.33</b></td><td><b>78.78</b></td>
  </tr>
</table>

Table 12: Performance of LLMs on three evaluation tasks with different GICS sectors. To save space, we only report the performance of three LLMs.
Annotation Guidelines

1. Purpose
These guidelines are intended to provide financial analysts with a clear framework for consistently annotating audio transcripts from financial roadshow meetings, extracting comprehensive and professional structured summaries and Q&A information. All information should be saved in JSON format to ensure consistency and repeatability.

2. Annotation Process
2.1 Listen to the Audio
Complete Listening: Analysts should listen to the entire meeting audio to ensure a full understanding of the content.
Note-taking: While listening, jot down key themes and important details for use in subsequent annotation steps.

2.2 Load ASR Text
Obtain and Verify: Obtain the automatic speech recognition (ASR) text of the audio.
Text Proofreading: Compare the ASR text with the actual audio to identify and correct apparent errors.

2.3 Professional Structured Summary Annotation
Theme and Information Extraction
Independent Analysis: Two analysts independently extract important themes and relevant key information from the text.
Themes and Information: Ensure each theme contains its key information.
JSON Format Saving
Save Format: Format each theme and its key information into a JSON structure, with each JSON element representing a thematic paragraph.
Merging and Review
Senior Analyst Review: A senior analyst reviews and consolidates the two reports to ensure completeness and accuracy.

Annotation Steps
Create Theme Object: Create a JSON object for each theme, specifying the theme name.
Record Key Information: Save all crucial information entries in the "Key Information" array.

2.4 Q&A Extraction Annotation
Question and Answer Identification
Mark Interaction: Mark the Q&A interaction parts in the text, ensuring the questioner and answerer are identified (e.g., speaker and audience).
Summarization and Extraction
Refine Summaries: Extract and summarize each Q&A interaction from the meeting.
JSON Format Saving
Save Format: Save the Q&A pairs in JSON format, with each Q&A pair being an individual JSON element.
Merging and Review
Comprehensive Integration: Two analysts independently annotate and a senior analyst consolidates the final results.

Annotation Steps
Create Q&A Object: Create a JSON object for each Q&A pair, recording the questioner and answerer.
Record Content: Record detailed content in the "Question" and "Answer" fields, respectively.

3. Annotation Check
Review and Confirm: A senior analyst reviews the annotations to ensure consistency, accuracy, and completeness.
Communicative Adjustment: If inconsistencies arise, the senior analyst should discuss with annotators to finalize the standard version.

4. Notes
Maintain Neutrality: Stay objective during the annotation process, avoiding personal views.
Team Collaboration: Record and discuss difficult-to-judge information with the team promptly.
Data Confidentiality: Ensure compliance with confidentiality and data protection regulations for the meeting content.

Figure 10: Annotation guidelines of M^3FinMeeting.

Reviewer paper summaries: The paper proposes $M^3FinMeeting$, a comprehensive benchmark on the finance domain. The evaluation tasks include summarization, QA pair extraction, and question answering. The authors have experimented with representative LLMs to evaluate the effectiveness of the proposed benchmark. The automatic metrics include neural-based similarity matching and GPT-4-based judgments.

The paper introduces M3FinMeeting, a novel multilingual, multi-sector financial meeting comprehension benchmark supporting English, Chinese, and Japanese across 11 GICS industries. It encompasses summarization, QA pair extraction, and question-answering tasks. Experiments with seven LLMs reveal substantial room for improvement, particularly in long-context handling, though significant reliance on GPT-4 raises reproducibility concerns.
Summary of strengths: 1) The proposed benchmark is novel and valuable for the research community. It includes different languages and sectors, which could benefit the development of AI4Fin.

2) The analyses are comprehensive as they include open-weight and closed-source LLMs. The ablations also demonstrate that different factors, such as long contexts and nuanced QA scenarios in the finance domain, are still challenging for current LLMs. This could be a good testbed for future research.

3) The authors have generally resolved the issues in the previous reviews, including more analysis on GPT-4-based evaluations and providing annotation guidelines.

- Novel Contribution: Addresses a critical gap in financial NLP by introducing a benchmark based on real-world financial meetings rather than static sources like news or earnings reports. 

 - Diversity: Multilingual data (English, Chinese, Japanese) spanning 11 GICS sectors enhances cross-linguistic and cross-domain applicability. 

 - Enhanced Annotation: Annotation guidelines and human evaluation metrics (Appendices H and F) improve reproducibility and consistency compared to the last version.
Summary of weaknesses: 1) **Agenda for publishing the benchmark for the research community**: The paper did not mention any timelines for publishing the benchmark/leaderboard, which could be crucial in publishing papers in *ACL venues. 

2) **Lack of case study**: The paper has mentioned several findings, and experimental settings, which are quite interesting to have more concrete examples to further demonstrate. For example, why not prompt LLMs to answer each question at one time? In financial meetings, I know the information is quite dense, and the answers could be separately distributed into different sections. But summarizing them and having atomic evaluation would be important in AI4Fin. Moreover, why do LLMs perform that high in Japanese? Further explanations are needed. 

3) **Anonymize the sensitive information in your datasets**: The authors claim to directly collect the data from financial firms. As the financial meetings might contain very sensitive information about the companies, how could the authors deal with such issues to have the benchmark benefit the general research community? Otherwise, this could only be some technical report instead of an *ACL paper.

- Data Accessibility: Dataset requires an application process for academic access, risking limited reproducibility and long-term availability. Criteria for approval are unspecified. 

 - Ethical Concerns: Anonymization via GPT-4 risks exposing sensitive financial data to external systems, raising ethical issues.
Comments, suggestions and typos: - Table 5 is not very clear. What is the language for the main evaluation results?
- Line 477-478, any idea for further explaining that the models seemed to follow Japanese instructions better? Is it something with the prompts formulation?

- Clarify the ethical justification regarding sensitive data handling in the Ethics section. 
 - The author can consider citing the following Multilingual Financial NLP paper if they find it relevant: https://aclanthology.org/2023.findings-eacl.66/
Overall assessment scores: 3

3.5