Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports

Authors: tianyu  cao
natraj  raman
danial  dervovic
chenhao  tan
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 03:09:04

Abstract: As large language models (LLMs) expand the power of natural language processing to handle long inputs, rigorous and systematic analyses are necessary to understand their abilities and behavior. A salient application is summarization, due to its ubiquity and controversy (e.g., researchers have declared the death of summarization). In this paper, we use financial report summarization as a case study because financial reports are not only long but also use numbers and tables extensively. We propose a computational framework for characterizing multimodal long-form summarization and investigate the behavior of Claude 2.0/2.1, GPT-4/3.5, and Cohere. We find that GPT-3.5 and Cohere fail to perform this summarization task meaningfully. For Claude 2 and GPT-4, we analyze the extractiveness of the summary and identify a position bias in LLMs. This position bias disappears after shuffling the input for Claude, which suggests that Claude seems to recognize important information. We also conduct a comprehensive investigation on the use of numeric data in LLM-generated summaries and offer a taxonomy of numeric hallucination. We employ prompt engineering to improve GPT-4's use of numbers with limited success. Overall, our analyses highlight the strong capability of Claude 2 in handling long multimodal inputs compared to GPT-4. The generated summaries and evaluation code are available at https://github.com/ChicagoHAI/characterizing-multimodal-long-form-summarization.

Main paper: 
 introduction: 
 Summarization, the task of condensing the input text while preserving important information, is ubiquitous and has attracted a lot of interest in the AI community. Recent work demonstrates the strong capability of large language models (LLMs) in summarization. In fact, Pu et al. (2023) finds a clear preference for LLM-generated summaries over humanwritten ones and even declares the death of summarization.Researchers have thus shifted to underexplored domains such as long-form summarization. For example, Chang et al. (2023) evaluates summaries of books and proposes novel measures of coherence. In these cases, human-generated summaries are rare, and it is no longer prudent to assume that human-generated summaries are golden standards. Furthermore, regardless of how good LLM-generated summaries are, it remains an open question how LLMs approach the summarization task.In this work, we contribute to this recent line of work on evaluating long-form summarization by studying financial reports. Financial reports provide a great case study for two reasons. First, financial reports tend to be very long. In particular, the length of the most important section in 10-K reports, "Management's Discussion and Analysis of Financial Condition and Results of Operations" (MD&A), can already exceed the context window of most current commercial models. Second, financial reports include many numbers and tables that are crucial for financial analysis, and this multimodal setting has been understudied in prior work. Thus, it is important to understand how large language models handle numbers in summarizing financial reports.We propose a computational framework to characterize multimodal long-form summarization in summarizing financial reports. We analyze summaries from the state-of-the-art commercial models. As there is no gold standard summary, we are primarily interested in how LLMs summarize these very long inputs.First, we investigate how the information in the reports is utilized. We quantify the extent to which the summaries are extractive and the source location for the extractive information. We find that extractive sentences represent 30% to 40% of the summary. Furthermore, these sentences tend to come from the beginning of the report, similar to the finding in Liu et al. (2023) about question answering, suggesting that LLMs have strong position biases. However, this position bias could be justified in our context because information at the beginning might be inherently more important. Indeed, the bias disappears for Claude after we shuffle the input, indicating that Claude seems to recognize important information in the input, whereas GPT-4 shows a consistent position bias towards the beginning of the input.Second, given the important role of numbers, we conduct a thorough investigation on the use of numbers in LLM-generated summaries. We find that small models like GPT-3.5 tend to include generic texts without mentioning any specific numbers. In comparison, large models use numbers more extensively and often perform simple operations on numbers such as rounding and taking the difference. Claude 2 outperforms GPT-4 with a higher tabular numbers utilization ratio and a larger number density. We further provide a taxonomy of numeric hallucinations through manual annotation. The results show that although LLMs hallucinate about 5% of the time, they may fail to capture the semantic relationship between numeric data and textual data, an instance of context mismatch.Finally, we explore the promise of prompt engineering to enhance the use of numbers by GPT-4. We find that GPT-4 extracts more numbers when prompted properly, but still uses fewer tabular numbers than the summary generated by a simple prompt with Claude 2. Also, fewer hallucinations are generated when using Chain-of-Thought prompting by GPT-4.In summary, we make the following contributions:• We develop a computational framework for characterizing multimodal long-form summarization that accounts for both information usage and numeric values. • We demonstrate that Cohere and GPT-3.5 cannot perform such long-form multimodal summarization. • We compare the behavior of Claude and GPT-4, and show that Claude demonstrates a stronger ability to use numbers and seems to recognize important information. • We provide a taxonomy of numeric hallucination and investigate the promise of prompt engineering to tackle some of the weaknesses in these models. 
dataset and methods: 
 In this section, we introduce the dataset used in this work and provide details of how we generate summaries.Dataset. Form 10-K filings serve as comprehensive annual reports that outline a company's financial condition, business overview, and other disclosures mandated by the U.S. Securities and Exchange Commission (SEC). An annual report on Form 10-K usually contains four main parts and sixteen specific items standardized by the SEC to ensure comprehensive coverage of key business aspects (SEC). Of particular interest for our analysis is Item 7, commonly referred to as "Management's Discussion and Analysis of Financial Condition and Results of Operations" (MD&A), where companies offer their perspectives on the preceding financial year's business outcomes. We randomly selected 1,000 HTML files of 10-K forms obtained from the Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system. These files are then converted into clean JSON format using EDGAR-CRAWLER (Loukas et al., 2021) and we use Item 7 as the target report to do summarization. Additionally, to explore whether LLMs have position biases, we generate shuffled versions of each report by randomly reordering its paragraphs, treating tables as individual paragraphs.Models. We evaluate summaries generated by five commercial models with top context window length (wl) in July 2023 when we started this work: (1) Claude 2.0 (claude-2.0, wl=100K tokens), (2) Claude 2.1 (claude-2.1, wl=200K tokens), 1 (3) GPT-3.5 (gpt-3.5-turbo-1106, wl=16K tokens), (4) GPT-4 (gpt-4-1106-preview, wl=128K tokens) (Achiam et al., 2023) , and (5) Cohere (command, wl=100K characters). For Claude 2.0 and Claude 2.1, we set max output tokens to 4,096 and temperature to 1. GPT-3.5 and GPT-4 are used with default hyperparameters. We use the Cohere co.summarize API and set the output summary length to "long", format as bullets, high extractiveness, and temperature to 0.3 according to the official recommendation. Except for Cohere, which does not need additional prompts with the specific endpoint, we use simple prompts for summarization. Detailed prompts are shown in Appendix D.General analysis of summary. A more detailed investigation of numeric values utilization will be presented in Section 4.It is worth mentioning that certain reports require truncation to meet the context limit of different models, with an averaged of 436, 143, 12,656, and 7,690 words truncated for Claude 2.0, GPT-4, GPT-3.5, and Cohere respectively.GPT-3.5 and Cohere are generally not good at summarization based on the simple prompt, reflected by too short summaries, meaningless contents, and referring to almost zero numbers. We investigate this further in Appendix A. Thus, the rest of this paper mainly focuses on Claude 2 and GPT-4. 
tracing the summary in the input: 
 Obtaining ground-truth summaries for very long inputs is a significant challenge, and there can be numerous good summaries for a given document. Therefore, we mainly focus on the behavior of the models and propose a computational framework to characterize such summary sentence: -For 2019, capital expenditures are estimated at $2.9 billion. 0.95 report sentence: We project our E&P capital and exploratory expenditures will be approximately $2.9 billion in 2019.summary sentence: -Operating cash flows increased by $98 million due to higher net working capital inflows.report sentence 1: The increase was primarily due to higher cash inflows for net working capital of $68.5 million and other current assets and liabilities of $28.2 million. report sentence 2: Cash flows from operating activities increased $98.0 million in 2018 compared to 2017.summary sentence:-Investing cash flows decreased due to lower cash paid for acquisitions. 0.92 report sentence 1: The decrease was primarily due to lower cash outflows for business acquisitions, net of cash acquired of $56.2 million, partially offset by higher cash outflows for capital expenditures of $30.1 million. report sentence 2: Cash flows from investing activities decreased approximately $28.1 million in 2017.Abstractive summary sentence: Results of operations: A detailed comparison between the years 2018 and 2017 displays an increase in net income and net sales, while gross profit as a percentage of net sales experienced a decrease. Table 3 : Extractive summary sentences statistics. "1-1" represents 1-1 extractive summary sentence with one source report sentence and "2-1" for 2-1 synthesizing summary sentence with two source report sentences.behavior. The first aspect is the extent to which the generated summaries are extractive.Second, we examine the information sources of the generated summaries, i.e., which part of the input a summary is derived from.3.1 To What Extent Are Generated Summaries Extractive?LLMs are generative models and thus in theory generate abstractive summaries. However, it is plausible that LLMs perform minimal paraphrasing of sentences in the input to produce the summary. Furthermore, extractive summaries can better maintain the fidelity to the original input. Therefore, we first attempt to match sentences in the summary to those in the input and measure to what extent generated summaries are extractive.Measuring extractiveness. We modify the coverage measure in Grusky et al. (2018) to measure the similarity between two sentences with a greedy match algorithm. Given a summary sentence S = {t S1 , t S2 , ..., t Sn } and a report sentence R = {t R1 , t R2 , ..., t Rm } after removing stopwords, where t i is a token, the length of S is n, and the length of R is m.At a high level, similarity(S, R) is defined as the ratio of matched tokens with a quadratic bonus that rewards longer-matched sequences. Specifically, for each summary sentence, this algorithm processes its tokens iteratively, discovering and aligning the longest matching token sequences from each report sentence. Then, for each report sentence R, we get a list M(S, R) = {{t i , t i+1 , ..., t i+p }, ..., {t j , t j , ..., t j+q }|t ∈ S ∩ R} consisted of the longest matching token sequences m compared with the summary sentence S. The similarity score is computed as:EQUATION)Following a meticulous manual check (Appendix F), we find sentences with a similarity score above 0.8 to contain almost verbatim excerpts from the source text, which we define as extractive. We calculate the similarity score between each summary sentence and report sentence. If the top similarity score is greater than 0.8, we consider the summary sentence a 1-1 extractive sentence. For each remaining summary sentence, we calculate a similarity score with a combination of two report sentences. If this score exceeds 0.8, we call it a 2-1 synthesizing sentence. Examples of 1-1 extractive sentences and 2-1 synthesizing sentences are shown in Table 2 .Extractive sentences represent 30% to 40% of the summary. Table 3 shows the percentage of extractive sentences. Claude 2.1 generates the most extractive contents with 41.50% of extractive summary sentences, which is much higher than 31.24% of Claude 2.0. This may be a result of the multi-level bullet formats of Claude 2.1. GPT-4 generates comparable extractive contents with Claude 2.0 but with a smaller percentage of 2-1 synthesizing sentences. Summaries generated by Claude 2 for the shuffled report exhibit more extractiveness, with 6-7% higher proportions of extractive sentences compared with that of original reports. In contrast, GPT-4 generates fewer extractive sentences from shuffled reports. This observation demonstrates the differing behavior of Claude and GPT-4 when processing incoherent texts.To gain insights into the abstractive capabilities of LLMs, we conduct a case study for the abstractive sentences present in the summaries. Specifically, we find GPT-4 generates abstractive summary sentences with a high degree of condensation, such as summarizing net income, net sales, and gross profit in a single sentence (Table 2 ). It paraphrases the original information across various locations and compacts it into one single sentence, generating new phrases such as "display an increase in net income". The entire meaning is retained, but the content organization is significantly different from the original report. See the Appendix G for similar examples from Claude 2.3.2 Where Does the Information Come From?We study this question by examining the distribution of source contents for the summary.Identify source contents. Based on the extractive analysis, we analyze the distribution of the 1-1 extractive sentences. For each summary sentence, the report sentence with the highest similarity score is treated as its source. If there exist several report sentences with the same score, we choose the one with the top cosine-similarity using sentence embeddings generated by SentenceTransformers (Reimers & Gurevych, 2019) . Then we calculate the position of each source report sentence and visualize the distribution in Figure 1 . We also run the same analysis by including the location of 2-1 synthesizing summary sentence sources. The results are similar and presented in Appendix B.Most summary information comes from the beginning. Figure 1 (a) shows that a high percentage of source contents are found at the first 20% of the report--models tend to use contents at the very beginning of the input, and pay less attention to the information within the middle and the end part of its context. For example, more than 60% of the summary contents generated by GPT-4 come from the first 20% of the report, and less than 10% of the summary contents are from the middle. GPT-4 also shows a preference for the information at the very end of the report, which is not as salient in the summary generated by Claude 2. OR and SR stand for original report and shuffled report respectively, while OS and SS stand for summaries of the original report and shuffled report respectively. For the summary generated from the original report, most information comes from the beginning of the report. However, for the shuffled reports, this trend disappears for Claude but stays for GPT-4.Shuffled report analysis. To understand whether the observed pattern is a bias in LLM's behavior or a consequence of document structure (there is an overview at the beginning of Item 7), we analyze the summaries after we shuffle the input reports. Figure 1 (b) SS-SR shows that the source information is evenly distributed for Claude 2. The distribution SS-OR becomes heavy in the beginning when comparing the summary of the shuffled report with the original report, suggesting that Claude 2 seems to recognize the important information even after shuffling. In comparison, GPT-4 maintains a strong position bias for the shuffled input: the summary of the shuffled input now favors the beginning part despite that it is essentially arbitrary text from the input.2 In summary, although we observe a strong position bias, it is not "lost in the middle" as in Liu et al. ( 2023). In fact, our analysis reveals intriguing behavioral differences between Claude 2 and GPT-4. 
numeric values utilization: 
 Multimodal financial reports convey crucial information through a combination of unstructured textual narratives and structured tabular data. It is important to effectively integrate both data modalities in the summaries. Meanwhile, it is commonly suggested that LLMs can have issues with using numbers (Dziri et al., 2024) . Thus, we present a detailed analysis of how LLMs leverage numeric values from both texts and tables in financial reports.To extract numeric values from the reports, we use a regular expression that matches numbers with commas as thousand separators and an optional decimal part. We excluded numbers from entity names (e.g., "COVID-19", "ATA190"), dates (e.g., "December 31, 2022"), and residual HTML table indices, focusing primarily on financially meaningful values (see Appendix C for details).Based on the source of the numeric values appearing in the summaries, we categorized them into four types:• Type A: Numeric values present only in the report's text.• Type B: Numeric values present only in the report's tables.• Type C: Numeric values present in both the text and tables of the report.• Type D: Numeric values not found in the report.One may hypothesize that it is challenging for LLMs to incorporate information from the table. Thus, numeric values of type A may be most likely to show up in the summaries, while numbers in type B are rarely incorporated. Alternatively, the model might consider type C to be the most important.Claude 2 demonstrates a more sophisticated use of numbers than GPT-4. As illustrated in Table 1 , Claude 2.1 demonstrates a strong ability to use tabular numeric values, with 8.37% of its numeric values belonging to type B, compared to only 4.98% for GPT-4. The difference is statistically significant according to a two-sample t-test (p < 0.001). However, it is noteworthy that in the reports, more than half of the numeric values were exclusively present in tabular formats, suggesting that LLMs tend to prioritize numeric values found in the text and pay relatively less attention to tabular data.Similarly, when comparing the overall density of numeric values in the summaries, Claude 2.1 uses numbers more frequently than GPT-4, with a summary number density of 5.03% versus 1.52% (p < 0.001). This discrepancy can be attributed to GPT-4's tendency to generate longer summaries with fewer numeric values, indicating a potential weakness in analyzing and incorporating numeric information, especially from tabular data, compared to Claude.As we mentioned earlier, GPT-3.5 and Command fail to use numbers in the summaries. Specifically, GPT-3.5's summaries do not include numeric values in 831 of 1,000 reports and only an average of 0.46 number is mentioned. For 46.7% of the reports, Cohere simply generates "unable to comprehend the table". However, for those completed summaries, it extracts an average of 6.90 numbers, which is larger than that of GPT-4. Among those, Command prefers numbers from the tables rather than text (2.12 vs. 1.42), which is quite different from the behavior of other models.-Total assets increased 25.9% to $9.7 billion at December 31, 2019 compared to December 31, 2018, largely due to the acquisitions of Banc Ed and IST. Type D numbers can come from numeric operations. Numeric values of type D, which represent mismatches between the summary and the report, accounted for an average of 2.76 instances per summary generated by Claude 2.1 and 2.66 for Claude 2.0. In contrast, GPT-4 has only 0.46 type D numbers. Through human annotation, we identified that these mismatches often stem from simple operations such as rounding, calculating differences, or computing rates of change (e.g., 86.36% of type D numbers generated by Claude 2.0 come from such operations).One notable example that showcases the LLMs' ability to synthesize numeric information is their accurate representation of numeric values in the correct units, such as billions instead of millions, as shown in Figure 2 . This exemplifies the LLMs' ability to extract and retain crucial details, coupled with their proficiency in comprehending and manipulating numeric data coherently throughout the summarization process. However, it is important to note that a subset of these type D numeric values could potentially be instances of numeric hallucinations, a phenomenon that warrants further investigation, as discussed in Section 4.2.Numeric hallucinations refer to cases where a numeric value in the generated output text is inconsistent with or unsubstantiated by the corresponding source data. Although numeric values of type D are reasonable candidates for hallucinations, we find that other numbers could be hallucinations too, e.g., referring to 75 million in profit as 75 million in debt. Therefore, we resort to manual annotation to assess the type and extent of hallunication.3 Annotation protocol. To comprehensively annotate numeric hallucinations in summaries of lengthy financial reports, we employ the following multi-step protocol:(1) Candidate extraction: For each numeric value mentioned in the summary, extract all sentences (quotes) from the source report that contain either the exact same numeric value or values in different formats (e.g., 1,000,000 and 1M).(2) Match identification: The annotator examines the list of extracted report quotes for a given summary number: a. If a quote accurately matches the context and usage of the numeric value in the summary, it is annotated as "No Hallucination". b. If no matching quote is found, the annotator conducts a comprehensive search through the entire report. b1. If a correct match is identified, the corresponding quote is recorded, and the instance is annotated as "No Hallucination." b2. If no substantiating evidence is found in the report, the instance is annotated as a specific type of numeric hallucination based on the taxonomy definitions.(3) Annotations and comments: For cases annotated as a hallucination type, the annotator provides detailed comments explaining the rationale behind the annotation decision.This protocol ensures a systematic and thorough annotation process, aiding in the accurate identification and categorization of numeric hallucinations. It emphasizes examining all potential evidence from the source report before finalizing annotations. The lead author performed this analysis and annotated numbers in the summaries of a random sample of 20 reports for Claude 2.0, Claude 2.1, and GPT-4.A taxonomy of numeric hallucination. We identify and categorize four types of numeric hallucinations, collectively forming the first taxonomy for this phenomenon:• Fabricated Number: The presence of a specific numeric value in a generated text that lacks any corresponding references. • Rounding Error: Discrepancy arising from the rounding off of numeric values.• Arithmetic Error: Numbers generated from incorrect mathematical calculations applied to numeric values from the report. • Context mismatch: An inconsistency where the same numeric value applies to different contexts in the report vs. in the summary.Instances of each type of hallucination are shown in Table 4 .LLMs hallucinate in only about 5% of numerical values. Overall, LLMs hallucinate at a low percentage when using numeric values, with total hallucination rates of 6.48%, 3.97%, and 5.74% observed for Claude 2.0, Claude 2.1, and GPT-4, respectively. Context mismatch hallucinations exhibited the highest percentages across all models, which suggests that current models still face challenges in accurately capturing the semantic relationships between numeric data and their corresponding textual descriptions when generating summaries. In contrast, fabricated number hallucinations occurred at relatively low percentages, indicating that most models demonstrated reasonable performance in avoiding blatantly unsupported numeric claims in their outputs. However, even low rates of such hallucinations can undermine the trustworthiness of the summaries.Performance analysis: GPT-4 vs. Claude 2. As shown in Table 4 , GPT-4 showcased impressive performance by generating zero instances of fabricated numbers and arithmetic errors. This can be attributed to GPT-4's tendency to produce extractive summaries, where most numeric values are directly copied from the source report, as evidenced by the numeric analysis in Section 4.1. In contrast, Claude 2 attempted to generate more abstractive content, involving a higher degree of arithmetic operations, which contributed to its presence of arithmetic error hallucinations. Notably, Claude 2 exhibited a lower percentage of numeric hallucinations, but this is not the case for Claude 2.1. This shows GPT-4's better ability to leverage CoT prompting for introspecting its own output. 
related work: 
 Evaluation of long-form summarization is very challenging as traditional evaluation requires reference summaries (Lin, 2004; Papineni et al., 2002) . The closest work to ours is Chang et al. (2023) , which presents the first study of LLM-based book-length summary. Their focus is on coherence, while our work focuses on the characterization of LLM behavior in a multimodal setting. Other work has shown that summaries generated by GPT-3 and current LLMs are overwhelmingly preferred by humans and these also do not suffer from common dataset-specific issues such as poor factuality (Goyal et al., 2023; Pu et al., 2023) .A battery of studies examine automatic evaluation using LLMs considering the high cost of human annotation (Min et al., 2023; Peng et al., 2023; Gilardi et al., 2023, i.a.) . However, Doostmohammadi et al. (2024) suggest that automatic evaluation can only approximate human judgments under specific conditions, and their reliability is highly context-dependent. 
conclusion and discussion: 
 We present the first study to characterize summaries generated by LLMs for multimodal long-form inputs. Our results reveal that different LLMs may approach this task quite differently in terms of extractiveness, position bias, and use of numbers. While the position bias of GPT-4 is probably sub-optimal, it is unclear what makes a good summary, especially for such long-form inputs. On the common concern about hallucinations, we find hallucinations do not happen very frequently. Future work may not only focus on the frequency but also the potential impact of such misuse. The underlying causes of our observations also remain unclear; we hypothesize that Claude may have more exposure to enterprise data/applications involving structured data and business metrics. This exposure could potentially explain Claude's stronger ability to use numbers in the summary. However, it is important to note that all the LLMs used in our work are closed and the hypotheses need further investigation.Our research community urgently need novel perspectives for evaluating summarization.The "death" of summarization is an opportunity for advancing summarization beyond simple relevance and human preference.• Integer numbers with or without commas as thousands separators.• Decimal numbers with or without commas as thousands separators and with or without a fractional part.r'(?<!\d)(?<![a-zA-Z-])\d{1,3}(?![a-jln-zA-JLN-Z\d])(?:,\d{3})*(?:\.\d+)?'When extracting numeric values from the report or summary, numbers from entity names (e.g., "COVID-19", "ATA190"), dates number (e.g., "December 31, 2022"), and residual HTML table indices are excluded, focusing primarily on financially meaningful values.MD&A: ... Let's generate the summary step by step.1. Read through the entire MD&A report carefully to understand the context. 2. Identify and extract the key topics and insights discussed in the report.  
 
 Appendix: We added Claude2.1 when it was out, but Claude 3 came out in February 2024, so we have not been able to include it.Due to cost reasons, the GPT-4 analysis is based on 100 shuffled reports.We experimented with automatic analyses using GPT-4, but GPT-4 has trouble making sense of these statements accurately. 


Decision: The paper studies the performance of LLMs on long-form summarization of financial documents. The main contributions are: (1) benchmarking LLMs on this new long document domain and characterizing their behaviors (position bias, extractiveness, etc.) (2) taxonomy of numerical errors in long-form summarization in this domain. 

Pros: The error taxonomy created for numerical errors, which the paper posits are the most critical errors in this domain, is useful. The analysis of how LLMs extract and summarize information rich in numerical data is interesting. 

Cons: 
While the analysis of extractiveness, position bias is interesting, it is not clear what the takeaways from these observations should be. For e.g., there is no discussion of what a "good" summary is, and therefore, not clear how to interpret one LLM being XX% more extractiveness than the another. The paper can be made stronger by analyzing what *gold* summaries look like for this domain and comparing LLM performances against that standard. 

The paper only focuses on one type of factual errors in the summary, and as such, it is unclear how important or prevalent these are compared to other factors. The lack of human evaluation of overall quality makes it difficult to ascertain the utility of any of these LLMs for this task.

Reviews: review: paper_summary: paper_summary: This paper presented a case study of long-form summarization in the finance domain in order to study LLMs’ behaviors when it comes to summarizing long input texts and numerical operations. Specifically, the authors first proposed a computational framework to characterize multimodal long-form summarization in summarizing financial reports and investigated how the information in a given financial report is utilized. Then the authors conducted a thorough inspection and evaluation of how selected LLMs handled numbers in financial reports, and proposed a taxonomy of categorizing errors related to numeric operations. The findings of the experimental results show that LLMs have strong position biases and can be mitigated by shuffling operations.
strength_weakness: strength_weakness: I don’t have much to say here, but I have some comments and questions that I would like to bring up in the following section.
questions: questions: 1. Regarding the authors’ comment and references about LLM-generated summaries overwhelmingly preferred by humans: I think this is only applied to single document one-sentence new summarization [1]. As some previous work showed , this is definitely not the case for non-news summaries ([2]---[4]). In other words, single document summarization of news is solved != summarization is solved. 

2. The authors may want to cite some prior work on chart-to-summary generation ([5]--[7]). 


[1] https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00632/119276/Benchmarking-Large-Language-Models-for-News 

[2] https://dl.acm.org/doi/abs/10.1145/3477495.3531802?casa_token=g-Uw8_87GzwAAAAA:gnrbj6AxvcswhwB7Mjzumbzynm0eG6aB-A1WYPlsKhyGjn-9p12bn71uCp8rde7fxMRgHRwHTMNswg 

[3] https://aclanthology.org/2023.findings-acl.593/

[4] NewSumm Keynote 1 by Kathleen McKeown 

[5] https://aclanthology.org/2022.acl-long.277/ 

[6] https://aclanthology.org/2021.ranlp-1.183/ 

[7] https://link.springer.com/chapter/10.1007/978-3-319-46478-7_41
paper_summary: paper_summary: This paper analyzes the use of LLMs for summarizing financial reports with a specific focus on how LLMs handle numbers since financial reports are often dense with numerical figures and a good summary needs to condense not just text but also numerical data correctly. They also do human labelled case studies on abstractive sentences and certain numerical figures found in the summary to check if these are correct or if they are hallucinations.
strength_weakness: strength_weakness: 1. While this work is a good effort in this direction, we are a long way away from ensuring that the summaries are factually correct.
2. Some more prompting methodologies such as chain of thought or speculative prompting could have been used to see if hallucinations can be fixed through LLM introspecting it's own output.
3. Most analysis is done via human labelled case studies and will be hard to reproduce
questions: questions: --
paper_summary: paper_summary: The paper conducts a comparison of various LLM models regarding their ability to summarize sections of financial reports (10-K), which encompass not only textual content but also tabular and numeric data. It highlights two key findings. Firstly, it identifies a position bias among all models, indicating a tendency to prioritize the initial sentences. Upon further analysis, the paper demonstrates that Claude exhibits less position bias compared to GPT-4 when assessed on randomly-shuffled texts. Secondly, the paper reveals that Claude demonstrates a more sophisticated utilization of numbers, including basic computation, compared to other models. Additionally, it categorizes different instances of hallucination by LLMs in handling numerical data.

The paper introduces intriguing phenomenon, such as Claude exhibiting reduced position bias and superior utilizing of numeric data. However, merely reporting these observations may not contribute significantly to the scientific values. The paper would be further strengthened if it could elucidate the underlying reason for the phenomenon. Another drawback is that it has been evaluated only on a single document type -- 10-K. This raises questions about the generalizability of the findings to the other document types.
strength_weakness: strength_weakness: - The scientific contribution of the paper appears to be limited. Its primary contribution lies in contrasting the behavior of Claude2, GPT-4 and other LLMs. However, the significance of these contributions is unclear, as even acknowledged by the authors themselves, who state in the paper, "...While the position bias of GPT-4 is probably sub-optimal, it is unclear what makes a good summary, especially for such long-form inputs…"

- The generalizability of the paper's claims is also uncertain. The evaluation is based solely on one type of financial document - 10-K reports, which are publicly available and likely used during the pre-training of the LLM models. It's questionable whether the findings can be applied to other financial documents  or to any document containing tables and numbers.

- Strengthening the paper would involve providing hypotheses to explain its findings. Specifically, the paper reveals that larger models tend to utilize numbers, while smaller models do not. Additionally, it demonstrates that Claude employs numbers in a more sophisticated manner than GPT-4. Offering potential explanations for these phenomena would enhance the paper's impact and interest.
questions: questions: - It looks like each different model received truncated texts due to the limited context window. It would be great if the paper adds the average amount of the truncated contents. 

- In Section 3.1, the paper says “Specifically, we find that the GPT-4’s summary shows a high degree of condensation such as summarizing net income, net sales, and gross profit in a single sentence (Table 2)”. What is the evidence backing this claim? 

- How were the tables represented? Were the HTMLs used as an input to LLM?
paper_summary: paper_summary: The paper presents a case study on summarization capabilities of LLMs. Specifically, the authors focus on summarization of financial reports with several current top commercial LLMs (GPT, Claude, Command).

The study focuses a lot on processing numbers, which are known to pose various difficulties to LLMs. The authors perform various analyses, including a small-scale manual annotation of number hallucinations. In general, the authors find that number processing is very strong in some of the models, especially Claude.

The study also analyzes positional bias with LLMs tending to mostly cover information from the beginning of the document, and find interesting differences between Claude and GPT.
strength_weakness: strength_weakness: - The study is only descriptive and observational, it covers in great detail the observed behavior and capabilities of the models, but does not attempt at all at interpreting the findings or hypothesizing about the reasons that lead to the findings. Of course this is somewhat hard since all of the investigated systems are closed-source with only limited information available about them, but still some information is provided by the models' authors and other information have been reported by independent researchers, so it would be good to at least try to link the known differences in the models' design and training to the observed differences in behavior on the examined task.
- The study mostly does not attempt to evaluate the quality of the reports, claiming there is no gold standard summary for their inputs. This is sad, as although we observe some differences in the models, in many cases we do not know for sure whether the differences mean that one of the models produces better summaries than the other, or whether these are just two possible approaches to the summarization task which are both good. I understand the difficulty in not having the gold standard data, but this should at least be discussed more explicitly and in more detail. For example, the fact that a model produces less numbers in the report, does this automatically mean the report is worse, or may it be that it is a highly abstractive report that interprets the numbers textually without mentioning them and thus this might also be a great report? I am not knowledgeable in the domain of financial reports and their summaries, so I do not know whether a good summary absolutely must contain a lot of numbers (which is somehow indirectly implied by the paper but not explicitly said or discussed). So this is a clear limitation of the paper. Still, I appreciate the manual evaluation of number hallucinations, which goes beyond this and actually does evaluate what is good and what is bad.
- For models with a short input context window, the authors simply truncate the input. This does not seem right, why not use a sliding window approach?
questions: questions: These are something between questions and tentative suggestions (which might or might not be good suggestions, so this is just for consideration).
- Is text+tabular data generally called "multimodal" data? I can understand that, but I am more accustomed to "multimodal" referring to other modalities than only text, while tables are still text. But maybe my terminology is wrong and yours is correct, I have not specifically focused on working with tabular data so my terminology is not strong there.
- In table 1, would it make sense to report A B C D as percentage instead of absolute numbers, probably a sort of recall of what proportion of the numbers from the report appear in the summary?
- Also, table 1 is quite far from it description, could it maybe be moved to a lter part of the paper?
- Also, A B C D are not very reader-friendly numbers, could they be maybe e.g. X B XB N (teXt taBle teXt+taBle None) or something more interpretable?

Comments: title: [Area Chair comment] Author response
[Area Chair Comment] Author Response
comment: Thank you for the positive feedback and we will incorporate the hypothesis in the final version of the paper!
Thanks a lot for the reply, which has addressed all of my major comments. I quite like the replies, and I think the paper can be made yet a little better by reflecting some of those in the final version of the text.

"Re: Hypotheses for the findings" -- thanks a lot for that, that's what I was looking for. Although I see now that this is necessarily quite vague, so I understand why you did not put this in the original paper, and I leave it up to your decision what to include and what is too vague to include. (Myself I would add the hypothesis about Claude's possible "exposure to enterprise data/applications involving structured data and business metrics" which I find insightful and sensible, but the rest maybe is too vague.)

I already gave the paper the "Good paper, accept" rating, and I stand by this rating.
Thank you very much for considering our feedback! We agree that generalization is a key issue and will further clarify this in Limitations.
Thanks for the rebuttal. I was initially concerned about the generalizability of the findings and limited scientific contribution, as the paper primarily reports only behavioral differences among various LLM models. After reading the other reviews and considering the authors' feedback, I still have concerns regarding the generalizability. However, I think that the observational findings reported in the paper could be valuable to the research community. Therefore, I have updated my score.
Dear reviewer ,

Please take a look at the author's rebuttal. If the rebuttal addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum.

Are your concerns about the descriptive vs prescriptive nature of the results addressed? (Reviewer Lb8C also had similar issues).

Other reviewers should also feel free to comment on these points! 

Thanks.
Dear reviewer ,

Please take a look at the author's rebuttal. If the rebuttal addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum.

In particular, are your concerns about the generalizability of the approach and the paper's main contributions addressed? 

Other reviewers should also feel free to comment on these points! 

Thanks.
Thanks for responding to my reviews. Best of luck with the revision.