Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports
Authors: Tianyu Cao
Natraj Raman
Danial Dervovic
Chenhao Tan
Venue: COLM (2024)
Keywords: summarization, long-form, multimodal, extractiveness
TL;DR: We proposes a computational framework to characterize and evaluate the ability of large language models to summarize long, multimodal financial reports.

Abstract: As large language models (LLMs) expand the power of natural language processing to handle long inputs, rigorous and systematic analyses are necessary to understand their abilities and behavior. A salient application is summarization, due to its ubiquity and controversy (e.g., researchers have declared the death of summarization). In this paper, we use financial report summarization as a case study because financial reports are not only long but also use numbers and tables extensively. We propose a computational framework for characterizing multimodal long-form summarization and investigate the behavior of Claude 2.0/2.1, GPT-4/3.5, and Cohere. We find that GPT-3.5 and Cohere fail to perform this summarization task meaningfully. For Claude 2 and GPT-4, we analyze the extractiveness of the summary and identify a position bias in LLMs. This position bias disappears after shuffling the input for Claude, which suggests that Claude seems to recognize important information. We also conduct a comprehensive investigation on the use of numeric data in LLM-generated summaries and offer a taxonomy of numeric hallucination. We employ prompt engineering to improve GPT-4's use of numbers with limited success. Overall, our analyses highlight the strong capability of Claude 2 in handling long multimodal inputs compared to GPT-4. The generated summaries and evaluation code are available at https://github.com/ChicagoHAI/characterizing-multimodal-long-form-summarization.

not only Published as a conference paper at COLM 2024
Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports
Tianyu Cao
Natraj Raman Danial Dervovic
Carnegie Mellon University cianyuca@andrev. cmu. edu
JPMorgan Al Research (natraj raman, danidn al, dervovic) pmerooricjjjmorgaa..co pmmorgann
Chenhao Tan
The University of Chicago chenhao@uchicago.. edu
Abstract
As large language models (LLMs) expand the power of natural language processing to handle long inputs, rigorous and systematic analyses are necessary to understand their abilities and behavior. A salient application is summarization, due to its ubiquity and controversy (e.g,, researchers have declared the death of summarization). In this paper, we use financial report summarization long but also use numbers and tables extensively. We propose a computa- tional framework for characterizing multimodal long-form summarization and investigate the behavior of Claude 2.0/2.1, GPT-4/3.5, and Cohere. We find that GPT-3.5 and Cohere fail to perform this summarization task meaningfully. For Claude 2 and GPT-4, we analyze the extractiveness of the summary and identify a position bias in LLMs. This position bias dis- appears after shuffling the input for Claude, which suggests that Claude seems to recognize important information. We also conduct a comprehen- sive investigation on the use of numeric data in LLM-generated summaries and offer a taxonomy of numeric hallucination. We employ prompt engi- neering to improve GPT-4's ' use of numbers with limited success. Over- all, our analyses highlight the strong capability Claude 2 in handling long multimodal inputs compared to GPT-4. The generated summaries and evaluation code are available at luttps://iithi: com hhmm//CCcagHAA/ pennterriiinn-mliiiiodee -lng -ooo--orm----- form summarzztiion
study because financial reports
Introduction
Summarization, the task of condensing the input text while preserving important informa- tion, is ubiquitous and has attracted a lot of interest in the Al community. Recent work demonstrates the strong capability of large language models (LLMs) in summarization. In fact, Pu et al. (2233) finds a clear preference for LLM-generated summaries over human- written ones and even declares the death of summarization.
Researchers have thus shifted to underexplored domains such as long-form summarization. For example, Chang al. (2023) evaluates summaries of books and proposes novel measures of coherence. In these cases,
summaries are rare, and it is no longer prudent to assume that semmanngenerated summaries are golden standards. Furthermore, regardless of how good LLM-generated summaries are, it remains an open question how LLMs approach the summarization task.
In this work, we contribute to this recent line of work on evaluating long-form summa- rization by studying financial reports. Financial reports provide a great case study for two reasons. First, financial reports tend to be very long. In particular, the length of the most important section in 10-K reports, "Management's Discussion and Analysis of Financial Condition and Results of Operations" (MD&A), already exceed the context window

---

Published as a conference paper at COLM 2024
of most current commercial models. Second, financial reports include many numbers and tables that are crucial for financial analysis, and this multimodal setting has been understud- ied in prior work. Thus, it is important to understand how large language models handle numbers in summarizing financial reports.
We propose computational framework to characterize multimodal long-for summa- rization in summarizing financial reports. We analyze summaries from the state-of-the-art commercial models. As there is no gold standard summary, we are primarily interested in how LLMs summarize these very long inputs.
First, we investigate how the information in the reports is utilized. We quantify the extent to which the summaries are extractive and the source location for the extractive information. We find that extractive sentences represent 30% 40% of the summary. Furthermore, these sentences tend to come from the beginning of the report, similar to the finding in Liu et al. (2023) about question answering, suggesting that LLMs have strong position biases. However, this position bias could be justified in our context because information at the beginning might be inherently more important. Indeed, the bias disappears for Claude after we we shuffle the input, indicating that Claude seems to recognize important information in the input, whereas GPT-4 shows a consistent position bias towards the beginning of the input.
Second, given the important role of numbers, conduct thorough investigation on the use of numbers in LLM-generated summaries. We find that small models like GPT--3.5 tend to include generic texts without mentioning any specific numbers. In comparison, large models use numbers more extensively and often perform simple operations on numbers such as rounding and taking the difference. Claude 2 outperforms GPT-4 with higher tabular numbers utilization ratio and a larger number density. We further provide taxonomy of numeric hallucinations through manual annotation. The results show that although LLMs hallucinate about 5% of the time, they may fail to capture the semantic relationship between numeric data and textual data, an instance of context mismatch.
Finally, we explore the promise of prompt engineering to enhance the use of numbers by GPT-4. We find that GPT-4 extracts more numbers when prompted properly, but still uses fewer tabular numbers than the summary generated by simple prompt with Claude 2. Also, fewer hallucinations are generated when using Chain-of-Thought prompting by GPT-4.
In summary, we make the following contributions:
We develop a computational framework for characterizing multimodal long-form sum- marization that accounts for both information usage and numeric values.
We demonstrate that Cohere and GPT-3.5 cannot perform such long-form multimodal summarization.
We compare the behavior of Claude and GPT-4, and show that Claude demonstrates stronger ability to use numbers and seems recognize important information.
We provide taxonomy of numeric hallucination and investigate the promise of prompt engineering to tackle some of the weaknesses in these models.
2 Dataset and Methods
In this section, we introduce the dataset used in this work and provide details of how we generate summaries.
Dataset. Form 10-K filings serve as comprehensive annual reports that outline company's financial condition, business overview, and other disclosures mandated by the U.S. Securities and Exchange Commission (SEC). An annual report on Form 10-K usually contains four main parts and sixteen specific items standardized by the SEC to ensure comprehensive coverage of key business aspects (SEC). Of particular interest for our analysis is Item 7, commonly referred to as "Management's Discussion and Analysis of Financial Condition and Results of Operations" (MD&A), where companies offer their perspectives on the preceding financial year's business outcomes.
2

---

Published as a conference paper at COLM 2024
Model
R/S
Words
Num. Total
Dens. %
Num. A
Num. Num. C Num. D
R
17146.70
478.78
2.79
136.90
295.75
46.14
Claude 2.0
223.82
11.09
4.95
4.62
0.92
2.89
2.66
R
17583.14
487.99
2.77
137.83
302.93
47.22
Claude 2.1
229.66
11.56
5.03
4.72
0.97
3.11
2.76
R
17439.68
485.50
2.78
137.63
301.08
46.79
GPT-4
421.51
5.81
1.38
3.34
0.29
1.73
0.46
R
4927.61
146.32
2,97
53.83
81.32
11.17
GPT-3.5
120.12
0.46
0.38
0.28
0.03
0.14
0.01
R
9893.39
260.74
2.64
98.28
139.68
22.79
Cohere
152.78
6.90
4.52
1.42
2.12
0.94
2.43
Table 1: Basic statistics of the reports and the summaries. "R" represents the report, and "S" represents the summary. "Words" denotes the average number of words in the report or summary. "Num." the average number of numeric values. "Dens.% " represents the ratio of numbers over the number of words. "A", "B", "C", and "D" represent different categories of numeric data and are defined in $4.1
We randomly selected 1,000 HTML files of 10-K forms obtained from the Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system. These files are then converted into clean JSON format using target report to do summarization. Additionally, to explore whether LLMs have position biases, we generate shuffled versions of each report by randomly reordering its paragraphs, treating tables as individual paragraphs.
(Loukas et al. al. 2021) and we use Item 7 as the
Models. We evaluate summaries generated by five commercial models with top con- text window length (wl) in July 2023 when we started this work: (1) Claude 2.0 (claude-2.c O, w1=100K tokens), (2) Claude 2.1 (claude-2.1) wl=200K tokens) (3) GPT-
3.5 (gpt-3.5-tuu) 5.t.turbb--116,, wl=16K tokens), (4) GPT-4 (gpt-4-1106- -reviie,, wI=128K to- kens) (Achiam et al. (003), and (5) Cohere (command, w|=100K characters). For Claude 2.0 and Claude 2.1, we set max output tokens to 4,096 and temperature to GPT-3.5 and GPT-4 are used with default typerparameters. We use the Cohere co. summarize API and set the output summary length to "long", format as bullets, high extractiveness, and temperature to 0.3 according to the official recommendation. Except for Cohere, which does not need additional prompts with the specific endpoint, we use simple prompts for summarization. Detailed prompts are shown in Appendix
General analysis of summary. Table presents basic statistics of the report and summary generated by each model. Notably, the summaries produced by GPT-4 exhibit the longest average word length of 421.51. With its substantial 200K-token context length, Claude 2.1 outperforms other models by extracting an average of 11.56 numeric values per summary. A more detailed investigation of numeric values utilization will be presented in Section 4 is worth mentioning that certain reports require truncation to meet the context limit of different models, with an averaged of 436, 143, 12,656, and 7,690 words truncated for Claude 2.0, GPT-4, GPT-3.5, and Cohere respectively.
GPT-3.5 and Cohere are generally not good at summarization based on the simple prompt, reflected by too short summaries, meaningless contents, and referring to almost zero num- bers. We investigate this further in Appendix A Thus, the rest of this paper mainly focuses on Claude 2 and GPT-4.
3 Tracing the Summary in the Input
Obtaining ground-truth summaries for very long inputs is significant challenge, and there can be numerous good summaries for a given document. Therefore, we mainly focus on the behavior of the models and propose computational framework to characterize such
We added Claude 2.1 when been able include
it was out, but Claude 3 came out in February 2024, have not
3

---

Published as a conference paper at COLM 2024
Type
Sentences
Score
1-1
summery sentence: As of March 31, 2020, accumulated deficit was $112.3 million. report sentence: As of March 31, 2020, we had an accumulated deficit $112.3 million.
1.70
summary sentence: For 2019, capital expenditures estimated at $2.9 billion.
0.95
report sentence: We project our E&P capital and exploratory expenditures will be approxi- mately $2.9 billion in 2019.
2-1 summaary sentence: Operating cash flows increased by $98 million due higher 0.94 net working capital inflows.
report sentence The increase primarily due to higher cash inflows for net working capital of $68.5 million and other current assets and liabilities of $28.2 mil- lion.
report sentence 2: Cash flows from operating activities increased 598.0 million in 2018 compared to 2017.
summmryy senfence: Investing cash flows decreased due lower cash paid for acquisitions. report sentence I: The decrease was primarily due lower cash outflows for business acquisitions. net of cash acquired of $56.2 million, partially offsct by higher cash outflows for capital expenditures of $30.1 million.
0.92
report sentence 2: Cash flows from investing activities decreased approximately $28.1
million in 2017.
Abstractive summary sentence: Results of operations: A detailed comparison between the years 2018 and 2017 displays an increase in net income and net sales, while gross profit percentage of net sales experienced decrease.
N/A
report sentences: Results of Operations Consolidated' Comparison of the years ended December 31, 2018 and 2017 |nn For the year ended December 31, 2018, net income was $80.9 million, compared with net income of $57.8 million in 2017. Net sales increased by $215.7 million, or 15.4%, the year ended December 31, 2018, compared with prior year, with increased sales Performance Coatings, Performance Colors and Glass and Solutions $139.9 million, $42.8 million and $33.0 million, respectively. Gross profit increased $39.7 million, 9.5%, in 2018 $455.9 million, compared with $416.2 million in 2017 and, percentage of net sales, it decreased 150 basis points to 28.3%.
Table 2: How generated summaries synthesize information from the input. Blue contents represent the matches from the first report sentence and green contents represent the matches from the second report sentence. "Abstractive" represents the abstractive sen- tences, where the report sentences were manually put together, and yellow is used for matches.
Model
Original Report
Shuffled Report
Pair
Claude 2.0 Claude 2.1 GPT-4 Claude-2.0 Claude 2.1 GPT-4
1-1
31.24%
41.50%
31.96%
38.32%
47.98%
27.70%
2-1
15.33%
14.33%
9.08%
15.37%
14.23%
9.04%
Table 3: Extractive summary sentences statistics. "1-1" represents 1-1 extractive summary sentence with one source report sentence and "2-1" for 2-1 synthesizing summary sentence with two source report sentences.
The first aspect is the extent to which the generated summaries are extractive. Second, we examine the information sources of the generated summaries, iee, which part of the input a summary is derived from.
behavior.
3.1 To What Extent Are Generated Summaries Extractive?
LLMs are generative models and thus in theory generate abstractive summaries. However, it is plausible that LLMs perform minimal paraphrasing of sentences in the input to produce the summary. Furthermore, extractive summaries can better maintain the fidelity to the original input. Therefore, we first attempt to match sentences in the summary to those in the input and measure to what extent generated summaries are extractive.
Measuring extractiveness. We modify the coverage measure in Grusky et al. 20018 to measure the similarity between two sentences with greedy match algorithm. Given summary sentence = (fsl ts., {sy) and a report sentence = {tRI fR2, .R,,.., f after removing stopwords, where 4; is token, the length of S is n, and the length of R is m.
4

---

Published as a conference paper at COLM 2024
At high level, similarity(S, R) is defined as the ratio of matched tokens with a quadratic bonus that rewards longer-matched sequences. Specifically, for each summary sentence, this algorithm processes its tokens iteratively, discovering and aligning the longest matching token sequences from each report sentence. Then, for each report sentence R, we get list M(S, R) = {( (tpp)-...ppppp..--...... (t,, fi...., fj=g) lt E S R consisted of the longest matching token sequences m compared with the summary sentence S. The similarity score is computed as:
similarity(S,R)
Σ (EM(S,
(mm) + 0.1 * [m{3)
(1)
S|]
we find sentences with a similarity score above 0.8 to contain almost verbatim excerpts from the source text, which we define as extractive. We calculate the similarity score between each summary sentence and report sentence. If the top similarity score is greater than 0.8, we consider the summary sentence 1-1 extractive sentence. For each remaining summary sentence, we calculate a similarity score with a combination of two report sentences. If this score exceeds 0.8, we call it 2-1 synthesizing sentence. Examples of 1-1 extractive sentences and 2-1 synthesizing sentences are shown in T
Following a meticulous manual check (Appendix F
Extractive sentences represent 30% to 40% of the summary. Table 3 shows the percentage of extractive sentences. Claude 2.1 generates the most extractive contents with 41.50% of extractive summary sentences, which is much higher than 31.24% of Claude 2.0. This may be a result of the multi-level bullet formats of Claude 2.1. GPT-4 generates comparable ex- tractive contents with Claude 2.0 but with smaller percentage of 2-1 synthesizing sentences. Summaries generated by Claude 2 for the shuffled report exhibit more extractiveness, with 6-7% higher proportions of extractive sentences compared with that of original reports. In contrast, GPTT4 generates fewer extractive sentences from shuffled reports. This observation demonstrates the differing behavior of Claude and GPT-4 when processing incoherent texts.
Analysis of the remaining abstractive sentences. To gain insights into the abstractive capabilities of LLMs, we conduct a case study for the abstractive sentences present in the summaries. Specifically, we find GPT-4 generates abstractive summary sentences with high degree of condensation, such as summarizing net income, net sales, and gross profit in single sentence ( It paraphrases the original information across various locations and compacts it into one single sentence, generating new phrases such as "display an increase in net income". The entire meaning is retained, but the content organization is significantly different from the original report. See the Appendix for similar examples from Claude 2.
3.2 Where Does the Information Come From?
We study this question by examining the distribution of source contents for the summary.
Identify source contents. Based on the extractive analysis, we analyze the distribution of the 1-1 extractive sentences. For each summary sentence, the report sentence with the highest similarity score is treated as its source. If there exist several report sentences with the same score, we choose the one with the top cosine-similarity using sentence embeddings generated by SentenceTTrane Transformers (Reimers & Gurevych, 2019). Then we calculate the position of each source report sentence and visualize the distribution in Figure We also run the same analysis by including the location of 2-1 synthesizing summary sentence sources. The results are similar and presented in Appendix B
Most summary information comes from the beginning. Figure aa) shows that a high percentage of source contents are found at the first 20% of the report models tend to use contents at the very beginning of the input, and pay less attention to the information within the middle and the end part of its context. For example, more than 60% of the summary contents generated by GPT-4 come from the first 20% of the report, and less than 10% of the summary contents are from the middle. GPT-4 also shows a preference for the information at the very end of the report, which is not as salient in the summary generated by Claude 2.
5

---

Published as a conference paper at COLM 2024
Summary Sentences %
60
Claude Claude 2.1
Claude
40
Claude 2.1
60
GPT-4
40
GPT-4
25
30
55. SR
05-0R
35
30
-B- 0S OS-OR
55-5R
SS-OR
* S5-5R
30
20
25
30
20
20
15
10
20%
40%
60%
80%
100%
20%
40%
60%
80%
100%
20%
40%
60% 80% 100%
20%
40%
60%
80%
100%
Position Source in the Report
(a) Source information distribution of different models
(b) Comparison of source information distribution before & after shuffling, and reordered
Figure 1: Source distribution of extractive summary sentences. OR and SR stand for original report and shuffled report respectively, while OS and SS stand for summaries of the original report and shuffled report respectively. For the summary generated from the original report, most information comes from the beginning of the report. However, for the shuffled reports, this trend disappears for Claude but stays for GPT-4.
Shuffled report analysis. To understand whether the observed pattern is a bias in LLM's behavior or a consequence document structure (there is an overview at the beginning of Item 7), we analyze the summaries after we shuffle the input reports. Figure [ig SS-SR shows that the source information evenly distributed for Claude 2. The distribution SS-OR becomes heavy in the beginning when comparing the summary of the shuffled report with the original report, suggesting that Claude 2 seems to recognize the important information even after shuffling. In comparison, GPT-4 maintains strong position bias for the shuffled input: the summary of the shuffled input now favors the beginning part despite that it is essentially arbitrary text from the input
In summary, although we observe strong position bias, it is not "lost in the middle" as in Liu et al. (2023). In fact, our analysis reveals intriguing behavioral differences between Claude and GPT-4.
Numeric Values Utilization
Multimodal financial reports convey crucial information through combination of unstruc- tured textual narratives and structured tabular data. It is important to effectively integrate both data modalities in the summaries. Meanwhile, it is commonly suggested that LLMs can have issues with using numbers (Dziri et al. 2024). Thus, we present detailed analysis of how LLMs leverage numeric values trom both texts and tables in financial reports.
4.1 Basic Numeric Values Analysis
To extract numeric values from the reports, we use a regular expression that matches numbers with commas as thousand separators and an optional decimal part. We excluded numbers from entity names (e.g., "COVID-19", "ATA190"), dates (e.g., "December 31, 2022"), and residual HTML table indices, focusing primarily on financially meaningful values (see Appendix[C [or details).
Based on the source of the numeric values appearing in the summaries, we categorized them into four types:
Type A: Numeric values present only in the report's text.
Type B: Numeric values present only in the report's tables.
Type C: Numeric values present in both the text and tables of the report.
Type D: Numeric values not found the report.
One may hypothesize that it is challenging for LLMs to incorporate information from the table. Thus, numeric values of type may be most likely to show up in the summaries,
Duu to cost reasons, the GPT-4 analysis based on 100 shuffled reports.

---

Published as a conference paper at COLM 2024
while numbers in type B are rarely incorporated. Alternatively, the model might consider type C to be the most important.
Claude 2 demonstrates a more sophisticated use of numbers than GPT-4. As illustrated in Table 1 Claude 2.1 demonstrates a strong ability to use tabular numeric values, with 8.37% of its numeric values belonging to type B, compared to only 4.98% for GPT-4. The difference is statistically sigmificant according to a two-sample f-test (p < 0.001). However, it is noteworthy that in the reports, more than half of the numeric values were exclusively present in tabular formats, suggesting that LLMs tend to prioritize numeric values found in the text and pay relatively less attention tabular data.
Similarly, when comparing the overall density of numeric values in the summaries, Claude 2.1 uses numbers more frequently than GPT--4, with summary number density of 5.03% versus 1.5 1.52% (p < 0.001). This discrepancy can be attributed to GPT-4's tendency to generate longer summaries with fewer numeric values, indicating potential weakness in analyzing and incorporating numeric information, especially from tabular data, compared to Claude.
As we mentioned earlier, GPT-3.5 and Command fail to use numbers in the summaries. Specifically, GPT-3.5's summaries do not include numeric values in 831 of 1,000 reports and only an average of 0.46 number is mentioned. For 46.7% of the reports, Cohere simply generates "unable to comprehend the table". However, for those completed summaries, it extracts an average of 6.90 numbers, which is larger than that of GPT-44 Among those, Command prefers numbers from the tables rather than text (2.12 vs. 1.42), which is quite different from the behavior of other models.
Type D numbers can come from numeric op- erations. Numeric values of type D, which rep- resent mismatches between the summary and the report, accounted for an average of 2.76 instances per summary generated by Claude 2.1 and 2.66 for Claude 2.0. In contrast, GPT-4 has only 0.46 type D numbers. Through hu- man annotation, we identified that these mis- matches often stem from simple operations such as rounding, calculating differences, or computing rates of change (e.g., 86.36% of type D numbers generated by Claude 2.0 come from such operations).
S Summary
Total assets increased 95 to 59.7 billion at
December 31, 2019 compared December $1, 2018, largely due to the acquisitions of Banc Ed and IST.
i R Report
Signiffoont Balance Sheet howd (dollars thouands
December 31, 2019
December 2018
5 Change % Change
Amats
Delt securices
availshh: for
1,643,25
5 697,685
950,572
136.2 Sp
Dale sccuriics
feld
mmaurity
608,660
(588,560)
(10209%
Pertilin Inans,
6,633,011
5,517,750
1,115,721
202 s
One notable example that showcases the LLMs' ability to synthesize numeric information is their accurate representation of numeric val- ues in the correct units, such as billions instead of millions, shown in Figure 2 This exem- plifies the LLMs' ability to extract and retain crucial details, coupled with their proficiency in comprehending and manipulating numeric data coherently throughout the summarization process. However, it is important to note that subset of these type D numeric values could po- tentially be instances of numeric hallucinations, a phenomenon that warrants further investiga- tion, as discussed in Section 4.2
Toul aads 9,000 5 7,702,577 $1,993,372
25,9 sp
TTible 62: etab lostrrtttttnn stdrrrs/ts stering/td earan cre-icchd
retdiige//dd etds te7tt <ds///tt dtsin//tes
itrmunirre /t <tainv/td edsbecaiier H, Incidd sa Me/tt dteiahh
trrptnctpeeeeeeeess celd-lax/dd ids <1a-8tthe//t eeztas addloett dis26:: atdiiee/ thangelns/te <td-stn</td-
reth 3, 635 7251 w/td- stdriw/tt vto-she/ttt std 7 782 357 //t1
its a0 <td-4ine/dd ett> 372 ete inc/te-
Figure 2: The summary sentence presents rounded figure with the unit adjusted as per the remark preceding the table. The input tables are in HTML format as shown in the figure's bottom section.
4.2 Numeric Hallucinations
Numeric hallucinations refer to cases where numeric value in the generated output text is inconsistent with or unsubstantiated by the corresponding source data. Although numeric values of type D are reasonable candidates for hallucinations, we find that other numbers
7

---

Published as a conference paper at COLM 2024
could be hallucinations too, e.g., referring to 75 million in profit as 75 million in debt. Therefore, we resort to manual annotation to assess the type and extent of hallunication 3
Annotation protocol. To comprehensively annotate numeric hallucinations in summaries of lengthy financial reports, we employ the following multi-step protocol:
(1)
Candidate extraction: For each numeric value mentioned in the summary, extract all sentences (quotes) from the source report that contain either the exact same numeric value or values in different formats (e.g,, 1,000,000 and 1M).
(2) Match identification: The annotator examines the list of extracted report quotes for given summary number:
If a quote accurately matches the context and usage of the numeric value in the sum- mary, it is annotated as "No Hallucination".
b. If no matching quote is found, the annotator conducts comprehensive search through the entire report.
b1. If correct match is identified, the corresponding quote is recorded, and the instance is annotated as "No Hallucination. "
b2. If no substantiating evidence is found in the report, the instance is annotated as specific type of numeric hallucination based on the taxonomy definitions.
(3) Annotations and comments: For cases annotated as hallucination type, the annotator provides detailed comments explaining the rationale behind the annotation decision.
This protocol ensures a systematic and thorough annotation process, aiding in the accurate identification and categorization of numeric hallucinations. It emphasizes examining all potential evidence from the source report before finalizing annotations. The lead author performed this analysis and annotated numbers in the summaries of random sample of 20 reports for Claude 2.0, Claude 2.1, and GPT-4.
A taxonomy of numeric hallucination. We identify and categorize four types of numeric hallucinations, collectively forming the first taxonomy for this phenomenón:
The presence of a specific numeric value in a generated text that lacks any corresponding references.
Fabricated Number:
Rounding Error: Discrepancy arising from the rounding off of numeric values.
Arithmetic Error: Numbers generated from incorrect mathematical calculations applied numeric values from the report.
Context mismatch: An inconsistency where the same numeric value applies to different contexts in the report vs. the summary.
Instances of each type of hallucination are shown Table 4
LLMs hallucinate in only about 5% of numerical values. Overall, LLMs hallucinate at low percentage when using numeric values, with total hallucination rates of 6.48%, 3.97%, and 5.74% observed for Claude 2.0, Claude 2.1, and GPT-4, respectively. Context mis- match hallucinations exhibited the highest percentages across all models, which suggests that current models still face challenges in accurately capturing the semantic relationships between numeric data and their corresponding textual descriptions when generating sum maries. In contrast, fabricated number hallucinations occurred at relatively low percentages, indicating that most models demonstrated reasonable performance in avoiding blatantly un- supported numeric claims in their outputs. However, even low rates of such hallucinations can undermine the trustworthiness of the summaries.
Performance analysis: GPT-4 Claude 2. As shown in Table GPT-4 showcased impressive performance by generating zero instances fabricated numbers and arithmetic errors. This can be attributed to GPTT-4ss tendency to produce extractive summaries, where most numeric values are directly copied from the source report, as evidenced by the numeric analysis in Section 4.1 In contrast, Claude attempted to generate more abstractive content, involving higher degree of arithmetic operations, which contributed to its presence of arithmetic error hallucinations. Notably, Claude 2 exhibited a lower percentage of
We experimented with automatic analyses using GPT-4, but GPT-4 has trouble making sense of these statements accurately.
8

---

Published as a conference paper at COLM 2024
Type
Example
%
Fabricated
Summory:
Net incurne for 2018 million, compared to 533 million in 2017.
0.81/0.79/0
number
Report: No exact source.
Arithmmetic
Summaryy: - the company has $730 million in revolving credit facilities and 5600 million in unsecured term loans to fund operations and acquuisitions.
1.21/0.4000
error
Report: Our principal debt obligations
December
2015 consisted
of borrowings under our $750,000 unsecured revolving credit facility, our loan, $250,000 unsecured term lean, $350,000 pub- licly issued senior unsecured notes and five secured morigage loans that were assumed connection with certain of our acquisitions.
$300,000 unsecured
Comments: 300,000 + 250,000 550,000
Rounding error
$234 million compared to loss $87 million 2014, driven by S191 million impairment charge related to the speaker & receiver product $144 million impairment intangible assets.
Sumuary:: net
0.40/1.19/164
Report: <t.d> Impairment of intangible asseta n<</d>> <td> <td>\n</td>> <td>144, //</td>>
Context mismatch
Summary: - marketplace subscription revenue represented 89% 2018 zevenue.
2.43/1.993.28
Report: Marketplace subseription revenue increased $123.1 million in the year ended December 2018 compared year ended December 31, 2017, represented 89% of total revenue in both 2018 and 2017.
Table 4: Taxonomy of numeric hallucinations and examples for each. The last column % represents the hallucinated numbers percentage of all annotated numbers from summaries generated by Claude 2.0, Claude 2.1, and GPT-4. Red represents the hallucinated number and blue represents the correct source.
context mismatch hallucinations compared to GPT-4, suggesting a more consistent semantic grounding when quoting numeric values from the report within the generated summaries.
Prompt Engineering to Improve Use of Numbers
To enhance GPT-4's performance in ex- tracting numeric values, particularly from tables, we design three explicit prompts and one cfain-of-thought (CoT) prompt (Appendix D): NUM to explicitly request the inclusion of nu- meric values. TAB to explicitly re- quest the inclusion of tabular numbers. NUMTAB to explicitly request both numeric values and tabular numbers. CoT to give intermediate reasoning steps of using numeric values.
Percentage %
Model
Prompt
Total Dens. %
A
B
C
D
simple
5.81
1.38
57.47
4.91
29.77
7.85
NUM
14.30
3.38
56.64
4.90
24.48
13.99
GPT-4
TAB
9.40
2.19
62.77
2.13
25.53
9.57
NUMTAB
18.90
4.48
55.03
6.88
21.16
16.93
CoT
11.30
3.49
43.36
19.47
28.32
8.85
Claude
simple
11.56
5.03
40.81
8.37
26.91 23.88
Table 5: Numeric statistics for the summary gener- ated by different prompts.
As shown in Table 5 the NUMTAB prompting strategy enables GPT-4 to extract the highest number of numeric values and achieve the highest sum- mary number density of 4.48% among the five prompt strategies. However, still falls short of the 5.03% density achieved by Claude 2.1 using a sim- ple prompt. Notably, the CoT prompt- ing substantially enhances GPT-4 us- ing tabular numbers, with 19.47% of the summary numbers being extracted exclusively from the report tables.
3.28
3.0
GPT-4
Claude 2.1
2.12
2.0
1.59
1.5
1.0
0,79
0.4
0.53
0.53
0.0
Febricated Number Arrdhmttic Error
Rounding
Context
Pabricaten Number
Arthhmetic Enar
Bounding Enror
Contest Moomatch
Error
Mismanch
simple
CoT
Figure 3: Hallucinated numbers frequency of summaries generated by simple prompt and CoT prompt respectively.
To assess the effectiveness of CoT prompting in reducing hallucinations, annotated random sample of 10
summaries generated by both GPT-4 and Claude 2.1. As shown in Figure much fewer hallucinations generated when using CoT prompts to guide GPT-4 on avoiding specific

---

Published as a conference paper at COLM 2024
numeric hallucinations, but this is not the case for Claude 2.1. This shows GPT-4's better ability to leverage CoT prompting for introspecting its own output.
Related Work
Evaluation of long-form summarization is very challenging traditional evaluation requires reference summaries [Lin 2004; Papineni et al., 2002). . The closest work to ours is Chang et al. (2023), which presents the first study of LLM-based book-length summary. Their focus is on coherence, while our work focuses on the characterization of LLM behavior in multimodal setting. Other work has shown that summaries generated by GPT-3 and current
LLMs are overwhelmingly preferred by humans and these also do not suffer from common
dataset-specific issues such as poor factuality (Goyal et al. 2023
Pu et al. 2023).
A battery of studies examine automatic evaluation using LLMs considering the high cost of human annotation (Min al. 2023 Peng et al. 2023 Gilardi et al., 2023] i.a.). However,
Doostmohammadi et al. (2024) suggest that automatic evaluation can only approximate hu- man judgments under specitic conditions, and their reliability is highly context-dependent.
7 Conclusion and Discussion
We present the first study to characterize summaries generated by LLMs for multimodal long-form inputs. Our results reveal that different LLMs may approach this task quite differently in terms of extractiveness, position bias, and use of numbers. While the position bias of GPT-4 is probably sub-optimal, it is unclear what makes good summary, especially for such long-form inputs. On the common concern about hallucinations, hallucinations do not happen very frequently. Future work may not only focus on the frequency but also the potential impact of such misuse. The underlying causes of our observations also remain unclear; we hypothesize that Claude may have more exposure to enterprise data/ tataappccpaaioons involving structured data and business metrics. This exposure could potentially explain Claude's s stronger ability to use numbers in the summary. However, important to note that all the LLMs used in our work are closed and the hypotheses need further investigation.
find that
Our research community urgently need novel perspectives for evaluating summarization.
The "death" of sunmarization is opportunity for advancing summarization beyond simple relevance and human preference.
Acknowledgements
We thank insightful comments from anonymous reviewers and members of the Chicago Human+ Al Lab, especially Chao-Chun Hsu for his early contribution in building the dataset. We sincerely thank Kaiqiao Han, Haoran Deng, and Yang Yang for their valuable advice on our work. This work has been supported in part by J.P. Morgan Al Faculty Research Award. This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates ("J.P. Morgan") and is not product of the Research Department of J.P. Morgan. Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document not intended as investment research or investment advice, recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction to such person would be unlawful.
10

---

Published as a conference paper at COLM 2024
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar arXiv:2303.08774, 2023.
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit lyyer. Booookscore: A e of book-length summarization in the era of llms. arXiv preprint urXio:2310.00785, 2023.
Ehsan Doostmohammadi, Oskar Holmström, and Marco Kuhlmann. How reliable are automatic evaluation methods for instruction-tuned Ilms?, 2024.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd work- ers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30): e2305016120, 2023.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3, 2023.
Max Grusky, Mor Naaman, and Yoav Artzi Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint ar Xix::884.11283,, 2018.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summa- rization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computa- tional Linguistics. URL httttp::/aalaanthooooyy/ wwwp:///oo..ooo////10110100
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint ar wixx:2307.0011 2023.
Lefteris Loukas, Manos Fergadiotis, Ion Androutsopoulos, and Prodromos Malakasiotis. Edgar-corpus: Billions of tokens make the world go round. arv preprint XXiv:2109.14394 2021.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computationnl Linguistics, pp. 311-318, 2002.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.
Xiao Pu, Minggi Gao, and Xiaojun Wan. Summarization is (almost) dead. arXiv preprint arXiv:2309.0995 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 11 2019. Uttptsss://aaxxx org/
10084
United States SEC. Form 10-k general instructions. URL forms/form10-x.ppp
11

---

Published as a conference paper at COLM 2024
A
Analysis of GPT-3.5 and Cohere's Performance
Model
R/S
avg
std
min
max
report
4927.61
139.03
3465
5000
GPT-3.5
summary
120.12
39.35
26
358
report
9893.39
179.64
8850
10000
Cohere
summary
152.78
30.32
13
231
Table 6: Summary length statistics of GPT-3.5 and Cohere. All the numbers represent the number of words.
A.1 Cohere
Among all the 1000 summaries, 46.7% only generate "I'm sorry, but am unable complete your request because am unable to identify the relevant information to complete the task." This suggests that Cohere fails to deal with tasks which the input length is almost equal to the context limit. However, to be noticed, Cohere is quite good at reading tables in its completed summary, where 30.69% of the numbers are extracted only from tables.
A.2 GPT-3.5
As shown in Figure 6. due the context limit of GPT-335, it could only accept reports with an average length of 4827.61 words and all the reports have to be truncated before summarization. On average, GPT-3.5 generates summaries that are only 120.12 words long due to its shorter input compared to other models. What's more, 831 summaries contain zero numbers and each summary only uses 0.03 tabular numbers on average, which demonstrates that GPT-3.5 fails to process multimodal documents.
B Source Information Distribution of 2-1 Synthesizing Summary Sentences
50
26
Claude 2.1
60
GPT-4
Summary Sentences
Claude 2.1
24
OS
OS-OR
40
GPT-4
22
=
S5-SR
SS-SR
20
..Bx:: SS-OR
40
SS.OR
18
16
Claude 2.0
30
20
-0- 05-OR
20
%6
SS SR
55 OR
10
10
20%
40%
60%
80%
100%
20%
40%
80%
100% 20%
40% 60%
80%
100%
20%
40%
60%
80%
100%
Position of Source the Report
Figure 4: Source location distribution of 2-1 synthesizing sentences.
C
Numeric Values Extraction Details
Dates number:
I' (January February March April |Myy June I July August September !0ctober November (Dcember) \d(1,2},'
HTML table indices:
r
Table \d+:'
Target numeric values:
12

---

Published as a conference paper at COLM 2024
Integer numbers with without commas as thousands separators.
Decimal numbers with or without commas as thousands separators and with or without a fractional part.
r' (?<!)d) (24 (?! (a ja- (?:, ( < (??)))
When extracting numeric values from the report summary, numbers from entity names (e.g., "COVID-19", "ATA190"), dates number (e.g., "December 31, 2022"), and residual
HTML table indices are excluded, focusing primarily on financially meaningful values.
D Prompts
Simple prompt
Summarize the following report.
MD&A:
The following is an MD&A report:
Please summarize this report.
NUM prompt
Summarize the following report.
MD&A:
Please include specific numeric values and key statistics.
TAB prompt
Summarize the following report.
MD&A:
Please include the numeric values in the tables.
NUMTAB prompt
Summarize the following report.
MD&A:
Please include specific numeric values and key statistics. Please include the numeric values in the tables.
Chain-of-Thought (CoT) prompt
Summarize the following report.
13

---

Published as a conference paper at COLM 2024
MD&A:
Let's generate the summary step by step.
1. Read through the entire MD&A report carefully to understand the context.
2. Identify and extract the key topics and insights discussed in the report.
3. Pay attention to any tables presenting numeric data, such as income statements, balance sheets, cash flow statements.
When including numbers in the summary, ensure they are:
a) Explicitly stated values from the original report (do not fabricate numbers).
b) Stemmed from step-by-step verified calculations.
c) Correctly rounded.
d) Appropriately represented with clear context from the original
source.
5. Synthesize the extracted information and numbers into concise summary
that flows logically.
Summary:
14

---

Published as a conference paper at COLM 2024
E Details of Similarity Score Calculating Algorithm
Function greedy_maattee sateedy_matchhie sumtcchsuumryryseteenees report_txt):
5 [], F F
// Preprocessing
for each word w tokenize(summaryy sentence): ) E stop_words if contains both letters and digits: digitss w. letter) Add w_digits, w_letter to 5
punctuation:
else: Add 5
Matching
for each report sentence resen in report_txt:
while length(S):
for takenizeir_sen) if stop words punctuation]
-
seq []
j Length(R):
if S[i] R[jl:
i_prime
p_prime -
while Lorime length(S) A j_prime length(R) S[i_prime] R(j_prime]:
i_prinett
__prime++
length(seq) Lorime
seq sli, i_prine]
1_prime
else:
j++ length(seq) 1f r_sen E F: Append to Fir_sen]
else:
Add (r_sen, [seq]) to F lexgthenhh) 5_ s_token len Length(S)
r_sen_list
for r_sen, token_ses token_seq_list in F.items(); patched__se__ees - for token_seq in token token_seq_list:
matched seq_ Len (length(token seq) length(token_seq) length(token_seq)) similarity_5core - matched_ Len__eeeeee_________________ee__ s_token_len Add (r_sen, similarity_score r r_sen_ list_istt_ (istes top_r_sen, max_similarity_scoree list(]
r_sen_list
// Use SentenceBERT embedding to choose the top natch report sentence with same similarity score summary_emeddóing S____enttTT_emeeddii Saarliingssmmarrsseeeteee
for similarity max_sinilarity_scors
r_sen_list::
similarity_score max_simi
topi_embedding oet_seetBBAR__mmedddintttooo
sen eigeemedddingg- se topi_embedding) cosl - wttp:/mmmmmartttssmmry..ocmmmm cos2 wosiine_siiilarttyyssmmrr..
sen_enbeddding)
if cos2 cosl:
top_r_sen r_sen
Figure 5: Greedy match algorithm for similarity score calculation.
15

---

Published as a conference paper at COLM 2024
F
Sample Sentence Pairs of Different Similarity Scores.
score range
sentence pairs
score
0.6-0.7
summary sentence: Boat segment sales declined due to production disruptions
0.67
from covid-19, but rebounded in the second half of 2020.
report senfence: The boat segment operating earnings were $70.2 million in 2020, a decrease of 7.9 percent compared with 2019, due to lower net sales along with unfavorable impact of absorption resulting from production disruptions, which were partially offset by benefits from cost reduction mea- sures.
summary sentence: Free cash flow $629 million in 2020, enabling debt paydown, share repurchases dividends.
0.65
report sentence: Generated free cash flow of $629.3 million 2020 enabling us to execute our capital strategy as follows:
0.7-0.8
summary sentence: Total operating expenses increased 2.0% $42.7 billion,
0.74
driven by higher salaries, maintenance, and regional capacity costs.
report sentence: The year-over-yeds increase in our pre-tax income on both a gaap basis and excluding pre-tax net special items was principally driven by higher revenues and lower fuel costs, offset part by increases in salaries, wages and benefits, maintenance expenses and costs associated with in- creased regional capacity.
summary sentence: At December 31, 2016, nhi had $2.6 billion invested in 205
0.73
facilities across states. report sentence: At December 31, 2016, we had investments in real estate, mortgage and other notes receivable involving 205 facilities located in 32 states.
0.8-0.9
summary sentence: American reached confidential settlement with Boeing 0.85
regarding financial damages from the 737 max grounding.
report sentence: previously announced in January 2020, we reached confidential agreement with Boeing on compensation related to financial damages incurred in 2019 due to the grounding of the Boeing 737 max aircraft.
summary senlence: Cash from operations dropped $209 million 2019 from $389 million 2018 due to lower earnings.
0.82
report sentence: Cash flows from operating activities decreased to $209.1 million in 2019 compared to $389.0 million in 2018 primarily due to lower earnings, partially offset by favorable changes in working capital.
0.9-1.0
summary sentence: Passenger revenue was $42.0 billion, up 3.3% due to higher passenger demand.
0.92
report senfence: Passenger revenue increased $1.3 billion, 3.3%, in 2019 from 2018 due continued strength in passenger demand resulting an increase in RPMs and a year-over-year increase in passenger load factor.
summarr senience: Company restaurant revenues come from food and bev- 0.94 erage sales company-operater restaurants.
report sentencc:: Company restaurant revenues consist of sales food and
beverages in company-operater restaurants.
>1.0
summary sentence: Net earnings were $374.7 million in 2020 versus $30.4
1.13
million in 2019
report sentence: Net carnings from continuing operations increased to $374.7
million in 2020 from $30.4 million in 2019.
summary senfence: Franchise royalty and other franchise revenues come from royalty income and initial and renewal franchise fees.
1.38
report sentence: Franchise royalty and other franchise revenues represents royalty income, and initial and renewal franchise fees.
Table Sample sentence pairs of different similarity scores.
16

---

Published as a conference paper at COLM 2024
G More Samples of Abstractive Sentences
Abstractive Sentence
summary sentence:
The decrease was primarily due to lower sales in the industrial solutions and fluid handling solutions segments, partially offset by higher sales in energy solutions.
report sentences: The decrease is primarily attributable to decreases of $24.5 million in our Industrial Solutions air pollution control technologies, $18.2 million in custom-designed FCC cyclone systems serving the refinery markets, and $4.4 million in volume decreases in the Company's filtration and pump solutions sales. These decreases in net sales are offset by increases of $10.1 million in the Company's custom acoustical technologies that serve the natural gas power generation markets, $8.1 million in volatile organic compounds ("VOC") abatement solutions from the EIS acquisition and $3.3 million in volume increases in our emissions management and water filtration solutions technologies.
summary sentence: The company has a credit facility with $72.6 million drawn and $104.7 million of available capacity of December 31, 2020. report sentences: <td> Total outstanding borrowings under Credit
Pacility\n</tal (tdd\///td>>>>> <td> ><>>><</t>>>>>> <td> 72, 72,616)n</oe <td>\n</td>> <t>>\n</td>> <td>\n</td> <td> 65, 60, /<//t>> <td> <td>\n</td>>>> </tr>\n<tr> In 2020, the Company made repayments of $2.5 million on the term loan and net
borrowings on the revolving credit lines of $9.5 million, consisting of $10.3 million used to fund the EIS acquisition on June 4, 2020 and $2.6 million used to pay term debt assumed connection with the Mader joint venture, offset by $3.4 million net repayments on the revolving credit facility.
Table 8: Samples of abstractive sentences.
17


Reviews:
[{"review_id": "K8tJl3d5Le", "reviewer": "Reviewer_e6Hp", "summary": "This paper presented a case study of long-form summarization in the finance domain in order to study LLMs’ behaviors when it comes to summarizing long input texts and numerical operations. Specifically, the authors first proposed a computational framework to characterize multimodal long-form summarization in summarizing financial reports and investigated how the information in a given financial report is utilized. Then the authors conducted a thorough inspection and evaluation of how selected LLMs handled numbers in financial reports, and proposed a taxonomy of categorizing errors related to numeric operations. The findings of the experimental results show that LLMs have strong position biases and can be mitigated by shuffling operations.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments! We appreciate your recognition of our contribution in developing a computational framework to characterize multimodal long-form summarization as well as the challenging nature of long-form summarization!\n\n**Re: Single document summarization of news is solved != summarization is solved** \n\nWe agree that claims about single document summarization based on one specific type don't necessarily generalize to other summarization tasks. As we mentioned, summarization is ubiquitous and controversial. We will further clarify this point in the revision.\n\n**Re: The authors may want to cite some prior work on chart-to-summary generation** \n\nThank you for the pointers! We will include the relevant citations in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well-written and well-organized, making it very easy to follow. The problem studied in this paper is well-motivated and took on a rather challenging and urgent task in the age of LLMs. The presented study asked important questions with regard to how LLMs use information in long input text in a specific domain that contains many numbers that are important to the interpretation of the original text. I also like that the authors showed concrete examples when detailedly explaining the four types of numeric value utilization. In addition, the authors also called for new approaches or perspectives of evaluating long-form summarization when the gold standard isn’t necessarily the sole ground truth and when such ground truth is getting more and more difficult to get when the source text gets longer and longer.", "reasons_to_reject": "I don’t have much to say here, but I have some comments and questions that I would like to bring up in the following section.", "questions_to_authors": "1. Regarding the authors’ comment and references about LLM-generated summaries overwhelmingly preferred by humans: I think this is only applied to single document one-sentence new summarization [1]. As some previous work showed , this is definitely not the case for non-news summaries ([2]---[4]). In other words, single document summarization of news is solved != summarization is solved. \n\n2. The authors may want to cite some prior work on chart-to-summary generation ([5]--[7]). \n\n\n[1] https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00632/119276/Benchmarking-Large-Language-Models-for-News \n\n[2] https://dl.acm.org/doi/abs/10.1145/3477495.3531802?casa_token=g-Uw8_87GzwAAAAA:gnrbj6AxvcswhwB7Mjzumbzynm0eG6aB-A1WYPlsKhyGjn-9p12bn71uCp8rde7fxMRgHRwHTMNswg \n\n[3] https://aclanthology.org/2023.findings-acl.593/\n\n[4] NewSumm Keynote 1 by Kathleen McKeown \n\n[5] https://aclanthology.org/2022.acl-long.277/ \n\n[6] https://aclanthology.org/2021.ranlp-1.183/ \n\n[7] https://link.springer.com/chapter/10.1007/978-3-319-46478-7_41", "extra_scores": {}}, {"review_id": "sZZc2KEOOX", "reviewer": "Reviewer_cjmF", "summary": "This paper analyzes the use of LLMs for summarizing financial reports with a specific focus on how LLMs handle numbers since financial reports are often dense with numerical figures and a good summary needs to condense not just text but also numerical data correctly. They also do human labelled case studies on abstractive sentences and certain numerical figures found in the summary to check if these are correct or if they are hallucinations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work on position bias and numeric hallucinations!\n\n**Re: A long way away from ensuring factual accuracy** \n\nWe agree that ensuring factual accuracy is challenging, especially when gold standards become scarce. It involves knowledge grounding, cross-verification, and incorporating human oversight, which is beyond the scope of this paper. \n\nMeanwhile, factual accuracy is not the only criterion for desirable summaries. Therefore, while our work does not directly tackle this issue, we propose a computational framework for characterizing multimodal long-form summarization. By investigating *how* LLMs approach the summarization task, our work takes a crucial step toward evaluating long-form summarization.\n\n\n**Re: Prompting**\n\n- To clarify, we find that LLMs hallucinate at a low rate when using numeric values. However, GPT-4 tends to generate long summaries with few numbers with a simple prompt. Therefore, we design experiments to enhance GPT-4's performance in extracting numeric values.\n- Following your suggestion, we designed a chain of thought (CoT) prompt and conducted experiments on Claude 2.1 and GPT-4. The experiment results on 10 reports are shown below:\n    ```\n    Model        Words\tTotalNum NumDens%\tA\tB\tC\tD\n    Claude 2.1   233.70\t18.90\t 8.09\t\t5.70\t3.00\t5.60\t4.60\n    GPT-4\t     323.40\t11.30\t 3.49\t\t4.90\t2.20\t3.20\t1.00\n\n    Model\\Hallu%\tFabNum\tArithErr   RoundErr   CtxtMis\tTotal\n    Claude 2.1\t1.59\t0.53\t   0.53\t\t2.12\t4.76\n    GPT-4\t\t0.00\t0.00\t   0.88\t\t1.77\t2.56\n    ```\n    The results show that while GPT-4 extracts more numbers, it still lags behind Claude 2.1. Notably, much fewer hallucinations are generated when using CoT to guide GPT-4 how to avoid specific numeric hallucinations. We will update the results in the final version of our paper.\n- We cannot find references to 'speculative prompting'. Could you please give us more information?\n\n\n**Re: Reproducibility**\n\n- Human annotation is often considered the gold standard in existing tasks. Although generating groundtruth summaries for long inputs is challenging, we use human annotation to verify whether a number is hallucinated given relevant contexts, a much more manageable task. We gave a detailed annotation protocol in Section 4.2 to facilitate reproduction, which is a common practice in human annotation.\n- Also, we did try to use GPT-4 to analyze numeric hallucinations automatically and found that GPT-4 was not able to perform this task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They analyze the summaries from both an abstractive and an extractive point of view\n2. They break down numerical figures into 4 types, one of those being \"not found in the report\". This 4th type is analyzed further to see if it was obtained by the LLM doing simple arithmetic or if it a made up hallucinated number. They report hallucinations to be minimal\n3. They also compare Claude 2 and GPT 4 summaries in terms of position bias i.e. are the LLMs able to find the truly important information or are they just focussing in the beginning/end\n4. They compare 4 leading LLMs including Claude 2 and GPT 4, with good conclusions and comparisons", "reasons_to_reject": "1. While this work is a good effort in this direction, we are a long way away from ensuring that the summaries are factually correct.\n2. Some more prompting methodologies such as chain of thought or speculative prompting could have been used to see if hallucinations can be fixed through LLM introspecting it's own output.\n3. Most analysis is done via human labelled case studies and will be hard to reproduce", "questions_to_authors": "--", "extra_scores": {}}, {"review_id": "kxtu8fm6jR", "reviewer": "Reviewer_Lb8C", "summary": "The paper conducts a comparison of various LLM models regarding their ability to summarize sections of financial reports (10-K), which encompass not only textual content but also tabular and numeric data. It highlights two key findings. Firstly, it identifies a position bias among all models, indicating a tendency to prioritize the initial sentences. Upon further analysis, the paper demonstrates that Claude exhibits less position bias compared to GPT-4 when assessed on randomly-shuffled texts. Secondly, the paper reveals that Claude demonstrates a more sophisticated utilization of numbers, including basic computation, compared to other models. Additionally, it categorizes different instances of hallucination by LLMs in handling numerical data.\n\nThe paper introduces intriguing phenomenon, such as Claude exhibiting reduced position bias and superior utilizing of numeric data. However, merely reporting these observations may not contribute significantly to the scientific values. The paper would be further strengthened if it could elucidate the underlying reason for the phenomenon. Another drawback is that it has been evaluated only on a single document type -- 10-K. This raises questions about the generalizability of the findings to the other document types.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments! We appreciate your recognition of financial report summarization as an under-explored domain.\n\n**Re: Scientific contribution of the paper**  \n\nOur core contribution is to propose the first computational framework for characterizing summaries of multimodal long-form inputs, thus providing insights into LLM capabilities for complex real-world scenarios. While what makes a good summary remains an open question, our work is a meaningful first step forward. We will clarify that in the revision.\n\n**Re: Generalizability of the paper's claims**\n\nWe acknowledge the possibility of 10-K reports used for pre-training, but there is no groundtruth summary. The challenge of summarization remains even if the model has seen these documents. We chose 10-K reports for their complexity and practical relevance.\n\nOur framework is readily transferrable to other domains. While we cannot provide formal guarantee, we expect the results from our study to be relevant and at least informative for other domains. \n\n\n**Re: Hypotheses for the findings**\n\nAs the LLMs used in our work are closed, we do not know the details of the models. We propose the following hypotheses and will add to the revision.\n\n- The enhanced capabilities of larger models including fine-grained detail extraction and advanced reasoning capabilities may enable them to use numbers effectively in summaries. In contrast, smaller models might overlook complex tables and detailed numbers.\n- Claude may have more exposure to enterprise applications involving structured data and business metrics. The objectives during fine-tuning of Claude might include tasks that emphasize the importance of utilizing numerical values.\n\n**Re: Average amount of the truncated contents** \n\nWe do summarization on 1,000 reports of each model, with an averaged of 436.44, 143.46, 12655.53, and 7689.75 words truncated for Claude 2.0, GPT 4, GPT 3.5, and Cohere respectively. We will update this in our final version.\n\n**Re: What is the evidence backing this claim?**\n\nThis paragraph highlights our case study on abstractive sentences whose source information cannot be matched automatically. We will modify it to \"Specifically, we find GPT-4 generates abstractive sentences with a high degree of condensation, such as summarizing net income, net sales, and gross profit in a single sentence (Table 2).\"\n\n**Re: Representation of tables and input** \n\nTables are represented in HTML. We use clean texts as inputs for LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents several evaluation results that may be of interest to the research community. Specifically, it demonstrates that Claude outperforms other LLMs in summarization tasks by considering actual content rather than blindly favoring beginning sentences. Moreover, Claude demonstrates a sophisticated utilization of numeric data, including the ability to perform numeric computations.\n\n- The research on financial document summarization remains under-explored. Financial documents present a unique challenge due to their inclusion of tabular and numeric data. Therefore, evaluation results in this under-explored domain could offer valuable insights to the research community.", "reasons_to_reject": "- The scientific contribution of the paper appears to be limited. Its primary contribution lies in contrasting the behavior of Claude2, GPT-4 and other LLMs. However, the significance of these contributions is unclear, as even acknowledged by the authors themselves, who state in the paper, \"...While the position bias of GPT-4 is probably sub-optimal, it is unclear what makes a good summary, especially for such long-form inputs…\"\n\n- The generalizability of the paper's claims is also uncertain. The evaluation is based solely on one type of financial document - 10-K reports, which are publicly available and likely used during the pre-training of the LLM models. It's questionable whether the findings can be applied to other financial documents  or to any document containing tables and numbers.\n\n- Strengthening the paper would involve providing hypotheses to explain its findings. Specifically, the paper reveals that larger models tend to utilize numbers, while smaller models do not. Additionally, it demonstrates that Claude employs numbers in a more sophisticated manner than GPT-4. Offering potential explanations for these phenomena would enhance the paper's impact and interest.", "questions_to_authors": "- It looks like each different model received truncated texts due to the limited context window. It would be great if the paper adds the average amount of the truncated contents. \n\n- In Section 3.1, the paper says “Specifically, we find that the GPT-4’s summary shows a high degree of condensation such as summarizing net income, net sales, and gross profit in a single sentence (Table 2)”. What is the evidence backing this claim? \n\n- How were the tables represented? Were the HTMLs used as an input to LLM?", "extra_scores": {}}, {"review_id": "Nk0itEo0QW", "reviewer": "Reviewer_KNTY", "summary": "The paper presents a case study on summarization capabilities of LLMs. Specifically, the authors focus on summarization of financial reports with several current top commercial LLMs (GPT, Claude, Command).\n\nThe study focuses a lot on processing numbers, which are known to pose various difficulties to LLMs. The authors perform various analyses, including a small-scale manual annotation of number hallucinations. In general, the authors find that number processing is very strong in some of the models, especially Claude.\n\nThe study also analyzes positional bias with LLMs tending to mostly cover information from the beginning of the document, and find interesting differences between Claude and GPT.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the insightful comments! We appreciate your recognition of our study on position bias and numeric hallucinations as well as the potential generalizability of our results.\n\n**Re: Hypotheses for the findings**\n\nThank you for your insightful suggestions. We provide the following hypotheses and will add to the revision.\n\n- Claude may have more exposure to enterprise data/applications involving structured data and business metrics. The objectives during the fine-tuning of Claude might include tasks that emphasize utilizing numerical values. Therefore, it may employ numbers in a more sophisticated manner than GPT-4.\n- Most documents follow a pattern of introducing key points early and GPT-4 may have followed it. Claude's training/fine-tuning data may have incentivized extracting crucial information as summaries. It may also develop context-based summarization strategies rather than over-relying on position.\n\n**Re: Evaluation of the quality of the reports.**\n\nSummary quality is often dependent on the target application/task, hence we do not provide quality scores, which may be misleading. Our study focuses on proposing a computational framework for characterizing multimodal long-form summarization. By investigating *how* LLMs approach the summarization task, we believe that our work is a meaningful first step towards this goal.\n\n**Re:  About sliding window approach** \n\nWe agree that using a sliding window to do long-form summarization is a great idea. However, because the best models can handle most of the input (200K tokens window length for Claude 2.1) and these models boast their ability to handle long inputs, so we would like to evaluate that directly.\n\n**Re: Questions**\n\n- Tabular data is structured while text is unstructured. And they should be processed differently when doing summarization. Therefore, Text+table data is treated as multimodal data.\n- Using exact quantitative values highlights the large differences in number utilization across models. For instance, while Claude 2.1 cites 0.97 tabular data, GPT-4 only cites 0.03. Using percentages alone may not capture as clearly.\n- Thank you for the insightful suggestion. Table 1's model, summary length, and total numbers information help readers learn about our work generally at the very first of the paper. We will try to improve readability by moving the numeric statistics data to a later section.\n- This is a great idea and we will modify it in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The study presents useful findings about the behavior of LLMs, partially in line with previous research, partially providing different or more complex observations than previous research, and partially adding new observations that had not been previously reported.\n- The manual analysis of number hallucinations is particularly interesting, delving quite deep and showing various types of number hallucinations (with some not actually being hallucinations but rather new numbers correctly derived by the LLM from the text).\n- The observed differences in the position bias are new and quite intriguing. This gives us a new finding that probably GPT4 is not great at summarizing documents where the important information is not at the beginning but elsewhere.\n- Although the study focuses on one specific document type, I believe the findings are probably generalizable to a great extent and are thus of general interest and usefulness.\n- The findings are not interpreted in detail, which provides nice future work avenues for trying to explain the observations presented.", "reasons_to_reject": "- The study is only descriptive and observational, it covers in great detail the observed behavior and capabilities of the models, but does not attempt at all at interpreting the findings or hypothesizing about the reasons that lead to the findings. Of course this is somewhat hard since all of the investigated systems are closed-source with only limited information available about them, but still some information is provided by the models' authors and other information have been reported by independent researchers, so it would be good to at least try to link the known differences in the models' design and training to the observed differences in behavior on the examined task.\n- The study mostly does not attempt to evaluate the quality of the reports, claiming there is no gold standard summary for their inputs. This is sad, as although we observe some differences in the models, in many cases we do not know for sure whether the differences mean that one of the models produces better summaries than the other, or whether these are just two possible approaches to the summarization task which are both good. I understand the difficulty in not having the gold standard data, but this should at least be discussed more explicitly and in more detail. For example, the fact that a model produces less numbers in the report, does this automatically mean the report is worse, or may it be that it is a highly abstractive report that interprets the numbers textually without mentioning them and thus this might also be a great report? I am not knowledgeable in the domain of financial reports and their summaries, so I do not know whether a good summary absolutely must contain a lot of numbers (which is somehow indirectly implied by the paper but not explicitly said or discussed). So this is a clear limitation of the paper. Still, I appreciate the manual evaluation of number hallucinations, which goes beyond this and actually does evaluate what is good and what is bad.\n- For models with a short input context window, the authors simply truncate the input. This does not seem right, why not use a sliding window approach?", "questions_to_authors": "These are something between questions and tentative suggestions (which might or might not be good suggestions, so this is just for consideration).\n- Is text+tabular data generally called \"multimodal\" data? I can understand that, but I am more accustomed to \"multimodal\" referring to other modalities than only text, while tables are still text. But maybe my terminology is wrong and yours is correct, I have not specifically focused on working with tabular data so my terminology is not strong there.\n- In table 1, would it make sense to report A B C D as percentage instead of absolute numbers, probably a sort of recall of what proportion of the numbers from the report appear in the summary?\n- Also, table 1 is quite far from it description, could it maybe be moved to a lter part of the paper?\n- Also, A B C D are not very reader-friendly numbers, could they be maybe e.g. X B XB N (teXt taBle teXt+taBle None) or something more interpretable?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies the performance of LLMs on long-form summarization of financial documents. The main contributions are: (1) benchmarking LLMs on this new long document domain and characterizing their behaviors (position bias, extractiveness, etc.) (2) taxonomy of numerical errors in long-form summarization in this domain. 

Pros: The error taxonomy created for numerical errors, which the paper posits are the most critical errors in this domain, is useful. The analysis of how LLMs extract and summarize information rich in numerical data is interesting. 

Cons: 
While the analysis of extractiveness, position bias is interesting, it is not clear what the takeaways from these observations should be. For e.g., there is no discussion of what a "good" summary is, and therefore, not clear how to interpret one LLM being XX% more extractiveness than the another. The paper can be made stronger by analyzing what *gold* summaries look like for this domain and comparing LLM performances against that standard. 

The paper only focuses on one type of factual errors in the summary, and as such, it is unclear how important or prevalent these are compared to other factors. The lack of human evaluation of overall quality makes it difficult to ascertain the utility of any of these LLMs for this task.
Decision: Accept