Which Information Matters? Dissecting Human-written Multi-document Summaries with Partial Information Decomposition (2024)

Abstract: Understanding the nature of high-quality summaries is crucial to further improve the performance of multi-document summarization. We propose an approach to characterize humanwritten summaries using partial information decomposition, which decomposes the mutual information provided by all source documents into union, redundancy, synergy, and unique information. Our empirical analysis on different MDS datasets shows that there is a direct dependency between the number of sources and their contribution to the summary.

heading: 1 Introduction
text: Multi-document Summarization (MDS) consists of providing an abridged version of multiple documents. While some abstractive summarization approaches concatenate all documents into a single input (Johner et al., 2021; Xiao et al., 2022), the large size of input text represents a major challenge in MDS. Therefore, most methods implement two-stage approaches that extract salient text spans based on different heuristics (Lebanoff et al., 2018; Liu et al., 2018; Liu and Lapata, 2019; Zhu et al., 2021). The text is then fed into a summarization model under the assumption that a high-quality summary is based on such information. While earlier work quantifies the properties of human-written MDS based on n-gram matching (Banko and Vanderwende, 2004), we argue that there is a lack of indepth analyses. Without an understanding of the nature of summaries, improving the quality of MDS remains vague and without clear interpretability.
This work sheds light on what information constitutes a high-quality multi-document summary. In particular, we propose to categorise the summary information into information provided by at least one source (union) or by a unique source, redundant information from all source documents, and even new information derived from considering
them jointly (synergy). In information theory, Partial Information Decomposition (PID) decomposes information in the same way to assess how information about a target is distributed among multiple source variables (Williams and Beer, 2010).
We therefore implement PID in MDS and present SPIDer; a novel approach to quantify the degree to which the PID components—such as redundancy or unique information—contribute to a summary.1 We then perform an empirical analysis on human-written summaries from different MDS datasets using our approach. Our results demonstrate that the number of sources has a direct dependence on how they contribute to the summary. We also show that, surprisingly, the order of the source documents matters, and the first three documents are frequently considered as the main source of unique information for any number of sources.
To the best of our knowledge, we present the first fine-grained information analysis in human-written MDS. We open-source SPIDer2 and hope that it helps to enhance the performance of future MDS methods. We suggest that our PID approach could also be used to automatically build MDS datasets that align with human quality in future work.

heading: 2 Information Theory Background
text: Mutual Information (MI) (Shannon, 1948) has been widely used in NLP tasks to quantify the information that a source provides about an output (Li et al., 2016; Li and Jurafsky, 2016; Takayama and Arase, 2019; Padmakumar and He, 2021; Mascarell et al., 2023). However, mutual information is insufficient in the context of MDS, as it can only be applied to pairs of random variables.
Partial Information Decomposition (PID) tackles the multivariate problem and decomposes the information that a set of sources conveys about a target
1Our implementation also works for single inputs. 2GitHub: mediatechnologycenter/SPIDer
5333
into union, redundancy, unique, and synergistic information (Williams and Beer, 2010). To date, PID has only been applied in NLP to measure morphological fusion in Socolof et al. (2022). Figure 1 illustrates the relationships between the different PID components for two sources, X1 and X2, and a target Y . See a textual example in Table 1.
In this work, we consider the PID approach proposed in Kolchinsky (2022), which is based on the definitions of union and intersection from set theory. In contrast to previous work, Kolchinsky (2022)’s approach can be applied to any number of sources, a crucial requirement in our MDS setting.
Formally, given a set of source random variables X = {X1, ..., Xn} and a target Y , redundancy I∩(X → Y ) is the intersection of information among the sources about Y . This redundancy is the maximum information Q we can obtain about Y that is less informative than any of the sources:
I∩(X → Y ) := sup Q I(Y ;Q) | ∀i, Q ⊏ Xi (1)
where ⊏ is an ordering relation that determines when Xi is more informative than Q. Conversely, union I∪(X → Y ) is the minimum information Q we can obtain about the target Y that is more informative than any of the sources:
I∪(X → Y ) := inf Q I(Q;Y ) | ∀i, Xi ⊏ Q (2)
Finally, unique and synergistic information are derived from Eq. (1) and Eq. (2), respectively.

heading: 3 Decomposing Information in MDS
text: We adopt the PID approach in Kolchinsky (2022) to MDS, considering sentences as units of infor-
mation. Formally, let X = {D1, ..., Dn} be a set of n source documents, where each document is a collection of sentences Di = {d1i , ..., d |Di| i }, and a multi-document summary of m sentences S = {s1, ..., sm}. Using Eq. (1) and (2), we define redundancy and union in MDS as follows:
IMDS∩ (X → S) := sup D∈D I(S;D)|∀i, D ⊏ Di (3)
IMDS∪ (X → S) := inf D∈D I(S;D)|∀i, Di ⊏ D (4)
where D = {d1, ..., d|D|} is a collection of document sentences from all possible sentences D,3 I(S;D) represents the MI between the summary sentences S and D, and ⊏ is an ordering relation. We formally define the ordering ⊏ and MI in our MDS setting later in this section.
We then define unique information and synergy using Eq. (3) and (4) as in Kolchinsky (2022)
UMDS(Di → S|X ) = I(S;Di)− IMDS∩ , (5)
SMDS(X → S) = I(S;X )− IMDS∪ , (6) where I(S;Di) and I(S;X ) are the MI between the sentences of a summary and a specific document Di or all source documents X , respectively. Pairwise Mutual Information I(S;D) quantifies the mutual information between summary sentences S and a collection of source sentences D. To compute I(S;D), we measure pairwise mutual information on all summary-source sentence pairs pmi(s; d) = log p(s;d)p(s)p(d) as in Padmakumar and He (2021), using a language model to estimate probabilities.4 To compute p(s; d) the sentences s and d
3Since optimizing over all collections of sentences is computationally intractable, we implement two optimization strategies. First, we assume that a summary should only contain information from the sources. Therefore, we restrict the set of all possible sentences to the sentences from the sources D := P(∪ni=1Di), which results in a finite optimization problem with set size |D| = 2 ∑ i |Di|. Second, since the growth of |D| is still exponential, we implement beam search to find an approximation of the optimal collection of sentences for union and redundancy.
4GPT-2 checkpoint: openai-community/gpt2-large
are concatenated and their joint probability is estimated with the language model. We define I(S;D) as the expected value of all pmi(s; d), which is the sum of all values weighted by their likelihood,
I(S;D) = E s,d∼S,D [pmi(s; d)] (7)
Ordering Relation D ⊏ D′ specifies that a collection of source sentences D′ is more informative than a different collection D. That is, the information that D provides to a summary S is contained within the information that D′ provides to S. Hence, we define our ordering relation as:
D ⊏ D′ ⇐⇒ I(S;D) ≤ I(S;D′) ∧∀s ∈ S : I(S;D)s ≤ I(S;D′)s
(8)
where I(S,D)s = Ed∼D[pmi(s; d)]. Note that the second component of Eq. (8) guarantees that all the information provided by the individual sentences in D is also contained in D′.

heading: 4 PID of Human-written Summaries
text: We analyze the information components of humanwritten summaries from multiple MDS datasets using our framework. Specifically, we consider a random sample of 100 instances per dataset and number of sources, if available (see Table 2).5

heading: 4.1 Datasets
text: MultiNews MDS dataset of news articles from the website newser.com, where summaries are written by professional editors. The sources are the cited articles in each summary (Fabbri et al., 2019).
WikiNewsSum The summaries are full articles from the collaborative news platform Wikinews.org and the sources are the cited references. Sources are at least 1.5 times larger than its summary on a character basis (Calizzano et al., 2022).
5Samples with less than 100 instances: MultiNews with 9 (89 instances) and 10 sources (33); WikiNewsSum with 8 (89 instances), 9 (77), and 10 sources (51).
DUC2004 This dataset consists of 50 sets of 10 articles from the Associated Press and New York Times newswires, each paired with four handwritten summaries. Humans were required to summarize each source independently before writing a multi-document summary of up to 665 characters.6
WCEP The dataset comprises summaries of news events from the Wikipedia Current Events Portal and the corresponding cited sources (only 1.2 on average). The set of sources are complemented with related articles from the Common Crawl archive (Gholipour Ghalandari et al., 2020), which results in a rather synthetic MDS dataset. Summaries are short, up to 30-40 words; hence, we only consider a subset consisting of multi-sentence summaries of 10 sources each.

heading: 4.2 Results
text: We compute SPIDer scores on all samples and compare the results among datasets and number of sources. Due to length differences in both summary and sources across datasets, we normalise all values by the total mutual information between the summary and the source documents I(S;D1, ..., Dn).
We observe in Table 3 that synergy is negligible in all datasets, and therefore, union represents the total mutual information (see Eq. (6)). Figure 2 compares redundancy and unique information across different numbers of sources. The results show that redundancy decreases with the number of sources, while the unique information increases. That is, the more sources, the more they contribute individually to the summary. Additionally, DUC2004 scores are comparable to MultiNews and WikiNewsSum with 10 sources, whereas WCEP
6https://duc.nist.gov/duc2004/
shows significantly lower redundancy and higher unique information. WCEP is extended with Common Crawl articles, so it is unclear how these additional articles should contribute to the summary in a MDS task. The difference in SPIDer scores with the other datasets highlights the importance of using real MDS datasets for this task.
To get better insights into the individual contributions of the sources, we analyze their variance, where a value of 0 indicates that all sources contribute equally to the summary. Table 4 demonstrates that variance, and hence, the variability of their individual contributions, increases with the number of sources. Furthermore, we compute the frequency at which each source contributes the most with their unique information (Figure 3). Interestingly, we observe that the first three sources consistently contribute the most in all datasets, regardless of the number of sources. Similarly, Wolhandler et al. (2022) report that the information in multi-document summaries is often covered by a single source document. Our results also highlight a strong bias towards the order of the sources when summarizing multiple documents, consistently observed across the datasets.

heading: 5 Measuring Synergistic Information
text: Since synergy is negligible in the analyzed MDS summaries (see Section 4.2), we perform an additional experiment to assess whether our approach can measure synergistic information using the MultiRC dataset (Khashabi et al., 2018). Specifically, MultiRC is a reading comprehension dataset consisting of multi-sentence paragraphs and multiplechoice questions, which also specifies the sentences that are required to answer each question. Most importantly, the dataset ensures that correct answers can only be derived by considering multiple sentences jointly. That is, synergistic information should be prominent in correct answers.
To apply our PID approach, we transform MultiRC into a MDS dataset, where each instance comprises the set of sentences (sources) required to answer a question, and a question-answer pair concatenated into a single sentence (summary). Although both correct and incorrect answers share the same question, the former should result in higher synergy than the latter. We also generate unrelated instances for each set of source sentences, where the question-answer pairs are randomly sampled from a different paragraph (see Figure 6).
Table 5 confirms that synergistic information is the dominant information component and correct answers achieve the highest synergy. However, synergy is also present in unrelated instances. Given that synergy represents new information, this raises the question whether it could also be an indicator
of hallucination. This is an interesting research direction for future work, since more experiments are needed to support this hypothesis.

heading: 6 Conclusion
text: We characterize the information present in humanwritten multi-document summaries to get insights into what information comprises a high-quality summary. In particular, we suggest to decompose the mutual information that the source documents provide about a summary and propose SPIDer, a novel approach to quantify such information using partial information decomposition. We then analyze human-written summaries from widely used MDS datasets. The results reveal that redundancy decreases, whereas unique information increases with the number of sources. Furthermore, the order of the documents has an impact on the summarization process, as the first three documents contribute the most in terms of unique information.
Limitations
Some limitations of our work can be traced to the beam-search-based approximation caused by the intractable sentence search space, or due to using sentences as our base unit of information. We also note that despite the remarkable recent advances, large language models’ probability distributions possibly diverge from the true underlying distribution. Their approximation is, however, continuously improving and future models can directly be substituted into our method.
Ethics Statement
From an ethical perspective, it is important to underline the importance of transparency when using language models, as they are becoming nearly indistinguishable from human writers. We advocate for clear transparency when they are used. Our
work promotes interpretability in the space of multidocument summarization, and we hope both interpretability and transparency will be cornerstones for future work in the field.
In all our experiments, we rigorously follow the ACL Code of Ethics, using pre-existing opensource benchmark datasets where privacy concerns were already addressed by the respective authors.

heading: Acknowledgements
text: This project is supported by Ringier, TX Group, NZZ, SRG, VSM, viscom, and the ETH Zurich Foundation.

heading: Future Work
text: Their approximation is, however, continuously improving and future models can directly be substituted into our method