SKGSum: Structured Knowledge-Guided Document Summarization (2024)

Abstract: A summary structure is inherent to certain types of texts according to the Genre Theory of Linguistics. Such structures aid readers in efficiently locating information within summaries. However, most existing automatic summarization methods overlook the importance of summary structure, resulting in summaries that emphasize the most prominent information while omitting essential details from other sections. While a few summarizers recognize the importance of summary structure, they rely heavily on the predefined labels of summary structures in the source document and ground truth summaries. To address these shortcomings, we developed a Structured KnowledgeGuided Summarization (SKGSum) and its variant, SKGSum-W, which do not require structure labels. Instead, these methods rely on a set of automatically extracted summary points to generate summaries. We evaluate the proposed methods using three real-world datasets. The results indicate that our methods not only improve the quality of summaries, in terms of ROUGE and BERTScore, but also broaden the types of documents that can be effectively summarized.1

heading: 1 Introduction
text: Automatic summarization is one of the most effective solutions to help people read lengthy content quickly and comprehend key information that interests them. Furthermore, well-structured summaries can significantly improve the accessibility of information by offering organized key points.
In linguistics, the notion that texts of the same type display similar structural features is widely accepted in Discourse Structure (Brown and Yule, 1983; Choubey et al., 2020) and Genre Theory (Swales and Swales, 1990; Martin, 1992).
*Kaiqi Zhao is the corresponding author. 1The code used in this paper will be released at https:
//github.com/77-qiqi-wang/SKGSum/.
Texts of the same genre share similar structures to meet specific communicative objectives. These structures have evolved based on cultural norms and the shared expectations of both the writer and reader (Swales and Swales, 1990).
Building on Genre Theory, numerous educational research studies strongly advise students or new writers to write structurally articulated summaries to enhance their readability, such as those related to news articles (Yang, 2015; Whiting et al., 2018), government reports (Keepnews, 2016), legal documents (Makdisi and Makdisi, 2009; Kurzon, 1985), medical reports (Yuan ke and Hoey, 2014), and research articles (Hill, 1991). Typically, a structured summary comprises several summary points, each representing a specific piece of information from the source document. We find that various open-source long document summaries exhibit certain structures (Hill, 1991; Yuan ke and Hoey, 2014; Whiting et al., 2018; Yang, 2015), as elaborated in Section 2.1.
However, most existing text summarization methods (Liu et al., 2022; Ravaut et al., 2022; Xu et al., 2022a; Yao et al., 2023; Phang et al., 2023) neglect summary structures. While many researchers agree that structured information is essential to summarization, most of the current models focus on the source document structures or the relations between source sentences instead of the structure of summaries (Balachandran et al., 2021; Xiao and Carenini, 2019; Cao and Wang, 2022; Qiu and Cohen, 2022). When implementing these models, a significant concern arises due to their tendency to overly emphasize certain sections while neglecting others. For instance, most news articles follow the inverted pyramid model under which the most crucial details (Introduction) are presented first, followed by supporting information (News Details), and then additional background or related commentary (Reactions). As observed in Table 1, when summarizing news articles from the Daily
1857
Mail2, existing State-of-the-Art (SOTA) summarizers fall short in adequately covering the three crucial summary points: “Introduction”, “News Details”, and “Reactions”, when compared to humangenerated summaries. More specifically, abstractive summarizers omit information regarding "Reactions" and place lesser emphasis on “News Details”. This is also observed in the knowledgeguided method (Dou et al., 2021), which similarly lacks coverage of “Reactions” and under-represents “News Details”. Table 1 also shows that Large Language Models (LLMs) like GPT-4 lose “Reactions” information, while Llama2 overly concentrates on “Reactions”, thereby losing details from the “News Details” section and introduces incorrect information. These discrepancies can be attributed to their primary focus on the most prominent information, leading to information loss.
The challenge in summarizing lengthy documents often arises from adopting an appropriate summary structure during the generation process. We argue that an effective document summarizer should generate structured summaries that cover diverse summary points, such as ’Introduction’, ’News Details’, and ’Reactions’, which originate from the source documents. While some studies recognize the importance of summary structure, they require manual assignment of each sentence in the source document to a predefined summary struc-
2The web link of the news: https://www.dailymail.co. uk/news/article-3026101/.
ture label (Gidiotis and Tsoumakas, 2020; Elaraby and Litman, 2022). Alternatively, other research relies on human input to create summary structures and manually generate summary structure labels (Wang et al., 2023). Obviously, such a method is practical only for documents that have a clear, consistent, and fixed summary structure, such as legal documents.
To overcome the above challenge, we propose a Structured Knowledge-Guided Summarization method (SKGSum). SKGSum employs a set of key phrases, automatically extracted from humanwritten summaries (as detailed in Section 4.1), to represent each summary point. These key phrases are utilized to identify relevant words or sentences in the source document through a summary pointdocument alignment layer. This method enables SKGSum to effectively capture pertinent information for each summary point, thus addressing the issue of information loss. However, training SKGSum still requires summary structure labels. To remove the need for these labels during training and to broaden the application scope, we introduce a variant of SKGSum, named SKGSum-W. This variant synthesizes information from all summary points to generate a comprehensive summary.
Our contributions are summarized as follows: (1) We propose a Structured Knowledge-Guided Summarization (SKGSum) to identify essential information for each summary point from the source documents. (2) We propose SKGSum-W, a vari-
ant of SKGSum that does not require summary structure labels. (3) Experiments on three open datasets show that SKGSum improves the existing summarizers by up to 7.05% and 1% in terms of ROUGE scores and BERTScores, respectively.

heading: 2 Motivation and Problem Formulation

heading: 2.1 Motivation
text: As mentioned in Section 1, high-quality summary is always structured. In this regard, we analyzed the summaries of three real-world datasets, namely CNN and Daily Mail (CNN/DM) News Articles (Hermann et al., 2015; Nallapati et al., 2016), Government Reports (Huang et al., 2021), and New Zealand Judgments (Wang et al., 2023), and observed that a common summary structure exists for each type of document. Table 2 shows some examples of our findings.
Previous studies, however, either overlook this aspect or require human-predefined structure labels. Motivated by this gap, we propose utilizing summary points as guiding information to generate structured summaries. This approach aims to harness the inherent structure in human-written summaries, applying it more broadly across various types of documents.

heading: 2.2 Problem Definition
text: Our approach aims to generate structured summaries for lengthy documents. Specifically, let Dtrain = {(D1, S1), (D2, S2), . . . , (DN , SN )} be a dataset comprising N documents and their corresponding summaries, where Di and Si represent the i-th source document and its human-written summary (also known as the gold summary), respectively. Each document is represented as a set of words, i.e., Di = {wi1, . . . , wi|Di|}.
The gold summary Si for the i-th document is partitioned into p gold summary parts according to p summary points, i.e., Si = {S1i , S2i , . . . , Spi }, where Ski is the summary of the k-th summary point for document Di, represented as a set of m words Ski = {ski1, . . . , skim}. The learning objective of structured summarization is to train a text summarizer M to generate a summary Gi = {G1i , . . . , Gpi } of p summary points for each document Di such that the sum of point-wise differences between the gold summaries and the generated summaries, i.e., ∑N i=1 ∑p k=1 d(S k i , G k i ), is minimized. Here, d(·, ·) is a distance function.

heading: 3 Methodology
text: Most summarizers employ a single document encoder and a single decoder, as illustrated at the top of Figure 1. Such an architecture cannot generate content regarding different summary points. Alternatively, one can train separate decoders to decode different parts of a summary as illustrated in the second part of Figure 1. To address the limitations of previous works, we propose a Structured Knowledge-Guided Summarization method (SKGSum) in Section 3.1. We further discuss a variant of SKGSum, named SKGSum-W, for datasets without gold part summaries in Section 3.2.

heading: 3.1 Structured Knowledge-Guided Summarizer
text: Drawing inspiration from the human writing process, SKGSum employs summary points as guiding information to find the content relevant to each summary point, as illustrated in part 3 of Figure 1.
The principal innovation of SKGSum lies in the Structure-Guided Document Encoder (Sec-
tion 3.1.2), which refines the encoder’s output to generate summary point-specific representations for the sentences or words in the source documents. These representations are subsequently utilized as input for the decoder to generate summaries specific to each summary point.

heading: 3.1.1 Document Encoder and Decoder
text: We design SKGSum with a flexible architecture that can accommodate various document encoders and decoders, catering to different scenarios, i.e., extractive and abstractive summarization. Formally, a document encoder converts a source document D = {w1, w2, ..., w|D|} to a sequence of ddimensional latent vectors h1, h2, ..., hn ∈ Rd. For extractive methods, each latent vector corresponds to a sentence in the source documents (n << |D|). For abstractive methods, each latent vector corresponds to a word (n = |D|). The document decoder generates a sequence of predictions {g1, g2, ..., gm} from the document vector representation {h1, h2, ..., hn}.
In this paper, we adopt the encoders and decoders in ExtSum-LG (Xiao and Carenini, 2019) and BART (Lewis et al., 2020) for extractive and abstractive summarization, respectively.

heading: 3.1.2 Structure-guided document encoder
text: Figure 2 demonstrates the architecture of our proposed structure-guided document encoder. Given the latent vector representations of sentences or words, i.e., {h1, ..., hn} and p summary points as inputs, the structure-guided document encoder outputs the source document representation {ĥk1, ..., ĥkn ∈ Rd} specific to the k-th summary point. These summary point-specific document representations will then be fed to the decoder to generate summaries regarding each summary point. The structure-guided document encoder contains three components: summary-point encoding, pointdocument alignment, and information fusion.
Summary-point Encoding Layer. We employ a set of key phrases pertinent to summary points
as guiding signals to identify relevant sentences or words for each part. Specifically, we encode each point into a latent vector representation, which is subsequently utilized in the point-document alignment component to align sentences or words from the source document with summary points. The summary-point encoding layer takes two steps.
In the first step, we extract a set of key phrases from ground truth summaries associated with the summary points (details are elaborated in Section 4.1). Given the key phrases, we employ the document encoder to initialize the embeddings of these key phrases. Formally, we denote the embedding of the i-th word for the k-th summary point as pki ∈ Rd. It is noteworthy that these word embeddings will be updated during the training process.
In the second step, we utilize Multi-head Pooling (MHP) (Liu and Lapata, 2019a) to learn the summary-point representation pk ∈ Rd of the k-th part from the word embeddings pk1, ..., pkn:
αzij = (W P z pki) T (WKz pkj)
headz = ∑
i
∑
j
exp(αzij)∑ j′ exp(α z ij′) WVz pkj
pk = Wo [head1∥head2∥...∥headr] ,
(1)
where WPz ∈ Rd ′×d,WKz ∈ Rd ′×d,WVz ∈ Rd ′×d are learnable weights for the z-th attention head, Wo ∈ Rd×rd′ is a learnable weight matrix to aggregate the outcomes from the r attention heads. The MHP extracts and aggregates essential information from the word embeddings to get a fixed-length vector representation for each summary point.
Point-Document Alignment Layer. Some previous works fall short in considering the interrelations between different summary points (Elaraby and Litman, 2022; Wang et al., 2023). However, sentences in source documents can pertain to multiple sections of the summary (Butt, 2013; BanksSmith, 2019). For example, sentences containing information about the news story can be crucial to both the “Introduction” and “News Details” parts.
To address this problem, we leverage summary points to identify relevant sentences or words from the source document. Specifically, we compute the semantic similarity between each sentence or word and the summary points, utilizing their latent vector representations. This approach allows us to determine a relevance score for each sentence or word in relation to each summary point. Then, we acquire a sentence- (or word-) specific summary point repre-
sentation, denoted as h̃ki , by aligning the relevance score with the summary point representation:
h̃ki = aikpk,
aik = exp(hTi pk)∑ k′ exp(h T i pk′) , (2)
where aik is the relatedness of the i-th sentence or word to the k-th summary point. The notation pk denotes the k-th summary point representation obtained from Equation 1.
Information Fusion Layer. The information fusion layer combines the sentence- (or word-) specific summary point representation, i.e., h̃ki , and the latent representation of the sentence or word, i.e., hi, to generate a unified representation for each sentence or word. The unified representation incorporates the information from the context of the words (or sentences) in the source document and their related summary point. The fusion layer can also incorporate additional knowledge Xi ∈ Rd for each word or sentence, such as section representations, sentence relations, previously selected sentences, etc. Specifically, the extra knowledge Xi can be incorporated by concatenating it with the corresponding h̃ki and hi. The information fusion layer is defined as:
ĥki = Wf [ hi∥h̃ki ∥Xi ] , (3)
where Wf ∈ Rd×3d represents a learnable weight matrix. The information fusion layer will output p representations for each sentence or word, each corresponding to a summary point. Then, the decoder utilizes the representations to generate the summary for the k-th summary point.
Following previous works (Xiao and Carenini, 2019), we also apply cross-entropy as loss functions. More details can be found in Appendix A.

heading: 3.2 Structured Knowledge-Guided Summarizer Without Label Requirement
text: Previous methodologies (Gidiotis and Tsoumakas, 2020; Elaraby and Litman, 2022; Wang et al., 2023) and SKGSum require structure labels of ground truth summaries for training. However, obtaining the structure labels demands significant human effort. To mitigate this limitation, we introduce a variation of SKGSum that eliminates the necessity for structure labels within each ground truth summary. We refer to this new variant as the
Structured Knowledge-Guided Summarizer without Label Requirement (SKGSum-W). SKGSumW is designed to adapt to a wider range of summarization scenarios, offering flexibility in scenarios where structured labels are not available.
Specifically, SKGSum-W retains most settings from SKGSum and adds an additional step at the end of the Fusion Layer to integrate information specific to all summary points. The fusion layer outputs p structure-aware representation for each document, each corresponding to one of the summary points. To ensure compatibility with most existing summarizers, we provide two distinct designs suited for extractive and abstractive summarizers, respectively.

heading: 3.2.1 Extractive summarizer version
text: Extractive summarizers aim to determine each sentence in source documents into a binary class label (selected or unselected) to combine selected sentences as generated summaries. This means that the length of selected sentences is fixed. Thus, we concatenate the structure-aware representation ĥki ∈ Rn×d, obtained from Equation 3, by sentences i. The structure-aware representation for extractive methods is defined as ĥ ∈ Rn×pd:
ĥ = [ĥ11∥...∥ĥp1, ĥ12∥...∥ĥp2, ..., ĥ1n∥...∥ĥpn]. (4)

heading: 3.2.2 Abstractive summarizer version
text: The decoder of abstractive summarizers needs to process all information sequentially and always use a decoder with fixed-dimension (e.g., d-dimension) hidden states. Thus, we concatenate the structureaware representation ĥki ∈ Rn×d, obtained from Equation 3, by summary points k. This results in a unified, structure-aware representation, denoted as ĥ ∈ Rpn×d, as illustrated in Equation 5. An added benefit of this process is that it enables the decoder to sequentially process information from each part of the source document.
ĥ = [ĥ11...ĥ 1 n, ĥ 2 1...ĥ 2 n, ..., ĥ p 1...ĥ p n]. (5)
We utilize the structured-guided document representations ĥ, obtained by Equations 4 and 5, as the input for the decoder. This strategic input enables the decoder to generate a complete summary informed by the summary points. Consequently, SKGSum-W can be trained on datasets with complete and undivided gold summary without the need for structure labels.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets. We evaluate our proposed methods on the following open datasets: (1) CNN/ DM News Articles, a widely used dataset for summarization tasks, comprising over 300K news articles sourced from CNN and Daily Mail websites (Hermann et al., 2015; Nallapati et al., 2016), with an average length of 750 words; (2) Government Reports, a dataset from the U.S. Government Accountability Office website (GAO) encompasses 12,228 reports, with an average length of 9,409 words (Huang et al., 2021); (3) New Zealand Judgment Dataset (NZJD), a law dataset that includes 6,155 judgments, with 2820 words on average, delivered by New Zealand courts from 2014 to 2021 (Wang et al., 2023)3.
GAO and NZJD include the ground truth summary for each part. In contrast, CNN/DM only contains the ground truth for the whole document, thereby can only be used in SKGSum-W.
Baselines. We compare with the following baseline models: (1) Traditional Summarizers, including BART (Lewis et al., 2020)4, MemSum (Gu et al., 2022), Seqo (Xu et al., 2022b), and GSum (Dou et al., 2021); (2) Summarizers using document structure, including ExtSum-LG (Xiao and Carenini, 2019) and Hiergnn (Qiu and Cohen, 2022); (3) Summarizers using summary structure, including IPS (Wang et al., 2023) and SSE (Wang et al., 2023); (4) Large Language Models, including GPT-series models (e.g., GPT-3 (Brown et al., 2020), GPT-3.5, and GPT-4 (OpenAI, 2023)) and Llama2 (Touvron et al., 2023).
Given the substantial cost of utilizing GPT, evaluations are performed on randomly selected subsets of the three datasets.
Key Phrases Extraction for Summary Points . Key phrases associated with summary points can be determined automatically using Large Language Models, such as ChatGPT. Initially, we randomly select several summaries from the same document type and use them as inputs to ChatGPT with the prompt, “Find a common and simple text structure in these examples. Based on the analysis of the provided text structure, provide ten possible highfrequency words for each part, excluding common
3All cases are accessible via the New Zealand Legal Information Institute website: http://www.nzlii.org/
4In this paper, we utilize the BART-base for efficiency considerations.
stop words”. GPT provides an initial summary structure and each section’s key phrases. Since SKGSum and SKGSum-W update the key phrase representation during training, they are robust to minor errors from ChatGPT. Experiments regarding the robustness are discussed in Section 4.3.
Evaluation Metrics . We employ ROUGE-1, ROUGE-2, and ROUGE-L, the prevalent evaluation metrics in text-generation tasks, to assess the performance of all methods. We utilize the summary-level versions of ROUGE-L, following the MemSum (Gu et al., 2022). Besides ROUGE scores, we employ BERTScore (Zhang et al., 2020) as an evaluation metric to match the generated summary and gold summary by their latent semantic similarity.
Hyper-parameter Setting. For the proposed structure-guided document encoder, the number of heads in MHP is set to 8, and all learnable parameters are initialized by a standard normal distribution. The hidden vector dimensions d′ = d are set to the same value as the backbone encoders or decoders.

heading: 4.2 Overall Comparisons
text: Table 3 presents a comparison between previous representative summarizers, large language models, and our proposed methods. Table 4 demonstrates the effectiveness of SKGSum and SKGSumW when applied to existing encoder-decoder summarizers. From the results, we can conclude:
• Compared to the conventional methods, methods that consider structure information significantly outperform conventional methods. This is because the conventional methods may lose information regarding some summary points as illustrated in Table 1.
• Compared to methods that leverage summary structure, SKGSum and its variant outperform the state-of-the-art method SSE (Wang et al., 2023) in generating the whole summary, as well as each individual summary point. Notably, SKGSum shows significant advancements in lengthier sections, like Fact and Reason on the NZJD dataset. This superior performance can be attributed to SKGSum’s ability to constrain the content of each summary section, informed by the entirety of the guidance information and the overall summary structure.
• Comparing SKGSum and SKGSum-W in Tables 3 and 4, SKGSum generally scores slightly higher than SKGSum-W. However, it’s crucial to remember that SKGSum-W does not require labels for each part of the gold summary.

heading: 4.3 Sensitivity to the Extracted Key Phrases
text: Since key phrases for summary points extracted by ChatGPT may vary, it is crucial to assess the effect of the summary point representation initialized by the extracted key phrases. To this end, we compare the summary point representations obtained
from the extracted key phrases (as detailed in Section 4.1) with random initialization. We test with SKGSum on the New Zealand Judgment dataset and report the results in Table 5. Notably, the GPTbased key phrase extraction process enhances the ROUGE scores for most of the summary sections, albeit the differences are subtle. This can be attributed to the trainability of the summary point representation. However, the GPT-based keyphrase extraction method may offer a better initialization, leading to better results.
Additionally, we evaluate GPT’s capability to capture the structure of summaries, with the details of this test outlined in Appendix B. After comparison between GPT observed summary structures with reference structure labels, we obtain 83% F1 scores and 0.26 Hamming Loss in CNN/DM datasets; 80% F1 scores and 0.31 Hamming Loss for Government reports test; and 79% F1 scores and 0.32 Hamming Loss for New Zealand Judgment comparisons. The high F1 and low Hamming Loss represent the GPT’s ability to mine the summary structure for input summaries. However, it is noteworthy that while GPT is adept at mining structures, it cannot generate high-quality summaries because GPT does not leverage the summary structure.

heading: 4.4 Case Study
text: Table 1 presents summaries generated by SKGSum using BART (Lewis et al., 2020) as the backbone.
SKGSum notably enhances the abstractive summarizers. The resultant summary encompasses all summary points with essential information. Compared to the knowledge-guided summarizer, GSum, SKGSum produces more structured and complete summaries, indicating the efficacy of the knowledge utilized in our method for generating news summaries. When compared to large language models such as GPT-4 and Llama2, SKGSum yields summaries that are well-structured and com-
plete. The results necessitate a specialized design for structured summarization. Similar findings can be observed in another case study on judgment data (Appendix C).

heading: 4.5 Comparison to Large Language Models
text: We compare our model with two most recent versions of GPT-3.5 models5, Gpt-3.5-turbo and textdavinci-003, and GPT-46. Among them, GPT-3.5turbo is tailored for handling dialogue, while textdavinci-003 is designed for general NLP tasks.
In this experiment, we randomly selected 50 cases from the test sets of each dataset. The results, presented in Table 6, show the superiority of SKGSum over the GPT-series models owing to its specialized design for structured summarization. An in-depth discussion of this comparison can be found in Appendix D.

heading: 5 Related Work
text: Extractive and Abstractive Summarizers. Existing summarizers can be roughly divided into two types: extractive and abstractive summarizers.
Extractive summarization classifies the sentences in the source document and determines whether a sentence should be included in the summary. NeuralSum (Cheng and Lapata, 2016) is one of the earliest methods using Encoder-Decoder structure. Recent research like ExtSum-LG (Xiao and Carenini, 2019) uses the RNN-based encoder for multi-level representation of sentences by considering document and sentence structure. Then, BERT encoder (Liu and Lapata, 2019b; Sotudeh and Goharian, 2022; Liu and Lapata, 2019b) and Transformer encoder (Zhang et al., 2019; Ruan et al., 2022; Zhang et al., 2019) are introduced to replace RNN-related encoders. Besides, MemSum (Gu et al., 2022) introduces reinforcement learning to reduce redundancy in generating summaries by considering selection histories.
Abstractive methods generate summaries word by word. BERT (Devlin et al., 2019) and BART (Lewis et al., 2020) are representative models in this type. Based on these models, recent studies introduce new knowledge to improve the summarization performance, such as Document Segmentation (Moro and Ragazzi, 2022), Role Labeling Detection (Elaraby and Litman, 2022),
5https://platform.openai.com/docs/models/ gpt-3-5
6https://platform.openai.com/docs/models/ gpt-4
Entity Aggregation (González et al., 2022), Key Phrases Detection (Liu et al., 2021), and Guidance Signals (Dou et al., 2021).
Knowledge-Guided Summarization. Recent research shows that introducing guiding information into summarizers can improve summarization quality. We refer to this approach as Knowledge-guided summarization, which combines aspects of both aspect-based and query-based methods. Because the guiding information can be any type, such as keywords (Li et al., 2018a), closest summary (Cao et al., 2018), key sentences (Wang et al., 2022) from the source document, salience allocation expectation (Wang et al., 2022) or relational triples (Jin et al., 2020). GSum (Dou et al., 2021) proposed a general and scalable guided summarization framework based on previous work, which can accept various signals like highlighted sentences to generate summaries, keywords, and relational triples as Guiding Knowledge to guide summary generation.
Structured Summarization. Two types of structure have been utilized in the literature: document structure and summary structure. Most existing works explored common document structures, such as source syntactic structures (Song et al., 2018) and document organizations (Li et al., 2018b). Recent studies have attempted to leverage summary structure, including: (1) methods that align the source document’s structure with that of the summary (Frermann and Klementiev, 2019; Balachandran et al., 2021; Cao and Wang, 2022); and (2) methods that leverage human predefined summary structure labels (Wang et al., 2023). However, these methods either rely on strong assumptions about the source document structure or heavily depend on human-annotated summary structures, limiting their applicability to diverse real-world documents.

heading: 6 Conclusion
text: In this paper, we analyze summaries of lengthy documents and observe that their summaries consistently adhere to a distinct structure. To harness this structural information, we introduce two methods, namely SKGSum and SKGSum-W. Our methods align their generated summaries closely with automatically extracted summary points. Experiments on three real-world open-source datasets have demonstrated the superiority of our method over state-of-the-art methods in summarizing news articles, legal judgments, and government reports.
Limitations
SKGSum achieves high performance by utilizing summary structure guiding knowledge, but this introduces additional work, such as using GPT to analyze the structure. Furthermore, although SKGSum-W shows promising capability in generating high-quality summaries without specific part labels from the gold summary, its performance in some scenarios still falls short when compared to SKGSum. These two aspects present research opportunities that we aim to explore in future work.

heading: Acknowledgment
text: This research is supported by the Marsden Fund Council from Government funding (MFPUOA2123), administered by the Royal Society of New Zealand.

heading: A Loss Functions
text: Extractive summarizers select sentences from the source document and utilize classification loss:
L = − ∑
(Di,Si)∈Dtrain
p∑
k=1
m∑
j=1
λ log p(skij |gkij)
+ ( 1− skij ) log p ( skij |gkij ) ,
(6)
where gkij is the decoder output and probability for selecting the j-th sentence of document Di for the k-th part, and skij is the binary ground-truth label of whether the sentence should be in the gold summary of Di. If the j-th sentence is selected then skij = 1, otherwise, s k ij = 0. The parameter λ is a weight to balance the importance of selected and unselected sentences in the loss function. Choosing a larger value of λ can mitigate the issue of imbalanced classes by boosting the importance of selected sentences. When λ = 1, the selected and unselected sentences are of the same importance (Liu and Lapata, 2019a).
As for abstractive summarizers, the loss is
L = − ∑
(Di,Si)∈Dtrain
p∑
k=1
m∑
j=1
skij log p(s k ij |gkij),
(7) where skij and g k ij are the j-th word token in the gold summary and generated summary of the k-th summary part for document Di.

heading: B Evaluation of GPT Capturing Summary Structure
text: GPT models are utilized to capture the structural essence of summaries, a task traditionally managed through human design, in an effort to maintain robustness. Therefore, we introduce additional tests to evaluate GPT’s capabilities in this regard.
Evaluation Formulation. Given summary S, and the reference summary structure comprising several parts, s1, s2, ·, sp, and ChatGPT-4 concluded summary structure, ŝ1, ŝ2, ·, ŝk. The challenge lies in determining the alignment between GPT-4’s generated summary structure, ŝj , and aligning the reference summary structure si.
This issue can be simplified into a multi-label classification problem. Assuming we have P possible summary points, where P = max(p, k). The objective is to assess the accuracy with ChatGPT-4 giving the same part labels.
Evaluation Metrics. To evaluate the performance of GPT-4 in accurately classifying summary points, we employ two well-known multi-label classification metrics: the F1 Score and Hamming Loss. The F1 Score ranges from 0 to 1, where 1 indicates perfect result, and 0 indicates the worst performance. In contrast, a lower Hamming Loss signifies better model performance, with 0 being the ideal score.
Dataset. We continue utilizing the previously employed datasets for evaluating the generated summaries’ quality, including CNN/DM, GAO, and NZJD. The GAO and NZJD datasets inherently contain structured summaries. For the CNN/DM dataset, we engaged an expert to delineate the summary points conclusively. Table 2 presents the summary structures for all three datasets.
Subsequently, we randomly constructed 50 test samples for each dataset, with each sample comprising 20 summaries. To identify the summary structure within these samples, we employed ChatGPT-4 using the same prompt as described in Section 4.1: "Find a common and simple text structure in these examples".
Results. After comparison between GPT observed summary structures with reference structure labels, we obtain 83% F1 scores and 0.26 Hamming Loss in CNN/DM datasets; 80% F1 scores and 0.31 Hamming Loss for Government reports
test; and 79% F1 scores and 0.32 Hamming Loss for New Zealand Judgment comparisons.
Result Analysis. The results show that ChatGPT4 can achieve approximately 80% F1 scores and a 0.3 Hamming Loss across all three datasets. This indicates that ChatGPT-4 has the capability to capture the structure of summaries. The majority of incorrect cases are caused by ChatGPT-4 providing excessive details. For example, ChatGPT-4 consistently splits "News Details," a summary point from CNN/DM, into two parts: an introduction and a description; it divides the "Fact" of NZJD into three parts, including background, legal claims, and previous decisions. Although these divisions are not incorrect, they are not suitable for all cases. For instance, some news summaries are too brief to require a two-part description of the news story.
Further, we want to highlight that while GPT is adept at mining structures, this does not necessarily equate to its ability to generate high-quality summaries. A key factor in this distinction is the length
of the input texts used for structure mining, which is significantly smaller than the length of the source documents.

heading: C Case Study Analysis
text: We conducted another case study on the New Zealand Judgment7.
Table 7 shows the generated summaries by existing state-of-the-art (SOTA) summarizers, large language model, GPT-4, and our proposed method.
In terms of legal judgments, compared with human-written summaries, existing state-of-theart (SOTA) summarizers cannot properly cover three essential categories consisting of "Decision", "Facts", and "Reasons". Specifically, extractive and abstractive summarizers lose information on "Decisions" and are less focused on "Reasons", while the guiding knowledge method (Dou et al., 2021) loses details of "Reasons" and makes mistakes in
7The case is shown on the web: http://www.nzlii.org/ cgi-bin/sinodisp/nz/cases/NZSC/2015/9.html
"Decisions". A similar finding can be found in the GPT-4 generated summary which misses the "Reasons" part. Because these models, especially guiding knowledge summarizers, focus solely on the most frequent information. Furthermore, current extractive summarizers tend to interleave different types of information, which reduces the generated summaries’ readability. Besides, summary sections can be relevant, and a sentence can be relevant to multiple summary parts. The current models do not capture the relevance across summary sections.
The bottom of Table 7 displays summaries generated by employing two backbone summarizers, ExtSum-LG (Gu et al., 2022) and BART (Lewis et al., 2020), integrated with SKGSum.
In comparison to results from prior models, the SKGSum adaptation of ExtSum-LG incorporates Reasons, rendering the generated summary more comprehensive. Additionally, summaries generated by the standalone ExtSum-LG are disordered and perplexing, making it challenging for readers to comprehend the central points of the summary. The integration of the SKGSum methodology significantly refines the abstractive summarizer, yielding a summary that encompasses all anticipated summary points and, notably, compared to summaries generated by BART, conveys more crucial information in the SKGSum version. These enhancements underscore the efficacy of the SKGSum methodology in refining existing summarizers.
Moreover, when compared to the knowledgeguided summarizer, GSum, SKGSum produces summaries that are organized, comprehensive, and accurate, indicating that the knowledge employed
in our method is efficacious for judgment summaries.
Lastly, when compared with the large language model-based summarizer, GPT-4, SKGSum yields summaries that are well-structured, complete, and balanced. It shows that the specialized design of structured summaries is essential, as the summary quality is better than that of the generalized language model for generating judgment summaries.

heading: D GPT-series Comparison Detail
text: We conduct comparisons with GPT-series models. We select two versions of GPT-3.5 models 8, text-davinci-003 and GPT-3.5-turbo, and GPT-4. Gpt-3.5-turbo is able to handle communication, and text-davinci-003 is focused on general NLP tasks.
We randomly chose 50 test cases per dataset. This means we will ask the GPT to generate summaries for 150 documents belonging to three different types of source documents (News, Judgments, and government reports).
Then, we set up two tests. First, we utilize GPT to generate the summary directly, in a way similar to what most previous summarizers did: without summary structure. We follow the GPT Official Document, which provides instructions and examples to utilize Tl;dr as prompts. Second, we utilize the summary point designed in Section 4.1 as prompts to generate structured summaries.
Table 8 shows the results. There are a few findings:
8https://platform.openai.com/docs/models/ gpt-3-5
• The proposed idea of structure summarization is effective. Our structure summarization approach improves the performance of GPT models by generating summaries part by part. When utilizing the designed summary points, we see a notable enhancement in the overall results. This improvement also can be found in Tables 3, and 4.
• Another key observation is that our proposed method SKGSum-W, which doesn’t necessitate part summaries, still outperforms both the GPT and its structured counterpart.
Furthermore, the cost is another factor in applying models in the real world. In total, the comparison costs US$ 76.45 to generate summaries for the 200 cases. If we had tested all the test sets, the total costs would have been more than US$ 700. This suggests that using GPT to do the summarization task can be a high-cost but low-performance choice.
In summary, this comparison shows that both proposed structure summarization methods are effective and essential, whether we use large language models to sum up long documents or not.

heading: Future Work
text: These two aspects present research opportunities that we aim to explore in future work