Venue: NAACL 2025

# Not all hallucinations are Good to Throw Away When it Comes to Legal Abstractive Summarization

 Nihed Bendahman \({}^{\diamond}\)  , Karen Pinel-Sauvagnat \({}^{\diamond}\),

Gilles Hubert \({}^{\diamond}\), Mokhtar Boumedyen Billami \({}^{\spadesuit}\)

\({}^{\diamond}\)Universite de Toulouse, IRIT, UMR 5505 CNRS, Toulouse, France

\({}^{\spadesuit}\)Berger-Levrault, 64 Rue Jean Rostand, 31670 Labge, France

{nihed.bendahman, karen.sauvagnat, gilles.hubert}@irit.fr

{nihed.bendahman, mb.billamij@berger-levrault.com

###### Abstract

Automatic summarization of legal documents requires a thorough understanding of their specificities, mainly with respect to the vocabulary used by legal experts. Indeed, the latter rely heavily on their external knowledge when writing summaries, in order to contextualize the main entities of the source document. This leads to reference summaries containing many abstractions, that data models struggle to generate. In this paper, we propose an entity-driven approach aiming at learning the model to generate factual hallucinations, as close as possible to the abstractions of the reference summaries. We evaluated our approach on two different datasets, with legal documents in English and French. Results show that our approach allows to reduce non-factual hallucinations and maximize both summary coverage and factual hallucinations at entity-level. Moreover, the overall quality of summaries is also improved, showing that guiding summarization with entities is a valuable solution for legal documents summarization.

## 1 Introduction

The extensive use of Large Language Models (LLMs) for summarization tasks brings new challenges. One of the most important is related to the LLMs' tendency to generate _hallucinations_, i.e., texts that give the impression of being fluid and natural, despite their lack of fidelity or nonsensical nature Ji et al. (2023). Hallucinations can be non-factual or factual. In this last case, a source of knowledge, such as an expert, an ontology or a knowledge base, can indeed certify their veracity Maynez et al. (2020); Feng et al. (2023).

Most of the current works on hallucinations aim at deleting, reducing or correcting non-factual hallucinations through dedicated models Narayan et al. (2021); Nan et al. (2021); Zhang et al. (2022) or post-processing approaches Cao et al. (2020); Zhao et al. (2020); Chen et al. (2021). Some other works try to control factual hallucinations through the use of external knowledge, while keeping low the number of non-factual hallucinations Cao et al. (2022); Dong et al. (2022); Chern et al. (2023).

This last line of research is particularly of interest while considering specific domains such as the legal field. Indeed, summaries of legal documents written by experts often contain domain-specific knowledge and references to previous laws that help the readers to fully grasp the content. This is illustrated in Figure 1 through an example of legal document with its associated reference summary, extracted from the EUR-Lex-Sum dataset1. It can be observed that both the document and the reference summary include a large number of entities of interest (in bold). Moreover, the reference summary mentions laws _Regulation 2019/979, Regulation 2020/1273_, and _Regulation 2020/1272_ (colored in dark and light blue on Figure 1) that do not appear in the document. Such entities have been added to the summary by experts in order to track all the amendments made to the _Regulation 2019/980_ presented in the document. They are named _abstractive_ entities, in opposition to faithful entities, i.e., entities present in the source document Dong et al. (2022) (in green in the reference summary). Abstractive entities, as they are written by experts, are de facto facto factual.

Footnote 1: [https://www-lex.europa.eu/eli/reg_del/2019/980/oj](https://www-lex.europa.eu/eli/reg_del/2019/980/oj)

In this paper, we focus on abstractive summarization of legal documents, with the main objective to produce enriched, useful, and factual summaries which will lighten the task of information monitoring by legal experts. To do so, we aim at controlling factual hallucinations by knowledge arisen from the source dataset. In other words, we propose an approach which aims at fulfilling a two-objectives task: (1) retrieve factual hallucinations from the dataset; and (2) keep very low the number of non-factu

Our work focuses on hallucinations and abstractions at entity level for two main reasons: (i) entities are more striking to human readers Ji et al. (2023), and (ii) the legal domain is overflowing with entities of interest (law references, institutions, etc.). First, we retrieve entities from the source dataset that closely match the set of abstractive entities of the reference summary. Then, we introduce various approaches for injecting the retrieved entities during the training of the model, in order to control the generated entities (faithful and hallucinated ones) at inference. To the best of our knowledge, this is the first approach aiming at controlling factual hallucinations using only the original dataset as a source of knowledge.

To evaluate our approach, we conducted various experiments on two different legal corpora in English and French. We aimed at answering the following questions:

**(Q1)** What are the best strategies to identify abstractive entities in the source dataset ?

**(Q2)** What are the best training scenarios to take advantage of the entities identified in Q1?

**(Q3)** Does guiding the model by entities improve the overall quality of the generated summary?

## 2 Context and Problem Formulation

### Hallucinations and Abstractions

As there is no formal consensus on the vocabulary used by state-of-the-art approaches about the concepts of hallucinations and abstractions in a summarization context, we detail in this section the definitions considered in this article.

In literature, hallucinations and abstractions are usually considered at two different levels: the entity one and the relationship/information one Lyu et al. (2022); Maynez et al. (2020); Ji et al. (2023). Since the approach proposed in this work is driven by entities of interest, we consider definitions at entity level in the following.

Figure 2 proposes a taxonomy of these definitions, while the colors highlighting entities on Figure 1 illustrate the taxonomy. _Faithful entities_ (in green) are entities in the reference or generated summary that can be found in the source document. _Abstractions_ may occur in the reference summary while _hallucinations_ are specific to the generated summary. Abstractions are produced by experts, who own an omniscient knowledge, when summarizing a document, incorporating factual entities in the (reference) summary they write that are not present in the document. We classify abstractions into two categories: those that can be found in the dataset the source document belongs to (_in-dataset_ abstractions, in dark blue) and those that come from external world knowledge (_out-dataset_ abstractions, in light blue), which we cannot know. Hallucinations can be of two types: non ascertainable from the source document but factual with respect to the domain knowledge (in yellow) and non factual (in red). Domain knowledge can originate from an ontology, a knowledge base or the expert's own prior knowledge.

Our approach aims at controlling factual halluci

Figure 1: Example of a EUR-Lex-Sum legal document with its reference summary (excerpts) produced by experts, as well as an example of generated summary. Entities of interest are marked in bold. In the reference summary, faithful entities are colored in green. Abstractive entities of the reference summary are colored in dark blue when present in the dataset (in-dataset abstractions) whereas the light blue one is an out-dataset abstraction. In the generated summary, faithful entities are also colored in green. The entities colored in red are non-factual hallucinations whereas the yellow one is a factual hallucination.

 

nations to make them as close as possible to the abstractions contained in reference summaries. To ascertain the factuality of hallucinations we propose to exploit the dataset of the source documents as knowledge base through the construction of an entity graph. As _out-dataset_ abstractions cannot be found from this graph, we simplify the problem as follows: we aim at controlling factual hallucinations to make them as close as possible to the _in-dataset_ abstractions contained in reference summaries.

Other notions are introduced in the literature at the relationship/information level, such as extrinsic or extrinsic hallucinations Maynez et al. (2020); Ji et al. (2023), but they are out of the scope of this paper.

### Problem Formulation

Let us consider a dataset of the legal domain composed of \({\cal D}\) a set of source documents \(d_{i}\) and \({\cal R}\) a set of reference summaries \(r_{i}\) such as \(r_{i}\) is the reference summary associated to \(d_{i}\). Let \(E_{d_{i}}\) and \(E_{r_{i}}\) be respectively the set of entities in \(d_{i}\) and \(r_{i}\). \(E_{r_{i}}=F_{r_{i}}\cup A_{r_{i}}\) where \(F_{r_{i}}\) is the set of faithful entities in \(r_{i}\) (\(F_{r_{i}}=E_{r_{i}}\cap E_{d_{i}}\)) and \(A_{r_{i}}\) is the set of abstractive entities in \(r_{i}\). \(A_{r_{i}}=A^{in}_{r_{i}}\cup A^{out}_{r_{i}}\) where \(A^{in}_{r_{i}}\) is the set of abstractive entities that can be found in \({\cal D}\) and \(A^{out}_{r_{i}}\) the set of abstractive entities that cannot be found in \({\cal D}\).

Our aim is to produce for each \(d_{i}\) in \({\cal D}\) a generated summary \(g_{i}\) such as \(E_{g_{i}}\) (the set of entities in \(g_{i}\)) is as close as possible to \(E_{r_{i}}\). In a realistic world, we simplify the problem as: our aim is to produce for each \(d_{i}\) in \({\cal D}\) a generated summary \(g_{i}\) such as \(E_{g_{i}}\) is as close as possible to \(E_{r_{i}}\setminus A^{out}_{r_{i}}\). In other words, we aim at having: \(F_{g_{i}}\) = \(F_{r_{i}}\) and \(H^{f}_{g_{i}}\)=\(A^{in}_{r_{i}}\) and \(H^{nf}_{g_{i}}\) =\(\emptyset\), where \(H^{f}_{g_{i}}\) and \(H^{nf}_{g_{i}}\) are respectively the factual and non-factual hallucinations of \(g_{i}\).

Notations are summarized in Appendix B.

## 3 Related Work

**Hallucination Detection** Several works have shown that abstractive summarization models suffer from generation of hallucinated entities Cao et al. (2022); Nan et al. (2018); Kryscinski et al. (2019). Cao et al. (2018) showed that 30% of the named entities generated by the BART model are hallucinated in the XSum dataset Narayan et al. (2018). Conventional summary evaluation metrics such as ROUGE Lin (2004), BLEU Papineni et al. (2002), or BERScore Zhang et al. (2019) consider lexical and semantic overlap but fail to assess factuality and faithfulness since they are not sensitive to the hallucinated content. Thus, several approaches have been developed to measure the hallucination rate in the text generated by the model. They rely on question-answering methods Wang et al. (2020); Durmus et al. (2020); Scialom et al. (2021a); Fabbri et al. (2022), text-entailment Falke et al. (2019); Goyal and Durrett (2020); Krycsinski et al. (2020), LLMs Manakul et al. (2023); Friel and Sanyal (2023); Liu et al. (2023); Mundler et al. (2023), or Information Extraction Nan et al. (2021); Goodrich et al. (2019); Dhingra et al. (2019). Given that legal texts contain many references to laws, which can be considered as entities of interest, we use the latter to evaluate generated summaries, personalizing the metrics presented by Nan et al. (2021).

**Entity-Centric Summarization** Several approaches focused on directing the model to generate named entities at the intersection of the source

Figure 2: Taxonomy of entities in the reference summary (left) and generated summary (right). Examples of entities from each set are presented on Figure 1, using the same colors (green for faithful entities, blue for abstractions, yellow and red respectively for factual and non factual hallucinations).

 

document and the reference summary. Zheng et al. (2020) extract key phrases at the intersection of the document and the reference summary and train LSTM models with a concatenation of the document and the extracted phrases. Liu and Chen (2021); Mao et al. (2020) select named entities instead. Fan et al. (2018); He et al. (2022) propose generated summaries controlled by several aspects, such as the summary subject, keywords or named entities, by concatenating them with the document or by adding them as a prompt in order to guide the model. Xiao and Carenini (2023) on the other hand, use a Global Relevance module to select important entities from the source document and introduce them to the decoder. Zhang et al. (2022) proceed differently, by calculating the coverage rate of entities Nan et al. (2021b) and introducing a control token based on this rate to guide the generation. In this paper, with the objective of getting as close as possible to the writing style of the legal domain, we investigate the possibility of fine-tuning the model to introduce entities that do not exist in the source document. We concatenate with the source document a set of entities from the reference summary or entities from the dataset related to the document.

**Legal Domain** The legal domain presents unique challenges due to its complex and highly specialized language, which explains the multitude of dedicated approaches in the literature Sansone and Sperfi (2022). Among neural network-based approaches, Chalkidis et al. (2020) proposed LEGALBERT. Regarding summarization, models do not only need to summarize content accurately but also to maintain the integrity of legal entities and terminology. Legal-Pegasus2 is pretrained on legal documents, whereas Legal-LED3 is specifically fine-tuned on legal data. At the same time, several datasets in legal domain have been created to fine-tune language models for automatic summarization. Among them, one can cite BillSum Kornilova and Eidelman (2019), a dataset for summarization of US Congressional and California state bills; EULex-Sum Aumiller et al. (2022), a dataset based on manually curated document summaries of legal acts from the European Union law platform (EURLex); and Multi-LexSum Shen et al. (2022), a collection of expert-authored summaries drawn from ongoing CRLC (_Civil Rights Litigation Clearinghouse_) writing. This diversity in corpus types highlights the potential adaptability of summarization models to various legal contexts, ranging from legislative documents to civil rights summaries. Most of the proposed datasets and fine-tuned models are extractive and available in English, with very few existing in French. The approach we propose in this paper is abstractive, and evaluated on two different languages, French and English.

Footnote 2: [https://nuggingface.co/nsi119/legal-pegasus](https://nuggingface.co/nsi119/legal-pegasus)

Footnote 3: [https://huggingface.co/nsi119/legal-led-base-16384](https://huggingface.co/nsi119/legal-led-base-16384)

## 4 Legal Summary Specificities

Our aim here is to study the characteristics of reference summaries in terms of entities of interest as well as abstractions. To this end, we analyzed two legal datasets for summarization: EUR-Lex-Sum and a private corpus of French legal news.

### Datasets Description

The EUR-Lex-Sum dataset Aumiller et al. (2022) contains 1,504 legal acts documents of 12,000 tokens in average, retrieved using web scraping from the European Union law platform website4. Associated reference summaries are composed of 800 tokens in average. The dataset is divided into training and test sets, containing 1,316 and 188 documents respectively.

Footnote 4: [https://www-legibase.fr/](https://www-legibase.fr/) An example document and its associated reference summary are presented in Appendix C.

We also used a private business dataset, named Legibase composed of 8,485 legal and regulatory documents of French local authorities and public administrations5. Each document includes (a) a title, (b) a text (body/content), (c) a set of associated metadata, notably the document topic, and (d) a manually-produced summary. Each information is written by experts in the field, whose aim is to keep the legal news up to date. These experts focus on what is new and evolving in French law, often starting with a presentation of the background of the law, the code and its various clauses, followed by a presentation of the evolution. Each summary sets out the general context of the document content and the news described within it. Documents and reference summaries are respectively composed of 8,325 and 1,110 tokens in average. The train and test sets are composed of 9,353 and 851 documents.

Footnote 5: [https://www.legibase.fr/](https://www.legibase.fr/) An example document and its associated reference summary are presented in Appendix C.

### Entities of Interest

We considered 4 types of entities: Organizations, Locations, Persons and Law entities (e.g., Regulation 2019/279, Decision 1147/366, Directive 2016/879 in English, or loi n\({}^{\circ}\) 93-22, article 57 du 

Code civil, decret n\({}^{\circ}\) 89-11 in French). As Law entities are written with very specific and repetitive patterns, we used a set of regex to extract them from the text. All other entities were extracted using a NER model fine-tuned on the Indian Court Judgements dataset6.

Footnote 5: [https://huggingface.co/openmyaiorg/en_legal_ner_trf](https://huggingface.co/openmyaiorg/en_legal_ner_trf)

Figure 3 presents the distribution of the extracted entities across the datasets. Law entities are, not surprisingly, by far the most present, both in source documents and in reference summaries. Another interesting result is that Persons are barely present in the reference summaries of EUR-Lex-Sum.

### Abtractiveness of Reference Summaries

Table 1 shows an analysis of faithful entities (\(F_{r_{i}}\)), as well as in-dataset and out-dataset abstractive entities (\(A_{r_{i}}^{in}\) and \(A_{r_{i}}^{out}\)). We observe that a large proportion of the entities present in the summaries are abstractive (66% and 68% for EUR-Lex-Sum and Legibase respectively). This confirms that legal experts rely heavily on their external knowledge when writing summaries, in order to contextualize the main entities of the source document and provide a history of them. This external knowledge may or may not be found in other documents in the dataset, as we can see from columns \(A_{r_{i}}^{in}\) and \(A_{r_{i}}^{out}\).

## 5 Selection of abstractive entities in the source dataset (Q1)

Our objective is to generate a summary as similar as possible to the reference summary, which therefore contains as many entities as possible from the set of abstractive entities. We propose a method based on an entity graph constructed from the dataset, to identify for each document, the set of entities that most closely matches the abstractive entities of the corresponding reference summary.

### Graph Construction

We constructed two different graphs: \(G_{D_{T}}\) and \(G_{D_{I}}\), containing respectively the sets of entities of our training and testing datasets. \(G_{D_{T}}\) (see Figure 4 for an example) is constructed as follows: (i) the nodes set is composed of documents \(d_{i}\in D_{T}\), entities \(E_{d_{i}}\) of each \(d_{i}\) and the subject of \(d_{i}\)7; (ii) edges represent 1) the similarity between documents, evaluated with a cosine similarity between embeddings of documents8, 2) the presence of an entity in a document, and 3) a link between a document and a subject. The construction of \(G_{D_{I}}\) is similar.

Footnote 7: The subjects of EUR-LEX-Sum documents are obtained by scrapping the associated web pages, while those of Legibase are given with the documents.

Footnote 8: [https://huggingface.co/facebook/bart-large-mlli](https://huggingface.co/facebook/bart-large-mlli)

### Entity Selection Strategies

We defined different strategies \(S_{i,j}\) to select entities from the created graphs. The idea is, for each document \(d_{i}\), to retrieve entities from related documents: for \(S_{i,1}\) we extract entities connected by a maximum of 1, 2, or 3 hops to \(d_{i}\); for \(S_{i,2}\) we extract entities of documents sharing the same subject than \(d_{i}\); for \(S_{i,3}\) we extract entities of documents semantically similar to \(d_{i}\); and finally for \(S_{i,4}\) we extract entities of documents sharing the same subject than \(d_{i}\) and being semantically similar to \(d_{i}\). Note that entities of \(d_{i}\) are systematically excluded from the resulting sets.

### Experiments and Results

In order to determine which selection strategy would be most effective, we measured the inter

Figure 4: Extract of \(G_{D_{T}}\) for the EUR-LEx-Sum dataset. Purple nodes are documents, red ones are Subjects, while orange, brown and green ones are respectively Law, Organization and Person entities. The two documents are semantically close and share the same subject ‘Internal Market’. They also share the same entity named ‘European Economic Committee’.

Figure 3: Distribution of entity types across source documents and reference summaries in the two datasets: EUR-Lex-Sum and Légbase. Distinct occurrences are considered.

 

section of their respective sets with the set of abstractions in the reference summaries. Results are shown in Table 2. None of the strategies strongly maximize the intersection with abstractions. The intersection of \(S_{i,4}\) is quite low, particularly for the Organizations and Locations, leading us to conclude that abstractive entities are widely dispersed in the corpus, and not necessarily linked to \(d_{i}\). We chose \(S_{i,1}\) as the best compromise for the rest of our experiments.

## 6 Entity-driven summarization (Q2)

Given the set of entities retrieved from the graph, we experimented various fine-tuning settings, by varying the input provided to the model. The quality of the generated summary was then evaluated using entity-oriented metrics.

### Experimental Setup

ModelsAs documents in the EUR-Lex-Sum dataset are very long (12k tokens on average), we evaluated our approach with a classic Long Encoder-Decoder model (LED)9 as well as with BART Lewis et al. (2020)10 using a Long Document Transformer framework Beltagy et al. (2020) with 4 epochs. Checkpoints were retrieved from the HuggingFace platform. Both models led to similar conclusions in terms of behaviour. We decided to report only LED results in the paper for space limitation reasons. BART results can be found in Appendix D as supplementary material11.

Footnote 8: [https://huggingface.co/allensi/led-base-16384](https://huggingface.co/allensi/led-base-16384)

Fine-Tuning FormatWe prepended the set of entities to the source document, using the special token [ENTITYSET] to introduce the entities and separated them with the token l. The source document was introduced using the special token [DOCUMET]. For EUR-Lex-Sum, we considered the 3 entity types: Law, Organization and Location. Person entities were removed since they are not present in the reference summaries (see Figure 3). For Legibase, the 4 entities were used.

**Fine-Tuning Settings** We considered 3 different settings to fine-tune the model (settings notations are summarized in Table 3), with the objective to evaluate how the entities provided during fine-tuning affect the model's performance during inference, and more broadly, to assess whether the model's behavior varies based on the set of entities used.

**- Setting 1:** During the training phase, we provide the model with the source document \(d_{i}\) and the \(A^{in}_{r_{i}}\) set. During inference, we concatenate the source document and 20 entities of \(E_{s_{i,1}}\), retrieved with \(S_{i,1}\). These entities are chosen randomly, maintaining a balance between the different entity types (law, org, loc, and per).

**- Setting 2:** During training, the model receives the source document, prepended by 20 entities from \(A^{in}_{ri}\) completed by some randomly chosen entities of \(E_{S_{i,1}}\). For inference, only the source document and 20 entities of \(E_{S_{i,1}}\) are provided.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline  & \multicolumn{5}{c}{EUR-Lex-Sum} & \multicolumn{5}{c}{LÉgibase} \\ \cline{2-17}  & \multicolumn{3}{c}{Total} & \multicolumn{3}{c}{Avg} & \multicolumn{3}{c}{Total} & \multicolumn{3}{c}{Avg} \\ \cline{2-17} Entity Type & \#F\({}_{r_{i}}\) & \#Aff_{r_{i}}^{out}\) & \#Arv\({}_{r_{i}}\) & \#Arv\({}_{r_{i}}\) & \#Arv\({}_{r_{i}}^{out}\) & \#Arv\({}_{r_{i}}\) & \#Arv\({}_{r_{i}}^{out}\) & \#Arv\({}_{r_{i}}^{out}\) & \#Arv\({}_{r_{i}}^{out}\) & \#Arv\({}_{r_{i}}^{out}\) & \#Arv\({}_{r_{i}}^{out}\) \\ \hline Law & 1,978 & 1,108 & 1,906 & 3.72 & 2.37 & 2.37 & 667 & 964 & 486 & 1.10 & 1.12 & 1.06 \\ Org & 340 & 309 & 457 & 1.67 & 1.71 & 1.32 & 222 & 266 & 140 & 1.04 & 1.05 & 1.01 \\ Loc & 233 & 247 & 95 & 1.68 & 2.14 & 1.06 & 351 & 245 & 89 & 1.12 & 1.11 & 1.01 \\ Per & 1 & 0 & 26 & 0 & 0 & 0 & 253 & 621 & 368 & 1.04 & 1.13 & 1.05 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Total number of faithful and abstractive entities in the reference summaries of EUR-Lex-Sum and Légbase. The average number of entities per reference summary is also reported.

\begin{table}
\begin{tabular}{l l 



**- Setting 3:** During training, the model receives only 20 entities of the \(E_{S_{i,1}}\) set and the source document. For inference, the same elements are provided.

The size of the entities set (20 entities) added to each document was chosen on the basis of preliminary experiments on EUR-lex-Sum.

### Entity-Level Metrics

To evaluate the generated summary \(g_{i}\) in terms of faithful and hallucinated entities according to the source document \(d_{i}\) and reference summary \(r_{i}\), we extended the metrics of Nan et al. (2021b):

- Coverage rate (Recall, Precision) in relation to the source document:

\[R(C)=\frac{\#(E_{g_{i}}\cap E_{d_{i}})}{\#E_{d_{i}}},~{}~{}P(C)=\frac{\#(E_{ g_{i}}\cap E_{d_{i}})}{\#E_{g_{i}}}\]

- Faithfulness rate:

\[R(F)=\frac{\#(E_{g_{i}}\cap F_{r_{i}})}{\#F_{r_{i}}},~{}~{}P(F)=\frac{\#(E_{g_{ i}}\cap F_{r_{i}})}{\#E_{g_{i}}}\]

- Factual hallucination rate (in relation to the abstractive entities of the reference summary):

\[R(FH)=\frac{\#(E_{g_{i}}\cap Ai^{n}_{r_{i}})}{\#A^{n}_{r_{i}}},~{}~{}P(FH)= \frac{\#(E_{g_{i}}\cap Ai^{nn}_{r_{i}})}{\#E_{g_{i}}}\]

- Non-factual hallucination rate:

\[P(\overline{FB})=\frac{\#(E_{g_{i}}-E_{r_{i}})}{\#E_{g_{i}}}\]

### Baselines

We used the following state-of-the-art baselines for comparison with our approach: (1) **ECC**Zhang et al. (2022) is a model training approach for improving the factuality of generated summaries based on the coverage metric; (2) **CTRLSum**He et al. (2022) is a training approach to guide the model to include a set of pre-selected keywords and entities in the generated summary; and (3) **Mixstral 8 * 7b**Jiang et al. (2024) is a LLM, chosen to compare the experiments conducted with PLMs against a LLM without fine-tuning. Additionally, we also report (i) **Setting 0**, in which we run a basic fine-tuning of the model, where only the source document is provided as input, for both training and inference. (ii) **Oracle FH**, in which the set of abstractive entities \(A^{in}_{r_{i}}\) is given at training and inference time. Resulting summaries can be considered as Oracle summaries, i.e., summaries maximizing the R(FH) and P(FH) metrics. They are the best possible summaries that can be obtained by our approach considering factual hallucinations.

### Results

Results are presented in Table 4. The high results obtained by Oracle FH summaries show that guiding the summary generation with entities allows to obtain better summaries in terms of faithfulness, factual and non-factual hallucinations. This is also true whatever the considered setting (1, 2 or 3), since all settings are better than setting 0, in which no entities are included. Another conclusion is that the quality of entities given as input of the model strongly impacts its performances (comparison of settings 1, 2 and 3). Giving only the set of abstractive entities to the model during the training phase (setting 1) does not allow to reach the best performances. Indeed, if some noise is added during training (settings 2 and 3), and consequently if the training is closer to the inference configuration, results are improved. Regarding SOTA baselines, we can see the ECC and CTRLSum models outperform Mixtral, probably because they are entity-oriented, unlike the latter. ECC is the most competitive baseline, with particularly good results regarding faithfulness and coverage (F and C metrics). This is not surprising since the approach is optimized on the F metric. Finally, we observe

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{EUR-Lex-Sum} & \multicolumn{6}{c}{Legibase} \\ \cline{2-15} Systems & \multicolumn{2}{c}{F \({}^{\dagger}\)} & \multicolumn{2}{c}{C \({}^{\dagger}\)} & \multicolumn{2}{c}{FH \({}^{\dagger}\)} & \multicolumn{2}{c}{FH \({}^{\ddagger}\)} & \multicolumn{2}{c}{F \({}^{\dagger}\)} & \multicolumn{2}{c}{C \({}^{\dagger}\)} & \multicolumn{2}{c}{FH \({}^{\dagger}\)} & \multicolumn{2}{c}{FH \({}^{\ddagger}\)} \\ \cline{2-15}  & R & P & R & P & R & P & R & P & R & P & R & P & R & P & R & P \\ \hline ECC & **0.33** & **0.21** & 0.09 & 0.34 & 0.20 & 0.07 & - & 0.70 & 0.47 & **0.06** & **0.30** & 0.16 & **0.11** & **0.09** & - & 0.82 \\ CTRLSum & 0.29 & 0.19 & 0.09 & 0.31 & 0.18 & 0.06 & - & 0.71 & 0.44 & 0.05 & 0.27 & 0.18 & 0.10 & 0.07 & - & 0.85 \\ Mixtral & 0.25 & 0.12 & 0.04 & 0.19 & 0.10 & 0.02 & - & 0.76 & 0.36 & 0.02 & 0.20 & 0.11 & 0.04 & 0.00 & - & 0.97 \\ _Oracle FH_ & _0.33_ & _0.21_ & _0.12_ & _0.35_ & _0.38_ & _0.11_ & _- & _0.60_ & _0.59_ & _0.05_ & _0.30_ & _0.44_ & _0.50_ & _0.07_ & _- & _0.80_ \\ Setting 0 & 0.29 & 0.17 & 0.05 & 0.24 & 0.19 & 0.04 & - & 0.73 & 0.53 & 0.04 & 0.25 & 0.12 & 0.09 & 0.04 & - & 0.94 \\ \hline Setting 1 & 0.30 & 0.15 & 0.08 & 0.25 & 0.23 & 0.05 & - & 0.78 & 0.54 & 0.02 & 0.22 & 0.17 & 0.10 & 0.07 & - & 0.93 \\ Setting 2 & **0.33** & 0.20 & **0.10** & 0.34 & **0.30** & **0.08** & - & **0.68** & 0.63 & 0.05 & 0.28 & **0.36** & 0.10 & 0.01 & - & **0.81** \\ Setting 3 & 0.32 & 0.20 & **0.10** & **0.35** & 0.29 & **0.08** & - & **0.68** & **0.64** & 0.05 & 0.29 & **0.36** & 0.09 & 0.01 & - & **0.81** \\ \hline \end{tabular}
\end{table}
Table 4: Results on EUR-Lex-Sum and Légbase, using Entity-Level metrics : Coverage (C), Faithfulness (F), Factual hallucination (FH) and Non-Factual hallucination rate (\(\overline{\text{FH}}\)). Results in bold and underlined are respectively the best and second best results for a given metric.



similar performance gains on both datasets, despite their language differences. One notable exception is the FH metric on Legibase, which decreases with settings 2 and 3 (compared to setting 1). Possible explanations include either a difference in behavior for French language, or that the number of entities provided as input no longer allows the model to hallucinate factually. This will be the subject of future experiments on the French language.

## 7 Evaluating the Overall Quality of the Generated Summaries (Q3)

The experiments presented in previous sections showed that injecting entities into the model allows us to improve entity-centric metrics. In this section, we aim at assessing the overall quality of the generated summaries.

### Contextual Metrics

To evaluate the quality of summaries at token-level, we report the well-known **ROUGE-L**[54] and **BLEU**[60] metrics. We also use 3 contextual metrics: **Bert-Score**[89] which compares contextual embeddings of sentences, **FactCC**[10] which assesses the factual consistency of the generated summaries using textual entailment, and **QuestEval**[11], a question generation and answering metric. As reference summaries are slightly different from source documents in terms of content, all metrics are evaluated both against source documents and reference summaries.

### Discussion and results

Results are reported in Table 5. ECC remains the strongest baseline, but is outperformed by our approach on all contextual metrics. This shows that our approach, although entity-oriented, is able to provide high-quality summaries. Whether the generated summaries are compared with the documents or with the reference summaries, we observed that settings 2 and 3 outperform all other systems. This is also true compared to Oracle FH. Adding noise to the training data thus allows the model to better generalize. This confirms our conclusions from Table 4. A last conclusion concerns the metrics evaluated wrt the source document. Reference summaries, when evaluated wrt the source document obtain weak results (first line of Table 5). Metrics evaluated against source documents are thus not relevant to compute on our datasets, since reference summaries of the latter are quite different from the source documents.

All these results motivate us to further explore the approach of guiding the summary generation model with additional entity-oriented information.

## 8 Case Study

In this section, we present some examples of generated summaries using baselines and our model, in order to provide a qualitative analysis. Those examples are showed in Table 5 for EUR-Lex-Sum. Examples of Legibase are present in Appendix C.

These examples illustrate that all approaches generate factual entities (in green), but also hallucinations (in red). The latter are often due to errors in Regulations or Laws numerical identification code. For instance, Regulation 2017/1939 is generated instead of Regulation 2017/1938 (Figure 5-Our generated summary). The same can be observed with Regulation 525/2013 instead of 347/2013 (ECC Generated Summary). Another type of hallucinations is also present at the level of entity headings: Decision (EU) No 994/2010 is prompted in Mistral Generated Summary instead of Regulation (EU) No 994/2010. Moreover, although

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{Wrt the Source Document} & \multicolumn{4}{c}{Wrt the Reference summary} \\ \cline{2-13}  & \multicolumn{3}{c}{EUR-Lex-Sum} & \multicolumn{3}{c}{LEgibase} & \multicolumn{3}{c}{EUR-Lex-Sum} & \multicolumn{3}{c}{LEgibase} \\ \cline{2-13} Systems & Bert-S \({}^{\dagger}\) & Q-E \({}^{\ddagger}\) & F-CC \({}^{\ddagger}\) & Bert-S \({}^{\dagger}\) & F-CC \({}^{\dagger}\) & Bert-S \({}^{\dagger}\) & Q-E \({}^{\dagger}\) & F-CC \({}^{\dagger}\) & R-I \({}^{\dagger}\) & B \({}^{\dagger}\) & Bert-S \({}^{\dagger}\) & F-CC \({}^{\dagger}\) & R-I \({}^{\ddagger}\) & B \({}^{\dagger}\) \\ \hline Reference Summaries & 0.85 & 0.35 & 0.20 & 0.84 & 0.30 & – & – & – & – & – & – & – & – & – & – \\ ECC & 0.86 & 0.33 & 0.35 & 0.84 & 0.37 & 0.84 & **0.38** & 0.36 & 0.22 & 0.40 & 0.83 & 0.34 & 0.19 & 0.31 \\ CTRLSum & 0.83 & 0.36 & 0.31 & 0.83 & 0.35 & 0.85 & 0.33 & 0.34 & 0.21 & 0.40 & 0.84 & 0.34 & 0.18 & 0.31 \\ Mixtral & 0.83 & 0.29 & 0.22 & 0.82 & 0.82 & 0.84 & 0.30 & 0.29 & 0.18 & 0.37 & 0.83 & 0.27 & 0.14 & 0.26 \\ _Gracle FH_ & _0.86_ & _0.36_ & _0.33_ & _0.84_ & _0.29_ & _0.86_ & 0.49 & 0.39 & 0.26 & _0.44_ & _0.87_ & _0.39_ & _0.20_ & _0.36_ \\ Setting 0 & 0.83 & 0.27 & 0.24 & 0.84 & 0.29 & 0.83 & 0.28 & 0.27 & 0.20 & 0.39 & 0.83 & 0.28 & 0.16 & 0.29 \\ Setting 1 & 0.85 & 0.33 & 0.31 & 0.83 & 0.36 & 0.86 & 0.34 & 0.34 & 0.23 & 0.39 & 0.84 & 0.36 & 0.16 & 0.33 \\ Setting 2 & **0.88** & **0.37** & **0.39** & **0.86** & 0.38 & **0.87** & 0.35 & 0.39 & 0.21 & 0.41 & **0.86** & **0.40** & **0.20** & **0.35** \\ Setting 3 & 0.87 & 0.35 & **0.39** & 0.85 & **0.41** & 0.86 & 0.37 & **0.40** & **0.24** & **0.42** & **0.86** & 0.39 & 0.19 & **0.35** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance of the different systems (baselines, Oracle FH, settings) evaluated using metrics Bert-Score (reported in the table as Bert-S), QuestEval (Q-E), FactCC (F-CC), Rouge-L (R-L) and Bleu (B). QuestEval has not been reported for Légbase as it is not available in French.

 

ECC is our strongest baseline, it still generates similar hallucinations as the other models.

Finally, we can see that guiding the generation model to generate abstrative entities is successful on these examples. Our model is the only one to generate factual hallucinations (in yellow), which are themselves present as abstractive entities (in light blue) in the reference summaries. Moreover, focusing on entities does not affect the overall quality of the generated summary in terms of readibility. This supports the analysis conducted in Section 7.

## 9 Conclusion

In this paper, we showed the specificities of legal corpora in terms of abstraction in reference summaries produced by legal experts, which requires an adaptation of state-of-the-art summarization approaches. We introduced a new method for selecting factual hallucinations on an entity-level based on a graph created from the source dataset, and proposed different training and inference settings for guiding the summary generation model with these entities. Our experiments on both French and English datasets show that this method significantly reduces non-factual entity hallucinations, increases coverage metrics and factual hallucinations while improving the overall quality of the generated summaries. To the best of our knowledge this is the first entity-centric summarization method using only the original dataset as a way to control factual and non-factual hallucinations.

Figure 5: Example of a legal document from EUR-Lex-Sum with its reference summary (excerpts) produced by experts, as well as 3 generated summaries (Setting 2, ECC and Mistral). Entities are marked in bold. In the reference summary, faithful entities are colored in green. Abstractive entities of the reference summary are colored in dark blue when present in the dataset (in-dataset abstractions) whereas the light blue one is an out-dataset abstraction. In the generated summary, faithful entities are also colored in green. The entity colored in red is a non-factual hallucination whereas the yellow one is a factual hallucination.

 



## 10 Limitations

Although promising, our approach has some limitations. First, we were unable to provide a human evaluation of our approach. Since legal language is particularly complex, collaboration with legal experts is necessary. We chose to use Legibase to work directly with the experts who wrote the articles and summaries. For time and organizational reasons, we were not able to evaluate a significant number of generated summaries in order to present the results of this evaluation, but this is currently in progress.

A second limitation of our approach is that it focuses exclusively on entity-level hallucinations. Although we showed that the overall quality of our generated summaries is good, we could extend our work to relation-level hallucinations, using the same framework with facts (subject, predicate, object) to ensure that the introduced entities and terms are correctly placed Huang et al. (2020). Additionally, we aim to expand our hallucination metrics to distinguish between intrinsic and extrinsic hallucinations at relation/information level. Intrinsic hallucination can be defined as terms or concepts from the source information that are misrepresented in the summary, making them unfaithful to the original source. On the other hand, extrinsic ones can be defined as information that cannot be verified from the source content. Another future work is to extend our external knowledge to ontologies and knowledge graphs in order to retrieve more facts from multiple sources Wang et al. (2022); Dziri et al. (2021), and not only from the datasets used in this study.

A third limitation concerns the strategies used to find abstractive entities in the source dataset. The results with these heuristics are promising but showed that abstractive entities are disseminated in all the datasets. Graph learning strategies will probably be very useful Ribeiro et al. (2022).

Finally, another limitation is the non-use of current large language models (LLMs). We compared our approach with the Miktral 8 * 7b Jiang et al. (2024), but used it with a basic prompt, without fine-tuning. Therefore, the next natural step of this research work is to fine-tune LLMs such as LLaMA 2 Touvron et al. (2023) and Mixtral.

## 11 Acknowledgements

This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011014740 made by GENCI.

## References

* Aumiller et al. (2022) Dennis Aumiller, Ashish Chouhan, and Michael Gertz. 2022. Eur-lex-sum: A multi-and cross-lingual dataset for long-form summarization in the legal domain. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 7626-7639.
* Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. _arXiv preprint arXiv:2004.05150_.
* Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009. _Natural language processing with Python: analyzing text with the natural language toolkit_. O'Reilly Media, Inc.
* Cao et al. (2022) Meng Cao, Yue Dong, and Jackie Chi Kit Cheung. 2022. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3340-3354.
* Cao et al. (2020) Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstractive summarization models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6251-6258.
* Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: fact-aware neural abstractive summarization. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence_, AAAI'18/IAAI'18/EAAI'18. AAAI Press.
* Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakaisotiis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 2898-2904, Online. Association for Computational Linguistics.
* Chen et al. (2021) Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021. Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5935-5941.
* a tool augmented framework for multi-task and multi-domain scenarios. _ArXiv_, abs/2307.13528


Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4884-4895.
* Dong et al. (2022) Yue Dong, John Wieting, and Pat Verga. 2022. Faithful to the document or to the world? mitigating hallucinations via entity-linked knowledge in abstractive summarization. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 1067-1082.
* Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020. Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5055-5070.
* Dziri et al. (2021) Nouha Dziri, Andrea Madotto, Osmar Zaiane, and Avishek Joey Bose. 2021. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 2197-2214, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Fabbri et al. (2022) Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacetval: Improved qa-based factual consistency evaluation for summarization. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2587-2601.
* Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In _Annual Meeting of the Association for Computational Linguistics_.
* Fan et al. (2018) Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_, pages 45-54.
* Feng et al. (2023) Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. 2023. Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 933-952.
* Fiel and Sanyal (2023) Robert Fiel and Atindriyo Sanyal. 2023. Chainpoll: A high efficacy method for llm hallucination detection. _ArXiv_, abs/2310.18344.
* Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In _proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 166-175.
* Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3592-3603, Online. Association for Computational Linguistics.
* Harris et al. (2020) Charles R Harris, K Jarrod Millman, Stefan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. 2020. Array programming with numpy. _Nature_, 585(7825):357-362.
* He et al. (2022) Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2022. Ctrlsum: Towards generic controllable text summarization. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5879-5915.
* Huang et al. (2020) Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5094-5107, Online. Association for Computational Linguistics.
* Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Estuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. _ACM Computing Surveys_, 55(12):1-38.
* Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. _arXiv preprint arXiv:2401.04088_.
* Kornilova and Edelman (2019) Anastassia Kornilova and Vladimir Edelman. 2019. BillSum: A corpus for automatic summarization of US legislation. In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_, pages 48-56, Hong Kong, China. Association for Computational Linguistics.
* Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 540-551, Hong Kong, China. Association for Computational Linguistics.
* Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9332-9346.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjam Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: 

Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL'20), pp. 7871-7880. Cited by: SS3.
* C. Lin (2004)ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, pp. 74-81. External Links: Link, Document Cited by: SS1.
* Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023)G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 2511-2522. External Links: Link, Document Cited by: SS1.
* Z. Liu and N. Chen (2021)Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 92-106. Cited by: SS1.
* Y. Lyu, C. Zhu, T. Xu, Z. Yin, and E. Chen (2022)Faithful abstractive summarization via fact-aware consistency-constrained transformer. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, New York, NY, USA, pp. 1410-1419. External Links: ISBN 978-1-4503-3213-3, Link, Document Cited by: SS1.
* Potsawee Manakul, A. Lisuei, and M. Gales (2023)SelfCheckKgPT: zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 9004-9017. External Links: Link, Document Cited by: SS1.
* Y. Mao, X. Ren, H. Ji, and J. Han (2020)Constrained abstractive summarization: preserving factual consistency with constrained generation. arXiv e-prints, pp. arXiv-2010. Cited by: SS1.
* J. Maynez, S. Narayan, B. Bohnet, and R. McDonald (2020)On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906-1919. Cited by: SS1.
* J. J. Miller (2013)Graph database applications and concepts with neo4j. In Proceedings of the southern association for information systems conference, Atlanta, GA, USA, Vol. 2324, pp. 141-147. Cited by: SS1.
* N. Mundler, J. He, S. Jenko, and M. Vechev (2023)Self-contradictory hallucinations of large language models: evaluation, detection and mitigation. arXiv preprint arXiv:2305.15852. Cited by: SS1.
* F. Nan, C. dos Santos, H. Zhu, P. Ng, K. McKeown, R. Nallapati, D. Zhang, Zhiguo Wang, Andrew O. Arnold, and B. Xiang (2021a)Improving factual consistency of abstractive summarization via question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6881-6894. Cited by: SS1.
* F. Nan, R. Nallapati, Z. Wang, C. Nogueira dos Santos, H. Zhu, D. Zhang, K. McKeown, and B. Xiang (2021b)Entity-level factual consistency of abstractive text summarization. In Conference of the European Chapter of the Association for Computational Linguistics, Cited by: SS1.
* S. Narayan, S. Cohen, and M. Lapata (2018)Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807. External Links: Link, Document Cited by: SS1.
* S. Narayan, Y. Zhao, J. Maynez, G. Simoes, V. Nikolaev, and R. McDonald (2021)Planning with learned entity prompts for abstractive summarization. Transactions of the Association for Computational Linguistics9, pp. 1475-1492. Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. External Links: Link, Document Cited by: SS1.
* A. Paszke, S. Gross, F. Massa, A. Lecer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019)Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems32. Cited by: SS3.
* L. F. R. Ribeiro, M. Liu, I. Gurevych, M. Dreyer, and M. Bansal (2022)FactGraph: evaluating factuality in summarization with semantic graph representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 3238-3253. External Links: Link, Document Cited by: SS1.
* C. Sansone and G. Sperli (2022)Legal information retrieval systems: state-of-the-art and open issues. Information Systems106, pp. 101967. Cited by: SS1.
* T. Scialom, P. Dray, P. Gallinari, S. Lamprier, B. Piwowarski, J. Staiano, and A. Wang (2021a)Questeval: summarization asks for fact-based evaluation. In Conference on Empirical Methods in Natural Language Processing, Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowski, Jacopo Staiano, Alex Wang, 

and Patrick Gallinari. 2021b. Questeval: Summarization asks for fact-based evaluation. In _2021 Conference on Empirical Methods in Natural Language Processing_, pages 6594-6604. Association for Computational Linguistics.
* Shen et al. (2022) Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022. Multi-lexsum: real-world summaries of civil rights lawsuits at multiple granularities. In _Proceedings of the 36th International Conference on Neural Information Processing Systems_, NIPS '22, Red Hook, NY, USA. Curran Associates Inc.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5008-5020, Online. Association for Computational Linguistics.
* Wang et al. (2022) Guan Wang, Weihua Li, Edmund Lai, and Jianhua Jiang. 2022. Katsum: Knowledge-aware abstractive text summarization. _arXiv preprint arXiv:2212.03371_.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_, pages 38-45.
* Xiao and Carenini (2023) Wen Xiao and Giuseppe Carenini. 2023. Entity-based spancopy for abstractive summarization to improve the factual consistency. In _The 61st Annual Meeting Of The Association For Computational Linguistics_.
* Zhang et al. (2022) Haopeng Zhang, Semih Yavuz, Wojciech Kryscinski, Kazuma Hashimoto, and Yingbo Zhou. 2022. Improving the faithfulness of abstractive summarization via entity coverage control. In _Findings of the Association for Computational Linguistics: NAACL 2022_, pages 528-535, Seattle, United States. Association for Computational Linguistics.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhao et al. (2020) Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive summarization. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 2237-2249.
* Zheng et al. (2020) Changmeng Zheng, Yi Cai, Guanjie Zhang, and Qing Li. 2020. Controllable abstractive sentence summarization with guiding entities. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 5668-5678.

## Appendix A Scientific Artifacts

This work would not be possible without many open-source scientific artifacts, including pytorch Paszke et al. (2019), transformers Wolf et al. (2020), Neo4jl Miller, 2013), Numpy Harris et al. (2020), nltk Bird et al. (2009), the language models used for training and inference: LED Beltagy et al. (2020) and BART Lewis et al. (2020), the different baselines used: ECC Zhang et al. (2022), CTRLSum He et al. (2022), Miztral Jiang et al. (2024).

## Appendix B Notations

\begin{table}
\begin{tabular}{p{42.7pt} p{34.1pt}} \hline Notation & Description \\ \hline \(\mathcal{D}_{T}\) & Source documents of the train split \\ \(\mathcal{D}_{I}\) & Source documents of the inference split \\ \(\mathcal{D}\) & The total set of source documents such as \\  & \(\mathcal{D}=\mathcal{D}_{T}\cup\mathcal{D}_{I}\) \\ \(\mathcal{R}\) & Reference summaries \\ \(\mathcal{G}\) & Generated summaries \\ \(d_{i}\) & Source document such as \(d_{i}\in\mathcal{D}\) \\ \(r_{i}\) & Reference summary of \(d_{i}\) such as \(r_{i}\in\mathcal{R}\) \\ \(g_{i}\) & Generated summary of \(d_{i}\) such as \(g_{i}\in\mathcal{G}\) \\ \(E_{d_{i}}\) & Entities of \(d_{i}\) \\ \(E_{r_{i}}\) & Entities of \(r_{i}\) \\ \(E_{g_{i}}\) & Entities of \(g_{i}\) \\ \(F_{r_{i}}\) & Faithful entities of \(r_{i}\), such as \(F_{r_{i}}\subseteq E_{r_{i}}\) and \(F_{r_{i}}=E_{r_{i}}\cap E_{d_{i}}\) \\ \(A^{in}_{r_{i}}\) & Abstractive entities of the reference summary \(r_{i}\) that can be found in the dataset \\ \(A^{out}_{r_{i}}\) & Abstractive entities of the reference summary \(r_{i}\) that cannot be found in the dataset \\ \(A_{r_{i}}\) & The overall set of the abstractive entities of \(r_{i}\) such as \(A_{r_{i}}=A^{in}_{r_{i}}\cup A^{out}_{r_{i}}\) and \(A^{in}_{r_{i}}\cap A^{out}_{r_{i}}=\emptyset\) \\ \(H^{f}_{g_{i}}\) & Factual hallucinations in \(g_{i}\) \\ \(H^{nf}_{g_{i}}\) & Non factual hallucinations in \(g_{i}\) \\ \(H_{g_{i}}\) & The overall set of hallucinations, such as \(H_{g_{i}}=H^{f}_{g_{i}}\cup H^{nf}_{g_{i}}\) and \(H^{f}_{g_{i}}\cap H^{nf}_{g_{i}}=\emptyset\) \\ \(E_{S_{i,j}}\) & Entities selected by strategy \(S_{i,j}\) such as \(i\) refers to the couple of source document, reference summary \((d_{i},r_{i})\) and \(j\in[1,4]\) (cf. table 2) \\ \(G_{D_{T}}\) & Entity graph of the train split \\ \(G_{D_{I}}\) & Entity graph of the inference split \\ \hline \end{tabular}
\end{table}
Table 6: Key Notation Summary 

C legibase Example

**Source Document :** Affection of un lueu autre que la matique pour la calibration des marriages Disposant parfois de sallees trop petites pour une celebration dans do benomes conditions, ou de lieux prestigues, les marires rewordignante la posisblitile de cebleber des marriages dans un leeuvre qu'est de la manie. Depuis le 4 mars 2017, ils pequvent donc affectorout autre dittement communal situe sur leur territe pour la clebreton d'unions, a condition de recueillier raulisation probable du precouver de la Replabilique en tu lustament son projet de decision of affectation, accompoende de tous documents utilites << C(COCT, art, R. 2123-35) bij mentale retardte a savarer = que la déscion du marine parasites les conditions d'une elekhorne et specule, puibliquete et republiees certaine [t] que les conditions relatives à la bonne tennee de l'Etat civil vont satisfiates >> (CGCT, art, L. 2121-30). Le procureur de la Republique dispose deux mois pour finette comunicine von opposition monotivie, au si les elements tamrisla barsintransils pour la formueler. Dans ces cas, le della est propeto d'un mois par le procurver, quien in morlee naire. En cas de non-optimen, cofinent pour post ondice attree au synthement que que chain de la main commune, connomiumau an provencur copie de as decision.Condition de delegation des directions du maize en qualitite of d'Etat civil. Le _devert_ no **2017-2706** exicass j Les conditions de delegation des actes d'att civil aux reactionnaiers. Ains le mater pour deux-a delbarge av un poliaeste fronticonnaire trialaires de la commone tour so partie des fonctions qu'il exerce en tut qu'officer et d'et civil, samles pretoves a **Varicle 75 du Code civil** --Pour rappel, cet article reserve la extremoine du marriage dans son ensemble, et nomament la lectureurs **articles 2145 da El 458 chedu eville**, volif i Teit d'et civil seval, es!-daire le miare, ses adjointes et uniquement sonserve de delegation spatiotque du mine et Can elas emplacement de celui; les connies municipaux. Autrement ditt, un forceinnaire de la commme ne petit obtenir du majere une delegation pour lire mes articles. Le derer precise gellement que longue les matter deplice goes fonctions, is les axes desseas dans le cadre des fonctions amistr degrees > convertporter la seule signature de delegate.Sources :Deret **no 2017-270 du letmars 2017 pari n la délegation des fonctions altier de ciudre ciud exercises par le mariee et au lieu de deliberation des margices Lo n 2016-1547 du lo 18 novembre 2016** de modernisation de la justice du XXLie sieclec.

**Reference Summary :** Le ter mas 2017, le Gouvernement apis le derenzo 2017-270 en application des dispositions de la dfai de la troyenne 2016 de modizensira de la justice du XXL siecle relatives a la defektion des fonctions / conflicts? colleciter de l'Etat civil du maire et du lieu de celebration des marriages. La 2016-1527 offerants, por an _affectain_ (edition a para 2016-21238 du GCCET). In possibiliale van masrice de eclipse des marriages dans un arste buitement communal que celui de la matrie, un que la facult de digezter plus relevant temotions. Des dispositions sent entrees en visquer le 4 mars 2017 et st cont codiffees aux _articles_ 2012-2129 du Code general des collecttivats territoriales.

* ECC Generated Summary :** Les maries peuvrent affecter un hbitement communal depruis le Ter mars 2017, autre que la marie, pour celebrer des marriages, a condition d'obtentioni tautoris du procurement de la Republique, qui diosse de movais pour's oppor (proprognable d'un mois si les pieces transmisses ont insuffisantes). Cette autoirusione y sa gravitante les conditions de solomnicie, be orbitic et la conforminer de l'Etat civil. En cas de non-opposition, la nature optimal fornasiers as decison et enrumer le proveruer. Le (decson 2017-270 pernt egalement au maze de deleger orientatives fonctions d'feat civil aux fonctionnaires de la commune, sauf celltes relatives la acfermoinie de murriage, notamument la lecture des (articles 2144 or code civil et degree 2144/code civil qui rest reserve au maize, i sees adjoints ou, en cas de delegation specificung, a des consellers municipalaux.

* Mixtl Generated Summary :** Less maries peuvrent ediferre das marriages dans un line autre que la marie, a condition que celui-ci suit sinte sur le terrorite commentant et que le groupeur de le groupeur de le groupeur de coeissole croise de Geacion, (art-Elzza