# Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection

Benjamin Vendeville, Liana Ermakova, Pierre de Loor

► To cite this version:

Benjamin Vendeville, Liana Ermakova, Pierre de Loor. Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection. SIGIR '25: The 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, Jul 2025, Padua Italy, France. pp.3723-3732, 10.1145/3726302.3730304. hal-05305779

HAL Id: hal-05305779

https://hal.science/hal-05305779v1

Submitted on 9 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection

Benjamin Vendeville

Université de Bretagne Occidentale
Lab-STICC (UMR CNRS 6285)

Brest, France

benjamin.vendeville@univ-brest.fr

Liana Ermakova

Université de Bretagne Occidentale
HCTI, Brest France

Brest, France

liana.ermakova@univ-brest.fr

Pierre De Loor

ENIB

Lab-STICC (UMR CNRS 6285)

Brest, France

deloor@enib.fr

## Abstract

The general public often encounters complex texts but does not have the time or expertise to fully understand them, leading to the spread of misinformation. Automatic Text Simplification (ATS) helps make information more accessible, but its evaluation methods have not kept up with advances in text generation, especially with Large Language Models (LLMs). In particular, recent studies have shown that current ATS metrics do not correlate with the presence of errors. Manual inspections have further revealed a variety of errors, underscoring the need for a more nuanced evaluation framework, which is currently lacking. This resource paper addresses this gap by introducing a test collection for detecting and classifying errors in simplified texts. First, we propose a taxonomy of errors, with a formal focus on information distortion. Next, we introduce a parallel dataset of automatically simplified scientific texts. This dataset has been human-annotated with labels based on our proposed taxonomy. Finally, we analyze the quality of the dataset, and we study the performance of existing models to detect and classify errors from that taxonomy. These contributions give researchers the tools to better evaluate errors in ATS, develop more reliable models, and ultimately improve the quality of automatically simplified texts.

## CCS Concepts

* Information systems → Specialized information retrieval; Information extraction; Clustering and classification; Test collections.

## Keywords

Automatic Text Simplification; Error Classification; Hallucinations; Test Collection; Natural Language Processing; Large Language Models

## ACM Reference Format:

Benjamin Vendeville, Liana Ermakova, and Pierre De Loor. 2025. Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection. In *Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '25)*, July 13–18, 2025, Padua, Italy. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3726302.3730304

<figure><img src="image_2.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License. SIGIR '25, Padua, Italy

© 2025 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-1592-1/2025/07 https://doi.org/10.1145/3726302.3730304

## 1 Introduction

The internet has made it easier for everyone to share and access information. This has had a big impact both in very general areas, but also in very technical fields such as science. While this democratization has increased availability, it has also highlighted a critical problem: complex language and the lack of background knowledge are challenging obstacles for the general public to understand scientific documents. To solve this, the field of Automatic Text Simplification (ATS) aims to develop and investigate methods to facilitate comprehension of complex documents. Though not new, the field has grown rapidly with advances in natural language processing and large language models (LLMs) [16]. Despite the impressive capabilities of LLMs in text generation, they often introduce various errors [19]. These errors can include fluency issues (e.g., incorrect syntax, punctuation, or grammar), hallucinations (e.g., generating unsupported or false information) [8], and simplification-specific mistakes such as misrepresenting claims or overgeneralizing concepts [6, 7].

Current efforts in evaluating errors in open-ended generation primarily focus on truthfulness with respect to the source document or world knowledge [8, 11]. While effective for identifying certain issues, this approach fails to capture the full range of possible errors [6, 7]. Error detection requires a task-specific approach [10], which has yet to be systematically explored in ATS research. The only related work [9] proposes an edit-level evaluation, where some edits (e.g., repetitions, grammar errors, contradictions) are labeled as errors. However, this typology omits certain types of erroneous generations, loosely defines others, and does not align with commonly used terms such as *faithfulness* or *factual hallucination*. Besides, the evaluation relies on aligning spans between source and simplified texts, a time-consuming process that becomes even more complex when moving from sentence-level to document-level simplification. These limitations point to the need for a comprehensive taxonomy and annotation framework specifically designed for ATS, which, to our knowledge, has not yet been established.

*Problem.* The lack of a comprehensive framework for defining, detecting, and evaluating error detection methods in ATS limits progress, new resources are needed.

*Contributions.* In this resource paper, we make the following contributions to support researchers in building better error detection methods for ATS:

**Taxonomy:** A new taxonomy of errors in ATS with a focus on information distortion.

**Test Collection:** A test collection based on our taxonomy, accompanied by a detailed annotation scheme.
<figure><img src="image_3.png" /><figcaption>Figure 1: Structure of our typology of errors in text simplification.</figcaption></figure>

**Showcase:** We showcase error detection methods for identifying and classifying errors using our test collection.

The current state of the art in ATS lacks a formal definition of errors and evaluation tools. To the best of our knowledge, this work provides the first comprehensive taxonomy of ATS errors. In particular, we put an emphasis on formally defining errors related to this task. These resources address key challenges in the field, making it now possible for researchers to more systematically categorize ATS errors and develop more effective tools to detect and avoid them. We will use this test collection to introduce a task on error detection at the SimpleText shared task at CLEF 2025 on automatic text simplification [3].

The full taxonomy with examples and proper definitions is provided in the Appendix. This annotation scheme is designed to be used in a variety of simplification strategies by users with or without expertise in text simplification. The annotated dataset is derived from sentences automatically simplified using various models, which were extracted from system runs submitted to the SimpleText track at CLEF 2024 [5, 7]. There is a range of evaluation measures for text simplification against reference simplifications, yet none exhibits very high agreement with human labels of simplification quality [1]. Moreover, earlier analysis revealed a range of information distortion issues in these runs, which were ignored by standard ATS evaluation measures [4, 5]. Other papers have observed similar issues in the output of generative text simplification models [12].

Our resources aim to set a new standard for detecting and evaluating errors in ATS. As IR advances and its applications expand, the demand for tools to manage complex information, especially scientific data, will only increase. We expect ATS research to grow, with our work serving as a foundation for future developments.

In Section 2, we introduce our taxonomy and the methodology used to develop it, using a formal, fact-based approach to identifying errors in ATS. Section 3 describes our data annotation process based on this taxonomy and presents the results. In Section 4, we examine how existing error detection measures perform on our annotated

data. We conclude with the implications of these results and a discussion of potential directions for future research.

## 2 Developing the Error Taxonomy

In Text Simplification, the goal is to make text more understandable to a target audience, but this goal may mean different things depending on the audience [17], including different characteristics (syntactic, lexical, ...) or different levels (5th grade, 3rd grade, ...). We define here errors that appear regardless of the specific simplification goal, as well as errors that are the result of misunderstanding the simplification goal.

### 2.1 Structure of the Taxonomy

The taxonomy is shown in Figure 1. It is a tree composed of the four greater error categories, each with its specific errors. We define these four greater types of errors, based on where that error comes from, and what made the model make the error:

* **A. Fluency.** Is the answer provided in a correct form that a fluent speaker would speak?

* **B. Alignment.** Is the format of the answer correct?

* **C. Information.** Is the information provided accurate and relevant to the input?

* **D. Simplification.** Does the response focus on simplification?

In the following subsections, we discuss each main error category and its subtypes in detail.

### 2.2 Fluency Errors

Errors in this category relate to the capacity of the model to generate human-looking text. The evaluation of the fluency of text is a well-researched area [15]. In particular, we identify five types of errors that appear in ATS:

**Random generation** At least part of the answer is just a random string of words or numbers.

**Contradiction** Answer contradicts itself.
<figure><img src="image_4.png" /><figcaption>Figure 2: Diagram illustrating the sets and intersections involved in defining information errors.</figcaption></figure>

**Simple punctuation / grammar errors** Answer has punctuation errors that don't hinder comprehension.

**Redundancy** Repeated sentences, parts of sentences, or groups of sentences that do not need to be repeated.

More details about these errors, including examples, can be found in Appendix A.1 Fluency.

## 2.3 Alignment Errors

Alignment errors occur when the model fails to follow structured prompts or generate responses in the expected format. LLMs learn to interpret tags (e.g., <query>, <answer>), handle one-shot prompts, and maintain proper formatting (e.g., brackets, quotes). Misalignment can cause parsing issues or lead to cascading errors. These errors typically fall into two categories:

**Format Misalignment** Missing tags or symbols used for formatting, such as JSON symbols (", }) or starting tags like <answer>.

**Prompt Misalignment** The model generates unnecessary additional content such as another question, a different source text, or an unrelated answer.

More details about these errors, including examples, can be found in Appendix A.2 Alignment.

## 2.4 Information Errors

In text simplification, some errors can arise from the model's treatment of the information in the source text. In particular, we are interested in two questions:

**Topicality** Is the answer on topic?

**Truthfulness** Is the answer true?

To answer these questions systematically, we adopt an approach based on sets of facts (defined as tuples of {subject, relation, target}) as the unit of information. Let $F_{src}$ be the set of facts in the source text and $F_{gen}$ those in the generated text. We have:

$$F_{src} = \{f_s | f_s = (subj_s, rel_s, obj_s)\}$$

$$F_{gen} = \{f_g | f_g = (subj_g, rel_g, obj_g)\}$$

Where each triplet (subj, rel, obj) consists of a subject, a relation, and an object identified in the source S or generated text G. Based on this definition, we represent the sets of facts in Figure 2. Figure 2 illustrates the relationship between these sets (source text

and simplification) and the two key questions we explore: topicality and truthfulness. In the following subsections, we will define and investigate related errors.

### 2.4.1 Topicality
Simplification models have a strong tendency to generate text that contains off-topic facts [6]. We define on-topic facts as such: Let $R_S$ denote the set of all subjects, relations, and objects related to the source text topic:

$$R_S = \{x | x \text{ is a subject, relation, or object related to the topic.}\}.$$

We define $F_{topic}$ as the set of all facts (subj, rel, obj) s.t. subj, rel, obj $\in R_S$:

$$F_{topic} = \{(subj, rel, obj) | subj, rel, obj \in R_S\}.$$

From this, we define the following error:

**Topic Shift** Generated facts that are off-topic. Formally, we define the set of topic shift errors as:

$$Error_{topic} = F_{gen} \setminus F_{topic}$$

### 2.4.2 Truthfulness
To ensure that the simplified document remains accurate, we examine its truthfulness in relation to two distinct sets of facts:

* $F_{true}$ The set of true facts

* $F_{false}$ The set of false facts

We assume the source document is entirely truthful. A false fact in the simplified text can then arise in two ways:

**Contradictory to the source** These facts directly contradict information in $F_{src}$. Formally, we define the set of contradictory facts as:

$$F_{cont} = \{f_{cont} | \exists f_{src} \in F_{src}, f_{src} \text{ contradicts } f_{cont}\}.$$

**Contradictory to general knowledge** These are hallucinated facts that conflict with widely accepted truths.

For simplicity, off-topic facts $f \notin F_{topic}$ are treated as topicality errors rather than truthfulness errors, even if they are false. We then define two errors:

**Faithfulness hallucinations** Generated facts that contradict the source document. Formally, we define the set of faithfulness hallucination Errors as:

$$Error_{faithfulness} = F_{gen} \cap F_{topic} \cap F_{cont}$$

**Factuality hallucinations** Generated facts that contradict general knowledge. Formally, we define the set of factuality hallucination Errors as:

$$Error_{factuality} = F_{gen} \cap F_{topic} \cap F_{false} \setminus F_{cont}$$

More details about these errors, including examples, can be found in Appendix A.3 Information.

## 2.5 Simplification Errors

In text simplification, the goal is to remove unimportant information and add important details (e.g., adding definitions or context) while reformulating the content for simplicity. However, errors can arise when these operations are performed incorrectly. In particular, we are interested in two questions:

* Does the answer contain important information?

* Are the performed reformulations correct?
<figure><img src="image_5.png" /><figcaption>Figure 3: Diagram illustrating the sets and intersections involved in defining simplification errors.</figcaption></figure>

As with Information errors, we take a formal approach to studying these questions.

### 2.5.1 Importance.
Important facts are those that are essential for understanding the central point or purpose of the source document. What we classify as important, however, depends on the goal of the simplification. For instance:

* If the goal is to simplify a scientific document for language learners, certain technical details might not be important and could be omitted.

* If the goal is to help a non-expert understand the document's content, these same technical details may become crucial.

This highlights that the importance of information varies based on the target audience and purpose. As with Information errors, we define $F_{src}$ as the set of facts in the source text and $F_{gen}$ as the set of facts in the generated text. We then consider the set $F_{imp}$ of facts defined as important by the simplification goal, and make the following assumption:

* Every important information is true: $F_{imp} \subset F_{true}$

* Every important information is on topic: $F_{imp} \subset F_{topic}$

Figure 3 illustrates the relationship between these source text and simplification sets, and importance. We then define a source text of facts $F_{src}$ as maximally simple if:

$$F_{src} = F_{imp}$$

We then observe that for every source text that is not maximally simplified, we have one or multiple of the following:

* Not every fact included in the source text is necessarily important.

$$\exists f_{src} \in F_{src}, f_{src} \notin F_{imp}$$

* There may be important information (e.g. a definition) that is not included in the source text.

$$\exists f_{imp} \in F_{imp}, f_{imp} \notin F_{src}$$

Therefore, we can identify five different sets:

* **Out-of-scope generation:** Facts in the generation that are new and not important

$$Error_{scope} = F_{gen} \setminus F_{imp}$$

* **Loss of informative content:** Source facts that are important and absent from the generation

$$Error_{Loss} = F_{src} \cap F_{imp} \setminus F_{gen}$$

<figure><img src="image_6.png" /><figcaption>Figure 4: Diagram illustrating the sets and intersections involved in defining concept oversimplification errors.</figcaption></figure>

* **Summarization:** Source facts that are not important and removed from the generation

$$Summ = F_{src} \setminus F_{imp} \setminus F_{gen}$$

* **Clarification:** Facts in the generation that are important and new

$$Clarif = F_{gen} \cap F_{imp} \setminus F_{src}$$

* **Potential Clarification:** Facts that are important and not included in the source text or the generation

$$Clarif_{potential} = F_{imp} \setminus F_{src} \setminus F_{gen}$$

Here, we identify two errors (*Out-of-scope generation* and *Loss of informative content*) and three other transformations that do not qualify as errors. The two errors we define here arise from the problem of evaluating what constitutes important information in the context of the specific simplification goal. Therefore, we decided to group them under the *Content relevance* subcategory. In the end, these errors will need to be more properly defined based on a proper definition of the needs of each target audience. The three other transformations (*Summarization*, *Clarification*, and *Potential Clarification*) can be studied as a way to measure the quality of the simplification.

### 2.5.2 Reformulation.
Some errors arise not from the *addition* or *subtraction* of facts, but from the *reformulation*. In that context, as shown in Figure 4, we may have elements of facts that are replaced by a more general or more specific concept:

**Generalization** Replacing a concept with a more general one

**Specification** Replacing a concept with a more specific one

More formally, let $E$ represent the set of elements in a fact: the *subject*, *relation*, or *object*. We define the source fact $f_{src} = (e_1, e_2, e_{src}) \in F_{src}$ and the generated fact $f_{gen} = (e_1, e_2, e_{gen}) \in F_{gen}$ such that only one element differs between them. We then have the following substitution errors:

**Overgeneralization** Replacing a concept with a more general one, which causes the simplified fact to contradict the source. Formally:

$$e_{src} \subset e_{gen}, \quad f_{src} \iff f_{gen}$$

**Overspecification** Replacing a concept with a more general one, which causes the simplified fact to contradict the source. Formally:

$$e_{src} \supset e_{gen}, \quad f_{src} \iff f_{gen}$$
<table><caption>**Table 1: Error Type Distribution: True vs. False Counts and Percentage of True Cases**</caption><thead><tr><th>Error Label</th><th>#Total</th><th>#True</th><th>#False</th><th>%True</th></tr></thead><tbody><tr><td>No error</td><td>2,659</td><td>820</td><td>1,839</td><td>30.84</td></tr><tr><td><strong>A. Fluency</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>A1. Random generation</td><td>2,659</td><td>142</td><td>2,517</td><td>5.34</td></tr><tr><td>A2. Syntax error</td><td>2,659</td><td>191</td><td>2,468</td><td>7.18</td></tr><tr><td>A3. Contradiction</td><td>2,659</td><td>23</td><td>2,636</td><td>0.86</td></tr><tr><td>A4. Punctuation/ grammar error</td><td>2,659</td><td>241</td><td>2,418</td><td>9.06</td></tr><tr><td>A5. Redundancy</td><td>2,659</td><td>112</td><td>2,547</td><td>4.21</td></tr><tr><td><strong>B. Alignment</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>B1. Format misalignment</td><td>2,659</td><td>47</td><td>2,612</td><td>1.77</td></tr><tr><td>B2. Prompt misalignment</td><td>2,659</td><td>96</td><td>2,563</td><td>3.61</td></tr><tr><td><strong>C. Information</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>C1. Factuality hallucination</td><td>2,659</td><td>23</td><td>2,636</td><td>0.86</td></tr><tr><td>C2. Faithfulness hallucination</td><td>2,659</td><td>360</td><td>2,299</td><td>13.54</td></tr><tr><td>C3. Topic shift</td><td>2,659</td><td>152</td><td>2,507</td><td>5.72</td></tr><tr><td><strong>D. Simplification</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>D1.1. Overgeneralization</td><td>2,659</td><td>306</td><td>2,353</td><td>11.51</td></tr><tr><td>D1.2. Overspecification</td><td>2,659</td><td>136</td><td>2,523</td><td>5.11</td></tr><tr><td>D2.1. Loss of Info. Content</td><td>2,659</td><td>520</td><td>2,139</td><td>19.56</td></tr><tr><td>D2.2. Out-of-Scope Gen.</td><td>2,659</td><td>418</td><td>2,241</td><td>15.72</td></tr></tbody></table>

A certain level of generalization or specification is often necessary when simplifying text for specific audiences. However, excessive simplification can result in errors, such as overgeneralization or overspecification, where important nuances are lost or altered. To address this, we classify these errors under the broader category of *Oversimplification*. More details about these errors, including examples, can be found in Appendix A.4 Simplification.

## 3 Test Collection

We use this taxonomy to build a test collection of errors in ATS. This collection is composed of a dataset of sentences extracted from scientific abstracts across diverse fields, including medicine, telecoms, and artificial intelligence. The simplifications were gathered during the SimpleText Track at CLEF 2024, where participants then used a variety of methods to generate simplifications, all of them based on LLMs.

We have collected a total of 2,659 individual annotations from ten annotators, as detailed in Table 1. The annotations reveal a high error rate of 69.16% in the existing simplifications, with high variability in the occurrence of specific error types. Notably, *Contradiction* appears infrequently (0.86% occurrence), while *Loss of informative content* is highly prevalent, accounting for 19.56% of the sentences. These findings show that while text generation models have improved, challenges in fluency remain. Simplification-related errors continue to occur at a high rate, suggesting that further improvements are needed in this area.

To evaluate inter-annotator agreement, we gathered a separate set of 104 instances, each annotated by five of the annotators. The instances were selected from the subset of data annotated by the lead author to ensure a balanced representation of each error class within the test dataset. This dataset also included eight duplicated

<table><caption>**Table 2: Annotator Consistency Rate**</caption><thead><tr><th>Annotator</th><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th></tr></thead><tbody><tr><td>Consistency Rate</td><td>0.78</td><td>1</td><td>0.56</td><td>1</td><td>0.78</td></tr></tbody></table>

instances to measure annotator self-consistency. As shown in Table 2, the consistency rates varied across annotators, with only two out of five achieving perfect consistency. While annotator C achieved low consistency (0.56), we decided to keep their annotations in the dataset to increase the data size. Users who prefer higher quality can choose to exclude their annotations.

Table 3 presents Cohen's Kappa scores for error classes across annotator pairs. Undefined values were set to 1 in cases of complete agreement. Cohen's κ indicates the level of inter-annotator agreement, with scores above 0.60 generally considered substantial [14]. Fluency errors show slight to moderate agreement (κ = 0.19-0.44) across all pairs. Alignment errors exhibit high variability, with agreement ranging from 0.25 to 1. Information errors shows only slight to fair agreement across the board (κ = 0.06-0.40). Simplification errors scores range from slight to moderate (κ = 0.14-49).

Table 3 also displays Fleiss' Kappa and unanimous agreement. Agreement is highest for Alignment (κ = 0.45) and Aggregated A errors (κ = 0.38), suggesting clearer definitions or easier detection. In contrast, Information and Simplification errors show lower reliability (κ = 0.02-0.26), indicating more subjective or ambiguous judgments. These results are very diverse and highlight both consistent and challenging areas in error classification.

This test collection provides a valuable resource for analyzing the challenges of LLM-based text simplification. The high prevalence of errors, particularly in information loss and fluency, underscores the need for further improvements in automatic simplification models. The inter-annotator agreement analysis highlights the importance of annotator training, as expertise significantly impacts the identification of certain error types. By making this test collection available, we aim to facilitate future research in evaluating and enhancing simplification systems, contributing to more reliable and comprehensible scientific text generation.

## 4 Showcase

Evaluating ATS requires accurate error detection. The question remains, then, whether current error detection models can reliably capture errors as defined by our taxonomy. This showcase will highlight the strengths and limitations of current approaches and guide improvements in ATS evaluation.

### 4.1 Methodology

Our showcase of error detection models focuses on three key questions:

* Can they accurately detect the presence of errors in ATS?

* Can they effectively identify greater types of errors in ATS?

* Can they effectively identify specific types of errors in ATS?

Given that hallucination has been one of the most extensively studied errors in recent years [8, 11] with sometimes imprecise definition of hallucinations, we will use hallucination detection models to evaluate the performance of existing tools. These models
<table><caption>**Table 3:** Cohen's Kappa scores for error classes across all annotator pairs, along with Fleiss' Kappa scores and percentage of unanimous annotations.</caption><thead><tr><th rowspan="2">Error Class</th><th rowspan="2">Fleiss' κ</th><th rowspan="2">Unanimous %</th><th colspan="10">Cohen's Kappa scores for Annotator pair</th></tr><tr><th>AB</th><th>AC</th><th>AD</th><th>AE</th><th>BC</th><th>BD</th><th>BE</th><th>CD</th><th>CE</th><th>DE</th></tr></thead><tbody><tr><td>No error</td><td>0.34</td><td>38.9</td><td>0.49</td><td>0.30</td><td>0.16</td><td>0.48</td><td>0.66</td><td>0.28</td><td>0.73</td><td>0.23</td><td>0.57</td><td>0.16</td></tr><tr><td>A. Fluency</td><td>0.38</td><td>67.3</td><td>0.44</td><td>0.23</td><td>0.22</td><td>0.44</td><td>0.40</td><td>0.37</td><td>0.75</td><td>0.19</td><td>0.40</td><td>0.37</td></tr><tr><td>B. Alignment</td><td>0.45</td><td>76.8</td><td>0.37</td><td>0.58</td><td>1.00</td><td>0.38</td><td>0.25</td><td>0.37</td><td>0.79</td><td>0.58</td><td>0.27</td><td>0.38</td></tr><tr><td>C. Information</td><td>0.02</td><td>47.3</td><td>0.06</td><td>0.22</td><td>0.14</td><td>0.07</td><td>0.17</td><td>0.40</td><td>0.38</td><td>0.04</td><td>0.27</td><td>0.28</td></tr><tr><td>D. Simplification</td><td>0.26</td><td>25.2</td><td>0.26</td><td>0.24</td><td>0.14</td><td>0.40</td><td>0.49</td><td>0.19</td><td>0.45</td><td>0.20</td><td>0.45</td><td>0.12</td></tr></tbody></table>

are well-suited for identifying factual or faithfulness inconsistencies, making them an ideal starting point for testing the broader framework of error detection in ATS. In addition, we will use In particular, we will use the following pre-trained models:

* Transformer-based models trained on synthetic data

- **FactCC**: Based on BERT, also trained on synthetic summarizations [13].

- **LENS**: Based on RoBERTa-large, trained on manually annotated simplifications [9].

* Question-Generation & Question-Answering (QGQA) models

- **FEQA**: Uses two T5 models to assess faithfulness via question-answering [2].

- **QAGS**: Another QGQA-based model leveraging two T5 models [18].

* Fact-based models

- **Factacc**: Uses Named Entity Recognition and a Relation Classifier to verify factual consistency [8].

In addition, we will also study the results of BERTScore [20] as it is often used as a benchmark for ATS performance.

To evaluate error detection, since about half of our dataset is error-free, we use AUROC to measure how well models distinguish between correct and erroneous simplifications. For rarer, specific error types, we rely on AUPRC. This will help us understand the strengths and limitations of hallucination detection in ATS.

## 4.2 Results & Analysis

### 4.2.1 Binary Error Detection AUROC.
Table 4 summarizes the binary error detection performance as measured by AUROC for the evaluated models. FactCC achieves the highest AUROC of 0.68, indicating a relatively strong ability to distinguish between error and non-error instances. QAGS and FEQA both record moderate performance with AUROC values of 0.51 and 0.52, while FactAcc trails at 0.44. Notably, BERTScore shows a very low AUROC of 0.23, suggesting that its capability for ranking errors in a binary setting is limited.

### 4.2.2 Greater Error Type Detection.
Table 4 presents the AUPRC scores for detecting broader error types—fluency, alignment, informativeness, and simplification errors—along with their prevalence. FactCC consistently achieves the highest scores across all categories, with AUPRC of 0.24 for fluency, 0.16 for alignment, 0.29 for information, and 0.51 for simplification. Despite these relative improvements, the overall performance remains low. In particular,

<table><caption>**Table 4:** Greater Error Detection AUPRC and Binary Detection "Any Error" AUROC for evaluated models.</caption><thead><tr><th>Metric</th><th>A. Fluency</th><th>B. Alignment</th><th>C. Information</th><th>D. Simplification</th><th>No Error</th></tr></thead><tbody><tr><td>%True</td><td>21.96</td><td>5.15</td><td>19.10</td><td>43.89</td><td>30.84</td></tr><tr><td>#True</td><td>584</td><td>137</td><td>508</td><td>1167</td><td>822</td></tr><tr><td>BERTScore</td><td>0.20</td><td>0.02</td><td>0.13</td><td>0.36</td><td>0.23</td></tr><tr><td>QAGS</td><td>0.22</td><td>0.05</td><td>0.19</td><td>0.45</td><td>0.51</td></tr><tr><td>FEQA</td><td>0.23</td><td>0.05</td><td>0.20</td><td>0.44</td><td>0.52</td></tr><tr><td>FactCC</td><td>0.24</td><td>0.16</td><td>0.29</td><td>0.51</td><td>0.68</td></tr><tr><td>FactAcc</td><td>0.24</td><td>0.05</td><td>0.19</td><td>0.42</td><td>0.44</td></tr><tr><td>LENS</td><td>0.13</td><td>0.03</td><td>0.19</td><td>0.53</td><td>0.37</td></tr></tbody></table>

detecting Alignment errors is challenging for every model. Surprisingly, even on fluency and alignment errors, which might be easier to detect, models exhibit low detection performance.

### 4.2.3 Detection for Individual Error Types.
Table 5 reports the AUPRC scores for each model across individual error types, grouped into four main categories: Fluency, Alignment, Information, and Simplification. The table also lists the percentage of true instances and the absolute number of true cases for each error type. For Fluency errors, FactCC generally achieves the highest AUPRC scores, though the absolute values remain very low. A similar trend is observed for Alignment and Information errors, where FactCC outperforms the other models. For Simplification errors, the highest scores are shown by LENS.

Overall, although FactCC consistently shows relatively higher performance, the results always show very poor performance, highlighting the need for improved error detection methods. Moreover, the very low instance counts for certain error types, such as Contradiction, prompt misalignment, and factual hallucination, limit the robustness of the measure.

## 5 Conclusion

We introduced here the first taxonomy of errors in Automatic Text Simplification and built a test collection through the annotation of real-world ATS examples. Our findings show that errors are still
<table><caption>Table 5: Caption: Error analysis table presenting various error types and measure scores across multiple metrics. The table reports the AUPRC for BERTScore, QAGS, FEQA, FactCC, FactAcc, and LENS, along with the number of true instances. Errors are grouped into greater categories.</caption><thead><tr><th>Error Type</th><th>#Total</th><th>#True</th><th>%True</th><th>BERTScore ↑</th><th>QAGS ↑</th><th>FEQA ↑</th><th>FactCC ↑</th><th>FactAcc ↑</th><th>LENS ↑</th></tr></thead><tbody><tr><td><strong>A Fluency</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>A1. Random generation</td><td>2,659</td><td>142</td><td>5.34</td><td>0.0275</td><td>0.0520</td><td>0.0584</td><td><strong>0.1113</strong></td><td>0.0514</td><td>0.0288</td></tr><tr><td>A2. Syntax error</td><td>2,659</td><td>191</td><td>7.18</td><td>0.0544</td><td>0.0776</td><td>0.0831</td><td><strong>0.0846</strong></td><td>0.0716</td><td>0.0431</td></tr><tr><td>A3. Contradiction</td><td>2,659</td><td>23</td><td>0.86</td><td>0.0066</td><td>0.0118</td><td>0.0130</td><td><strong>0.0208</strong></td><td>0.0086</td><td>0.0094</td></tr><tr><td>A4. Punctuation/grammar errors</td><td>2,659</td><td>241</td><td>9.06</td><td>0.1371</td><td>0.0883</td><td>0.0952</td><td>0.0710</td><td><strong>0.1506</strong></td><td>0.0542</td></tr><tr><td>A5. Redundancy</td><td>2,659</td><td>112</td><td>4.21</td><td>0.0279</td><td>0.0419</td><td>0.0453</td><td><strong>0.0626</strong></td><td>0.0404</td><td>0.0278</td></tr><tr><td><strong>B Alignment</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>B1. Format misalignement</td><td>2,659</td><td>47</td><td>1.77</td><td>0.0061</td><td>0.0178</td><td>0.0176</td><td><strong>0.0633</strong></td><td>0.0170</td><td>0.0184</td></tr><tr><td>B2. Prompt misalignement</td><td>2,659</td><td>96</td><td>3.61</td><td>0.0213</td><td>0.0349</td><td>0.0400</td><td><strong>0.1200</strong></td><td>0.0352</td><td>0.0209</td></tr><tr><td><strong>C Information</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>C1. Factuality hallucination</td><td>2,659</td><td>23</td><td>0.86</td><td>0.0059</td><td>0.0099</td><td><strong>0.0175</strong></td><td>0.0105</td><td>0.0084</td><td>0.0093</td></tr><tr><td>C2. Faithfulness hallucination</td><td>2,659</td><td>360</td><td>13.54</td><td>0.1080</td><td>0.1421</td><td>0.1397</td><td><strong>0.1704</strong></td><td>0.1327</td><td>0.1577</td></tr><tr><td>C3. Topic shift</td><td>2,659</td><td>152</td><td>5.72</td><td>0.0306</td><td>0.0565</td><td>0.0617</td><td><strong>0.1498</strong></td><td>0.0572</td><td>0.0485</td></tr><tr><td><strong>D Simplification</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>D1.1. Overgeneralization</td><td>2,659</td><td>306</td><td>11.51</td><td>0.1011</td><td>0.1154</td><td>0.1157</td><td>0.1210</td><td>0.1102</td><td><strong>0.1871</strong></td></tr><tr><td>D1.2. Overspecification</td><td>2,659</td><td>136</td><td>5.11</td><td>0.0404</td><td>0.0542</td><td>0.0464</td><td><strong>0.0563</strong></td><td>0.0517</td><td>0.0540</td></tr><tr><td>D2.1. Loss of Informative Content</td><td>2,659</td><td>520</td><td>19.56</td><td>0.1712</td><td>0.1970</td><td>0.1848</td><td>0.2066</td><td>0.1899</td><td><strong>0.2252</strong></td></tr><tr><td>D2.2. Out-of-Scope Generation</td><td>2,659</td><td>418</td><td>15.72</td><td>0.1056</td><td>0.1657</td><td>0.1803</td><td><strong>0.2488</strong></td><td>0.1517</td><td>0.2010</td></tr></tbody></table>

prevalent in ATS and that existing methods fail to detect them reliably. While our taxonomy provides a structured approach to error classification, effective annotation requires careful selection and training of annotators. The test collection will be used for the SimpleText shared task at CLEF 2025 and will be published freely after the CLEF 2025 evaluation cycle [3] in a .csv format on GitHub.<fnref n="1" /> The repository also provides annotation scheme and the code for analyzing the annotated dataset introduced in this paper. By releasing our test collection, taxonomy annotation scheme, and analysis code, we aim to support further research in this area.

Future work could explore automatic evaluation aligned with our taxonomy, improved annotator training, and the use of synthetic data to augment test collections and facilitate the development of ATS models that better mitigate these errors.

## Acknowledgments

We thank master students in translation and technical writing from the University of Brest for their participation in data annotation. This research was funded by the French National Research Agency (ANR) under the projects ANR-22-CE23-0019-01 and ANR-19-GURE-0001 (program *Investissements d'avenir* integrated into France 2030).

# A Taxonomy

## A.1 A. Fluency

Category focus: is the answer provided in a correct language that a fluent speaker would speak, regardless of the correctness or relevance of the answer?

### A1. Random generation

* **Definition** At least part of the answer is just a random string of words/numbers

* **Example:**

- **Source:** In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- **Simplification:** Current academic and industrial research is interested in autonomous vehicles.1.2.3.4.5.6.7

### A2. Syntax error

* **Definition** The syntax is incorrect and doesn't make sense.

* **Example:**

- **Source:** In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- **Simplification:** In time now of robot and auto, cars that drive self are study much by school and work people.

### A3. Contradiction

* **Definition** Answer contradicts itself.

* **Example:**

- **Source:** In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- **Simplification:** In today's age of automation and robotics, autonomous vehicles are both widely researched and completely ignored by academics and industry.

### A4. Simple punctuation or grammar errors

* **Definition** Answer has punctuation errors that don't hinder comprehension.

* **Example:**

<footnote n="1">¹https://github.com/bVendeville/Salted</footnote>
- *Source:* In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- *Simplification:* Current academic and industrial research are interested in autonomous vehicles....................

### A5. Redundancy

* **Definition** Repeated sentences, parts of sentences, or groups of sentences that do not need to be repeated. This is an error regardless of the quality of the sentence.

* **Example:**

- *Source:* In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- *Simplification:* Current academic and industrial research is interested in autonomous vehicles. **Current academic and industrial research is interested in autonomous vehicles.**

## A.2 B. Alignment

Category focus: Does the answer suggest that the model correctly interpreted the prompt, including tags and format?

### B1. Format misalignment

* **Definition** Some tags or symbols used for formatting are missing. They can include symbols used for JSON parsing (like here with `"` and `)` or any "prompt tag" typically `<query><answer>` etc...

* **Example:**

- *Source:* In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- *Simplification:* `{"Current academic and industrial research is interested in autonomous vehicles."}`

### B2. Prompt misalignment

* **Definition** The model generated one or more of the following:

- unneeded prompt tags (like `<query><answer>` etc...) that lead to another question/source etc...

- another question (different or not)

- another source text (related or not)

- another answer (related or not)

* **Example:**

- *Source:* In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- *Simplification:* `{"Current academic and industrial research is interested in autonomous vehicles.<Query> simplify this: <example>...`

## A.3 C. Information

Category focus: Does the answer suggest that the model knows and understands everything needed to simplify the input?

### C1. Factality hallucination

* **Definition** The simplification contains facts that are contrary to (i.e., can be proven wrong from) "general knowledge" but not directly contrary to the input text.

* **Example:**

- *Source:* In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- *Simplification:* Current academic and industrial research is interested in autonomous vehicles, which are **vehicles that can fly**

### C2. Faithfulness hallucination

* **Definition** The simplification contains facts that are contrary to (i.e., can be proven wrong from) the input text.

* **Example:**

- *Source:* In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- *Simplification:* Current academic and industrial research is **not at all** interested in autonomous vehicles.

### C3. Topic shift

* **Definition** The generation contains at least some information related to the task (simplification) or the prompt (one-shot encoding) but not to the source document. It can be a text about simplification, or, in the case of one-shot inference, it can be something related to the example given but not to the document that should be simplified.

* **Example 1:**

- *Source:* In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- *Simplification:* **Simplification, in the context of language and communication, refers to the process of making text or information easier to understand.**

* **Example 2:**

- *Source:* Simplify the following document: `<source>In an attempt to achieve the above-mentioned tasks, we propose an imitation learning based, data-driven solution to UAV autonomy for navigating through city streets by learning to fly by imitating an expert pilot.<answer>Researchers propose data-driven solutions allowing drones to autonomously navigate city streets, learning to fly by imitating an expert pilot.<source>In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research. <answer>`

- *Simplification:* **We propose a data-driven imitation learning method for UAVs to navigate city streets by mimicking an expert pilot.**

## A.4 D. Simplification

Category focus: Does the answer suggest that the model understands the task of simplification?

### D1.1 Overgeneralization of Concepts

* **Definition** The simplification removes some precision and generalizes concepts that shouldn't be generalized, making them ambiguous and false. This may include:

- Replacing entities with the greater category of entities

- Using vague or ambiguous pronouns in place of clear subjects.
- Removing the target of a sentence, implying that a fact applies generally when it only applies in a specific case, to a specific entity

- Omitting critical context, such as targets, qualifiers, or conditions.

- Generalizing numerical or conditional statements into absolutes.

* **Example 1: Replacing Entities with a Broader Category**

- **Source:** Insects like bees and butterflies are vital for pollination, which is essential for producing many fruits and vegetables.

- **Simplification:** Insects are vital for pollination.

* **Example 2: Replacing a Specific Entity with a Pronoun**

- **Source:** The study found that aspirin reduced the risk of heart attack in patients over 50 but had no effect on younger individuals.

- **Simplification:** It reduces the risk of heart attack.

* **Example 3: Removing a Target, Leading to Unwarranted Generality**

- **Source:** This vaccine has been shown to be effective in preventing measles in children.

- **Simplification:** This vaccine prevents diseases.

## D1.2. Overspecification of Concepts

* **Definition** This error occurs when a broad entity or category in the source text is replaced with a specific example or subcategory during simplification. The source text may intentionally use a general term to avoid unnecessary detail or to maintain flexibility in interpretation. By introducing specificity, the simplified text risks reducing the meaning to an incorrect or unintended entity, misrepresenting the original intent.

- Replacing entities with the greater category of entities

- Using vague or ambiguous pronouns in place of clear subjects.

- Removing the target of a sentence, implying that a fact applies generally when it only applies in a specific case, to a specific entity

- Omitting critical context, such as targets, qualifiers, or conditions.

- Generalizing numerical or conditional statements into absolutes.

* **Example:**

- **Source:** The study examined the impact of climate change on wildlife.

- **Simplification:** The study examined the impact of climate change on polar bears.

## D2.1. Loss of Informative Content

* **Definition** Simplifications can omit critical information, making the content uninformative rather than misleading. This omission limits the reader's understanding of the broader context or key points, leaving them unaware of significant elements like parts of a research question, conclusions, or applications. This includes

- A completely empty simplification.

- A simplification so general it loses the source's novelty or explanatory value.

- Simplifying only one argument when the source has two independent ones.

**Note:** Deciding what qualifies as "important information" depends on the context. Defining "important information" should consider the audience's needs (e.g., non-native speakers, non-experts, or those with disabilities) and the desired simplicity level (e.g., 3rd grade, 10th grade).

* **Example 1 Replacing Entities with a Broader Category:**

- **Source:** Insects like bees and butterflies are vital for pollination, which is essential for producing many fruits and vegetables.

- **Simplification:** Insects are vital for pollination.

* **Example 2 Replacing a Specific Entity with a Pronoun:**

- **Source:** The study found that aspirin reduced the risk of heart attack in patients over 50 but had no effect on younger individuals.

- **Simplification:** It reduces the risk of heart attack.

* **Example 3 Removing a Target, Leading to Unwarranted Generality:**

- **Source:** This vaccine has been shown to be effective in preventing measles in children.

- **Simplification:** This vaccine prevents diseases.

* **Example 4 Generalizing Findings Beyond Their Scope:**

- **Source:** In the controlled study, the intervention improved test scores among high school students in urban areas.

- **Simplification:** The intervention improves test scores.

## D2.2. Out-of-Scope Generation

* **Definition** The generation contains information that is unrelated to the task of simplification. The generation may have something to do with the source document to be simplified, but is not about simplifying it. The generation might be:

- An opinion about the source document.

- A completion of the source document (more information).

- Questions about the source document.

- A translation of the source document.

* **Example 1:**

- **Source:** In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- **Simplification:** Current academic and industrial research is interested in autonomous vehicles. **In the show KITT with David Hasselhoff the car is an autonomous vehicle and on episode...**

* **Example 2:**

- **Source:** In the modern era of automation and robotics, autonomous vehicles are currently the focus of academic and industrial research.

- **Simplification:** Current academic and industrial research is interested in autonomous vehicles. In the show KITT with David Hasselhoff the car is an autonomous vehicle and on episode...

## References

[1] Dennis Davari, Liana Ermakova, and Ralf Krestel. 2024. Comparative Analysis of Evaluation Measures for Scientific Text Simplification. In *Linking Theory and Practice of Digital Libraries - 28th International Conference on Theory and Practice of Digital Libraries*, TPDL 2024, Ljubljana, Slovenia, September 24-27,
2024, Proceedings, Part 1 (Lecture Notes in Computer Science, Vol. 15177). Apostolos Antonacopoulos, Annika Hinze, Benjamin Piwowarski, Mickaël Coustaty, Giorgio Maria Di Nunzio, Francesco Gelati, and Nicholas Vanderschantz (Eds.). Springer, 76–91. https://doi.org/10.1007/978-3-031-72437-4_5

[2] Esin Durmus, He He, and Mona Diab. 2020. FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 5055–5070. https://doi.org/10.18653/v1/2020.acl-main.454

[3] Liana Ermakova, Hosein Azarbonyad, Jan Bakker, Benjamin Vendeville, and Jaap Kamps. 2025. CLEF 2025 SimpleText Track - Simplify Scientific Text (and Nothing More). In *Advances in Information Retrieval - 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6-10, 2025, Proceedings, Part V (Lecture Notes in Computer Science, Vol. 15576)*, Claudia Hauff, Craig Macdonald, Dietmar Jannach, Gabriella Kazai, Franco Maria Nardini, Fabio Pinelli, Fabrizio Silvestri, and Nicola Tonellotto (Eds.). Springer, 425–433. https://doi.org/10.1007/978-3-031-88720-8_63

[4] Liana Ermakova, Sarah Bertin, Helen McCombie, and Jaap Kamps. 2023. Overview of the CLEF 2023 SimpleText Task 3: Simplification of Scientific Texts. In *Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023 (CEUR Workshop Proceedings, Vol. 3497)*, Mohammad Aliannejadi, Guglielmo Faggioli, Nicola Ferro, and Michalis Vlachos (Eds.). CEUR-WS.org, 2855–2875. https://ceur-ws.org/Vol-3497/paper-240.pdf

[5] Liana Ermakova, Valentin Laimé, Helen McCombie, and Jaap Kamps. 2024. Overview of the CLEF 2024 SimpleText Task 3: Simplify Scientific Text. In *Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024 (CEUR Workshop Proceedings, Vol. 3740)*, Guglielmo Faggioli, Nicola Ferro, Petra Galuscáková, and Alba García Seco de Herrera (Eds.). CEUR-WS.org, 3147–3162. https://ceur-ws.org/Vol-3740/paper-307.pdf

[6] Liana Ermakova, Eric SanJuan, Stéphane Huet, Hosein Azarbonyad, Olivier Augereau, and Jaap Kamps. 2023. Overview of the CLEF 2023 SimpleText Lab: Automatic Simplification of Scientific Texts. In *Experimental IR Meets Multilinguality, Multimodality, and Interaction*, Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Anastasia Giachanou, Dan Li, Mohammad Aliannejadi, Michalis Vlachos, Guglielmo Faggioli, and Nicola Ferro (Eds.). Springer Nature Switzerland, Cham, 482–506. https://doi.org/10.1007/978-3-031-42448-9_30

[7] Liana Ermakova, Eric SanJuan, Stéphane Huet, Hosein Azarbonyad, Giorgio Maria Di Nunzio, Federica Vezzani, Jennifer D'Souza, and Jaap Kamps. 2024. Overview of the CLEF 2024 SimpleText Track: Improving Access to Scientific Texts for Everyone. In *Experimental IR Meets Multilinguality, Multimodality, and Interaction: 15th International Conference of the CLEF Association, CLEF 2024, Grenoble, France, September 9–12, 2024, Proceedings, Part II*. Springer-Verlag, Berlin, Heidelberg, 283–307. https://doi.org/10.1007/978-3-031-71908-0_13

[8] Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing The Factual Accuracy of Generated Text. In *Proceedings of the 25th ACM*

SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19). Association for Computing Machinery, New York, NY, USA, 166–175. https://doi.org/10.1145/3292500.3330955

[9] David Heineman, Yao Dou, Mounica Maddela, and Wei Xu. 2023. Dancing Between Success and Failure: Edit-level Simplification Evaluation Using SALSA. https://doi.org/10.48550/arXiv.2305.14458 arXiv:2305.14458 [cs]

[10] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating Factual Consistency Evaluation. https://doi.org/10.48550/arXiv.2204.04991 arXiv:2204.04991 [cs]

[11] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. https://doi.org/10.48550/arXiv.2311.05232 arXiv:2311.05232 [cs]

[12] Sebastian Joseph, Kathryn Kazanas, Keziah Reina, Vishnesh Ramanathan, Wei Xu, Byron Wallace, and Junyi Jessy Li. 2023. Multilingual Simplification of Medical Texts. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 16662–16692. https://doi.org/10.18653/v1/2023.emnlp-main.1037

[13] Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the Factual Consistency of Abstractive Text Summarization. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 9332–9346. https://doi.org/10.18653/v1/2020.emnlp-main.750

[14] J. R. Landis and G. G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. *Biometrics* 33, 1 (March 1977), 159–174.

[15] Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. https://doi.org/10.48550/arXiv.1808.07048 arXiv:1808.07048 [cs]

[16] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, et al. 2024. GPT-4 Technical Report. https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774 [cs]

[17] Advaith Siddharthan. 2014. A Survey of Research on Text Simplification. *ITL - International Journal of Applied Linguistics* 165, 2 (Dec. 2014), 259–298. https://doi.org/10.1075/itl.165.2.06sid

[18] Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 5008–5020. https://doi.org/10.18653/v1/2020.acl-main.450

[19] Xuanxin Wu and Yuki Arase. 2024. An In-depth Evaluation of GPT-4 in Sentence Simplification with Error-based Human Assessment. arXiv:2403.04963 [cs]

[20] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. arXiv:1904.09675 [cs]