<figure><img src="image_1.png" /></figure>

HAL
open science

# Context-Aware Document Simplification

Liam Cripwell, Joël Legrand, Claire Gardent

► To cite this version:

Liam Cripwell, Joël Legrand, Claire Gardent. Context-Aware Document Simplification. Findings of the Association for Computational Linguistics: ACL 2023, ACL, Jul 2023, Toronto, Canada. pp.13190-13206, 10.18653/v1/2023.findings-acl.834. hal-04369783

HAL Id: hal-04369783

https://hal.science/hal-04369783v1

Submitted on 2 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Context-Aware Document Simplification

Liam Cripwell

Université de Lorraine
CNRS/LORIA
liam.cripwell@loria.fr

Joël Legrand

Université de Lorraine
Centrale Supélec
CNRS/LORIA
joel.legrand@inria.fr

Claire Gardent

CNRS/LORIA
Université de Lorraine
claire.gardent@loria.fr

Abstract

To date, most work on text simplification has focused on sentence-level inputs. Early attempts at document simplification merely applied these approaches iteratively over the sentences of a document. However, this fails to coherently preserve the discourse structure, leading to suboptimal output quality. Recently, strategies from controllable simplification have been leveraged to achieve state-of-the-art results on document simplification by first generating a document-level plan (a sequence of sentence-level simplification operations) and using this plan to guide sentence-level simplification downstream. However, this is still limited in that the simplification model has no direct access to the local inter-sentence document context, likely having a negative impact on surface realisation. We explore various systems that use document context within the simplification process itself, either by iterating over larger text units or by extending the system architecture to attend over a high-level representation of document context. In doing so, we achieve state-of-the-art performance on the document simplification task, even when not relying on plan-guidance. Further, we investigate the performance and efficiency tradeoffs of system variants and make suggestions of when each should be preferred.

# 1 Introduction

Text simplification transforms a given text into a simpler version of itself that can be understood by a wider audience, while preserving the same core meaning (Gooding, 2022). It has also proven useful as a preprocessing step for downstream NLP tasks such as machine translation (Chandrasekar et al., 1996; Mishra et al., 2014; Li and Nenkova, 2015; Štajner and Popovic, 2016) and relation extraction (Miwa et al., 2010; Niklaus et al., 2016).

Most previous work has focused on sentence-level simplification by training neural models on

complex/simple sentence pairs under the assumption that they will learn to perform required operations (e.g. sentence splitting, lexical substitution or syntactic rephrasing) implicitly from the training data (Zhang and Lapata, 2017; Nisioi et al., 2017; Jiang et al., 2020). However, the imbalanced representation of simplification operations throughout popular datasets, and the overly-conservative models arising from their use, have led to attempts at controllable simplification to achieve more variation and diversity in output texts (Alva-Manchego et al., 2017; Cripwell et al., 2021; Maddela et al., 2021).

Recently, strategies from controllable simplification have been leveraged to achieve state-of-the-art results on the document simplification task (Cripwell et al., 2023). Specifically, by using a planning model capable of considering the sentences surrounding a complex sentence, a sentence-level simplification model can be guided such that the structure of the resulting document remains more coherent. Despite this success, the sentence simplification model still has no direct access to document context which we believe limits the extent to which it can accurately produce simplified sentences that are consistent with the larger document.

As such, we propose various systems that allow access to some representation of surrounding content within the simplification module, while still allowing for the possibility of plan-guidance. We show that in doing so, we are able to achieve state-of-the-art document simplification performance on the Newsela dataset, even without relying on a generated plan. Further, we investigate the performance and efficiency tradeoffs of various system variants.<fnref n="1" />

Our key contributions are (i) a detailed investigation of how document context, input text and simplification plans impact document-level simplifica-

<footnote n="1">¹Pretrained models, code, and data are available at https://github.com/liamcripwell/plan_simple.</footnote>
tion and (ii) several state of the art models for document simplification. We show in particular that document level simplification is improved by combining a representation of the local context surrounding complex sentences with a simplification plan indicating how complex sentences should be simplified (whether they should be deleted, rephrased, split or copied).

## 2 Related Work

**Context in Controlled Text Generation** The use of external context within controlled text generation pipelines has seen recent success in areas outside of simplification. Li et al. (2021) control review generation by using document and sentence-level plans in the form of knowledge graph subgraphs. Smith et al. (2020) control the style of generated dialog responses by conditioning on a desired style token appended to other contextual utterances. Hazarika et al. (2022) modulate the amount of attention paid to different parts of a dialog context and show that using contextual encoding of question phrases can guide a model to more often generate responses in the form of questions. Slobodkin et al. (2022) consider summarisation where salient spans are first identified before being used to control the generation, while Narayan et al. (2023) first generate a summarisation plan consisting of question-answer pairs.

**Simplification Planning** Certain controllable sentence simplification works have approached simplification as a planning problem whereby an operation plan is first generated before being realised downstream to form the simplified text. The first of these are revision-based models that predict a sequence of token-level operations (delete, substitute, etc.), allowing for more control and interpretability (Alva-Manchego et al., 2017; Dong et al., 2019; Kumar et al., 2020; Omelianchuk et al., 2021; Dehghan et al., 2022). Others have taken a sentence-level approach by predicting a high-level operation (sentence split, rephrase, etc.) and using this to condition more typical neural systems (Scarton and Specia, 2018; Scarton et al., 2020; Garbacea et al., 2021; Cripwell et al., 2022).

Recently, the latter approach was leveraged for document simplification where it obtained state-of-the-art performance (Cripwell et al., 2023). Here, a sequence of sentence-level operations is predicted for an entire document and then used to iteratively condition a sentence-level simplification model.

The system considers both local (token representation of the sentence) and global document context (sequence of sentence-level encodings) when predicting an operation for a given sentence.

**Document-Level Simplification.** Initial attempts at document simplification simply applied sentence simplification methods iteratively over documents (Woodsend and Lapata, 2011; Alva-Manchego et al., 2019b; Sun et al., 2021). However, it was noted this alone is insufficient for performing certain operations, often leading to poor discourse coherence in the output (Siddharthan, 2003; Alva-Manchego et al., 2019b).

Various sub-problems of document simplification have been approached in isolation, such as sentence deletion (Zhong et al., 2020; Zhang et al., 2022), insertion (Srikanth and Li, 2021), and reordering (Lin et al., 2021). Sun et al. (2021) took a holistic approach by iteratively applying a sentence-level model, but with additional encoders to embed the two preceding and following sentences, which are used as additional input during generation. However, this was unable to outperform baselines.

Recently, Cripwell et al. (2023) achieved state-of-the-art performance by producing a document simplification framework capable of performing all of the most common operations. Specifically, they use both high-level document context and sentence-level features to generate a plan specifying which operations to be performed on each sentence in a given document, which is then used to condition a sentence simplification model.

## 3 Problem Formulation

The goal of text simplification is to generate a text S that simplifies an input text C. In the document-level case, $C = c_1...c_n$ is a sequence of complex sentences and $S = s_1...s_m$ is a sequence of simple sentences. Cripwell et al. (2023) further decompose this task into a two-stage process wherein a generated plan conditions the simplification:

$$P(S | C) = P(S | C, O)P(O | C)$$

where $O = o_1...o_n$ is a simplification plan, i.e. a sequence of sentence-level simplification operations for C (copy, rephrase, split, or delete). The motivation here is that the plan provides a high-level description of how to transform C into S,
which can in turn be used to guide the iterative generation of the simplified document across sentences.

Although the use of such plans has shown improved results, little attention has been given to how the generation stage itself can be modified to improve document-level simplification. In this work, we investigate whether further changes can be made to simplification models in order to make better use of high-level plans, or alternatively, whether it is possible to forego the planning stage entirely by incorporating high-level document context into the generative model directly.

**Terminology and Notations.** We use the following terminology and notational conventions:

* $C = c_1...c_n$ is a complex document of $n$ sentences;

* $p_i$ is the $i$th paragraph from the complex document $C$;

* $S = s_1...s_m$ is a ground-truth simplified version of $C$, containing $m$ sentences;

* $\hat{S} = \hat{s}_1...\hat{s}_m$ is a predicted simplification of $C$, generated by a simplification model;

* $o$ is a simplification operation with value *copy*, *rephrase*, *split*, or *delete*;

* $\hat{O} = \hat{o}_1...\hat{o}_n$ is a predicted simplification plan stipulating specific sentence-level operations that should be applied to each $c_i \in C$ so as to arrive at some $\hat{S}$;

* $Z_i$ is a high-level representation of the document context for $c_i$. It is a sequence of vector encodings for a fixed window of sentences surrounding $c_i$ within $C$.

## 4 Data

For all experiments, we use Newsela-auto (Jiang et al., 2020) which is currently the highest-quality document-level simplification dataset available. It consists of 1,130 English news articles from the original Newsela (Xu et al., 2015) dataset which are each manually rewritten at five different levels of simplification, corresponding to discrete reading levels (0-4) of increasing simplicity. It also includes both sentence and paragraph alignments for each document pair. Like previous works, for all our models we prepend a control-token to the input specifying the target document reading level.

We use the same filtered version of Newsela-auto used in Cripwell et al. (2023), along with the same train/validation/test splits to allow for model comparison. This also includes plan labels, consisting of an operation (*copy*, *rephrase*, *split*, or *delete*) assigned to each sentence pair. Statistics of this data can be seen in Table 1.

<table><caption>Table 1: Statistics of the filtered Newsela-auto dataset from Cripwell et al. (2023). There is a train/validation/test split of 92.5%/2.5%/5%, assigned at the document-level (i.e. sentences and paragraphs from the same document will be contained within the same set). All reading-level variations of a specific article are also contained within the same set.</caption><thead><tr><th></th><th>Train</th><th>Validation</th><th>Test</th></tr></thead><tbody><tr><th># Document Pairs</th><td>16,946</td><td>457</td><td>916</td></tr><tr><th># Paragraph Pairs</th><td>335,018</td><td>9,061</td><td>17,885</td></tr><tr><th># Sentence Pairs</th><td>654,796</td><td>17,688</td><td>35,292</td></tr></tbody></table>

## 5 Models

We distinguish three model categories: (i) models whose sole input is text and which simplify a document either by iterating over its sentences/paragraphs or by handling the entire document as a single input; (ii) models that take both a complex sentence and some representation of its document context as input and simplify a document by iterating over its sentences; and (iii) models that are guided by a plan via control-tokens denoting sentence-level simplification operations prepended to the input sequence. These are illustrated in Table 2 and presented in more detail in the following subsections. Additional training details are outlined in Appendix A.

### 5.1 Text-Only Models

The most basic group of models we test are those that simply take a text sequence as input. We use baseline models trained to take entire documents or individual sentences. We also experiment with using paragraph inputs, the results of which we believe should scale better to the document-level than isolated sentences. Because paragraphs contain a wider token-level representation of local context this might provide enough information to maintain coherency in the discourse structure of the final document.

**BART.** We finetune BART (Lewis et al., 2020) to perform simplification at the document (BART<sub>doc</sub>),
<table><caption>Table 2: Different system types and the specific forms of text, context, and plan inputs they consume. C is a complex document, cᵢ is the ith sentence of C, and pᵢ is the ith paragraph of C. Ŷ is a predicted document simplification plan, ˆoᵢ is the individual operation predicted for the ith sentence, and ˆo<sub>j..j+|pᵢ|</sub> is the plan extract for a specific paragraph pᵢ, where j is the index of the first sentence in pᵢ.</caption><thead><tr><th rowspan="3">System</th><th colspan="9">Input</th></tr><tr><th colspan="3">Text</th><th colspan="3">Context</th><th colspan="3">Plan</th></tr><tr><th>Document</th><th>Paragraph</th><th>Sentence</th><th>Document</th><th>Paragraph</th><th>Sentence</th><th>Document</th><th>Paragraph</th><th>Sentence</th></tr></thead><tbody><tr><td>BART</td><td>C</td><td>p<sub>i</sub></td><td>c<sub>i</sub></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>LED</td><td>C</td><td>p<sub>i</sub></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ConBART</td><td>-</td><td>-</td><td>c<sub>i</sub></td><td>Z<sub>i</sub></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PG<sub>Dyn</sub> (2023)</td><td>-</td><td>-</td><td>c<sub>i</sub></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>hat<sub>i</sub></td></tr><tr><td>hat<sub>i</sub> → BART</td><td>C</td><td>p<sub>i</sub></td><td>c<sub>i</sub></td><td>-</td><td>hat<sub>i</sub></td><td>hat<sub>j..j+|p<sub>i</sub>|</sub></td><td>hat<sub>i</sub></td><td>-</td><td>hat<sub>i</sub></td></tr><tr><td>hat<sub>i</sub> → LED</td><td>C</td><td>p<sub>i</sub></td><td>-</td><td>-</td><td>hat<sub>i</sub></td><td>hat<sub>j..j+|p<sub>i</sub>|</sub></td><td>-</td><td>-</td><td>-</td></tr><tr><td>hat<sub>i</sub> → ConBART</td><td>-</td><td>-</td><td>c<sub>i</sub></td><td>Z<sub>i</sub></td><td>-</td><td>-</td><td>-</td><td>-</td><td>hat<sub>i</sub></td></tr></tbody></table>

sentence (BART<sub>sent</sub>), and paragraph (BART<sub>para</sub>) levels.<fnref n="2" /> Both BART<sub>sent</sub> and BART<sub>para</sub> are applied iteratively over a document and outputs are concatenated to form the final simplification result.

**Longformer.** Encoder-decoder models like BART often produce worse outputs and become much slower the longer the input documents are. Longformer (Beltagy et al., 2020) is one proposal that aims to overcome these limitations by using a modified self-attention mechanism that scales linearly with sequence length. We finetune a Longformer encoder-decoder to perform the simplification on documents (LED<sub>doc</sub>) and paragraphs (LED<sub>para</sub>).<fnref n="3" />

## 5.2 Context-Aware Model (ConBART)

We propose a context-aware modification of the BART architecture (ConBART) that is able to condition its generation on both an input sentence cᵢ and a high-level representation of its document context Zᵢ (a sequence of vectors representing surrounding sentences in the document). This is done via extra cross-attention layers in each decoder attention block that specifically focus on Zᵢ. The ConBART architecture is illustrated in Figure 1.

We produce Zᵢ by employing the same context representation strategy used for planning in Cripwell et al. (2023). Specifically, the document context is obtained by taking a fixed window of sentences surrounding the target cᵢ, encoding

them with Sentence-BERT (SBERT, (Reimers and Gurevych, 2019)), and applying custom positional embeddings to represent location within the document.

By generating the plan autoregressively, it is also possible to use previously simplified sentences within the left context of the current complex sentence, a method we refer to as *dynamic context*. In this case, the window of sentences represented within Zᵢ is defined:

$$Context_{i,r} = Concat(\hat{s}_{i-r..i-1}, c_{i..i+r}) \quad (1)$$

where r is the context window radius and ˆsᵢ is the simplification output for the ith sentence cᵢ. We use the same recommended setting of r = 13.

The intuition behind the ConBART architecture is that the contextual information should allow for the simplification model to implicitly learn useful features of the discourse structure of the document in a similar way to the planner in Cripwell et al. (2023).

## 5.3 Plan-Guided Systems

**Existing System.** We compare with the state-of-the-art system proposed by Cripwell et al. (2023), PG<sub>Dyn</sub>, which consists of a standard sentence-level BART model that is guided by a planner, which predicts the simplification operation to be applied each input sentence given a left and right context window of sentence representations, Zᵢ. The planner uses dynamic document context, allowing it to auto-regressively update the left context part of Zᵢ during planning as each sentence is simplified (see Equation 1).

<footnote n="2">²All models are initialised with the pretrained facebook/bart-base model from https://huggingface.co/facebook/bart-base.</footnote>

<footnote n="3">³All models are initialised with the pretrained allenai/led-base-16384 model from https://huggingface.co/allenai/led-base-16384.</footnote>
<table><caption>Table 3: **Results of document simplification systems on Newsela-auto.** For BARTScore, *h* is the hypothesis and *r* is the reference. Scores significantly higher than PG<sub>Dyn</sub> are denoted with * (*p* < 0.005). Significance was determined with Student's *t*-tests.</caption><thead><tr><th rowspan="2">System</th><th colspan="3">BARTScore ↑</th><th colspan="3">SMART ↑</th><th>FKGL ↓</th><th>SARI ↑</th><th colspan="2">Length</th></tr><tr><th>P<br>(r → h)</th><th>R<br>(h → r)</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th></th><th></th><th>Tok.</th><th>Sent.</th></tr></thead><tbody><tr><td>Input</td><td>-2.47</td><td>-1.99</td><td>-2.23</td><td>63.2</td><td>62.7</td><td>62.8</td><td>8.44</td><td>20.5</td><td>866.9</td><td>38.6</td></tr><tr><td>Reference</td><td>-0.93</td><td>-0.93</td><td>-0.93</td><td>100</td><td>100</td><td>100</td><td>4.93</td><td>99.9</td><td>671.5</td><td>42.6</td></tr><tr><td>BART<sub>doc</sub></td><td>-2.68</td><td>-2.76</td><td>-2.72</td><td>61.9</td><td>43.9</td><td>50.6</td><td>10.01</td><td>47.1</td><td>600.8</td><td>20.7</td></tr><tr><td>BART<sub>sent</sub></td><td>-1.63</td><td>-1.56</td><td>-1.60</td><td>78.9</td><td>80.1</td><td>79.3</td><td>5.03</td><td>73.0</td><td>666.4</td><td>42.6</td></tr><tr><td>BART<sub>para</sub></td><td>-1.85</td><td>-1.49*</td><td>-1.67</td><td>77.2</td><td>82.8*</td><td>79.6</td><td>5.28</td><td>73.7</td><td>752.8</td><td>45.6*</td></tr><tr><td>LED<sub>doc</sub></td><td>-1.68</td><td>-1.73</td><td>-1.70</td><td>75.3</td><td>74.9</td><td>74.8</td><td>4.87</td><td>68.7</td><td>643.7</td><td>41.5</td></tr><tr><td>LED<sub>para</sub></td><td>-1.61</td><td>-1.40*</td><td>-1.50*</td><td>81.1</td><td><b>85.5</b>*</td><td>83.0*</td><td>5.15</td><td>76.9*</td><td>712.9</td><td>44.9*</td></tr><tr><td>ConBART</td><td>-1.59</td><td>-1.50</td><td>-1.54*</td><td>81.2</td><td>82.5*</td><td>81.7</td><td>5.01</td><td>75.8</td><td>669.8</td><td>42.8</td></tr><tr><td>PG<sub>Dyn</sub> (2023)</td><td>-1.60</td><td>-1.54</td><td>-1.57</td><td>80.2</td><td>81.0</td><td>80.5</td><td>4.98</td><td>75.0</td><td>667.2</td><td>42.6</td></tr><tr><td>Ô → ConBART</td><td>-1.52*</td><td>-1.45*</td><td>-1.48*</td><td>82.8*</td><td>84.0*</td><td>83.2*</td><td>4.96</td><td>78.3*</td><td>671.6</td><td>43.0</td></tr><tr><td>Ô → BART<sub>para</sub></td><td>-1.75</td><td>-1.47*</td><td>-1.61</td><td>79.4</td><td>81.9</td><td>80.4</td><td>5.11</td><td>74.9</td><td>715.3</td><td>42.7</td></tr><tr><td>Ô → LED<sub>para</sub></td><td><b>-1.50</b>*</td><td><b>-1.42</b>*</td><td><b>-1.46</b>*</td><td><b>83.7</b>*</td><td>84.9*</td><td><b>84.1</b>*</td><td>5.09</td><td><b>78.5</b>*</td><td>683.1</td><td>42.8</td></tr><tr><td>PG<sub>Oracle</sub> (2023)</td><td>-1.39*</td><td>-1.40*</td><td>-1.40*</td><td>85.5*</td><td>85.0*</td><td>85.3*</td><td>4.91</td><td>80.7*</td><td>655.6</td><td>42.1</td></tr><tr><td>O → ConBART</td><td><b>-1.32</b>*</td><td><b>-1.32</b>*</td><td><b>-1.32</b>*</td><td><b>88.0</b>*</td><td><b>87.7</b>*</td><td><b>87.8</b>*</td><td>4.92</td><td><b>83.8</b>*</td><td>659.6</td><td>42.3</td></tr><tr><td>O → BART<sub>para</sub></td><td>-1.60</td><td>-1.36*</td><td>-1.48*</td><td>83.6*</td><td>85.3*</td><td>84.3*</td><td>5.07</td><td>79.7*</td><td>706.2</td><td>42.3</td></tr><tr><td>O → LED<sub>para</sub></td><td>-1.36*</td><td>-1.33*</td><td>-1.35*</td><td>87.0*</td><td>87.3*</td><td>87.1*</td><td>5.03</td><td>82.3*</td><td>673.6</td><td>42.4</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>Figure 1: **ConBART model architecture.** The added context attention layer is shown in yellow, which allows for cross-attention over high-level document content, Z<sub>i</sub>.</figcaption></figure>

**Pipelines.** We construct pipeline systems that consist of each of our proposed models, guided by a document plan generated by the planner from Cripwell et al. (2023) (the same as is used by PG<sub>Dyn</sub>). For this, we use modified versions of each simplification model that are trained to take an operation control-token at the beginning of each text input.

We refer to each of these pipeline systems as �Ô → *h*, where *h* is the simplification model. We also report results where the ground-truth/oracle plans are used to condition models (*O* → *h*).

Note that because the planner updates its document context autoregressively at the sentence-level, this does not interface perfectly with paragraph-level simplification models. As such, for pipelines using a paragraph-level simplification model (Ô → BART<sub>para</sub>, �Ô → LED<sub>para</sub>), we only update the planner's context after each paragraph has been processed. Thus, for those paragraph level models, the left context of a complex sentence *c*<sub>i</sub> is only simplified up to the first sentence of the paragraph containing *c*<sub>i</sub>, i.e.

$$Context_{i,r} = Concat(\hat{s}_{i-r..j-1}, c_{j..i+r}) \quad (2)$$

where *j* is the index of the first sentence within the same paragraph as *c*<sub>i</sub>, assuming *j* > *i* − *r*.

We also experimented with multi-task systems that are trained to perform both planning and sim-
plification within a single model, therefore not requiring a pipeline setup. However, this ultimately proved unsuccessful (further details in Appendix B).

# 6 Evaluation

## 6.1 Automatic Evaluation

Text simplification is often evaluated on the basis of 3 criteria: adequacy (or meaning preservation), fluency, and simplicity. For automatic evaluation, we use BARTScore (Yuan et al., 2021) and SMART (Amplayo et al., 2022) as analogs for both adequacy and fluency. Both are reference-based metrics that have previously been used for document simplification as well as other text generation tasks.

For assessing simplicity, we use both the Flesch-Kincaid grade level (FKGL) and SARI (Xu et al., 2016). FKGL is a document-level metric of text readability that has the highest correlation with human judgements (Scialom et al., 2021), while SARI is a simplification metric that has become a staple in the sentence-level simplification literature. We use EASSE (Alva-Manchego et al., 2019a) to calculate both of these.

At test time we generate sequences using beam search with a beam size of 5 and a maximum length of 1024 tokens.

## 6.2 Human Evaluation

Historically, automatic evaluation of long-form text generation has been very difficult to perform (Howcroft et al., 2020; Thomson and Reiter, 2020). As such, we conduct a human-evaluation of proposed systems to more accurately gauge performance.

As full documents are very long and difficult to compare, we conduct evaluations at the paragraph-level. For each comparison, a complex paragraph is shown next to an extract from a generated simplification corresponding to that paragraph. Evaluators are then asked to judge whether the generated text (i) is fluent (**fluency**); (ii) preserves the core meaning of the input (**adequacy**); and (iii) is simpler to read/understand (**simplicity**).

Using the test set, we randomly sample 33 complex paragraphs from each non-adjacent reading-level transition pairing, for a total of 198 paragraphs. We take the references and outputs from 4 high performing systems ($PG_{Dyn}$, $LED_{para}$, $\hat{O} \to LED_{para}$, $\hat{O} \to ConBART$) for each (990 outputs

in total) and have an annotator rate them on each of the 3 criteria. Because we use a large pool of annotators we impose a binary answering scheme (yes/no) in order to avoid the inter-annotator subjectivity that is inherent when using a Likert scale. The proportion of positive results is used as the final score for a given system.

Further details of the human evaluation are given in Appendix C.

# 7 Results and Discussion

Results are shown in Table 3. We also report results for other commonly used metrics in Appendix D.

**Context Awareness Matters.** Considering all metrics, we find that text-only models that take as input either a sentence ($BART_{sent}$) or a whole document ($BART_{doc}$, $LED_{doc}$) underperform models whose input is more local to the input sentence, either because they work at the paragraph level ($LED_{para}$) or because they take both the complex sentence and its local document context as input (ConBART). In other words, models that have access to a *local document context* ($LED_{para}$, ConBART) perform best overall.

**LED vs BART.** LED models ($LED_{doc/para}$) outperform their standard counterpart ($BART_{doc/para}$) showing that modified self-attention is not only more efficient but also more precise than standard self-attention in the case of long input.

**The Utility of Planning.** Plan guided models (4th horizontal block in Table 3) outperform their standard couterpart on all metrics, showing that a predicted plan has a positive impact on simplification. This is further supported by the fact that models guided by an oracle plan (5th block) provide even greater performance.

**Comparison with the State-of-the-Art ($PG_{Dyn}$).** $O \to ConBART$ is similar to $PG_{Dyn}$ in that, in both cases, a document is simplified by iterating over its sentences and prediction is guided by the local context of the sentence to be simplified. A key difference is that in $PG_{Dyn}$, this context is exclusively used to predict a simplification operation, while in $O \to ConBART$ it is additionally used to condition the generation of the simplified sentences. We find that adding this extra control results in significantly better scores compared to the state-of-the-art $PG_{Dyn}$ model. This illustrates that document context has utility for both planning (predicting the cor-
<table><caption>Table 4: **Human evaluation results for selected simplification systems.** The *minor* group includes those examples with a reading-level transition of 2 levels (e.g. 0-2, 1-3, etc.), whereas the *major* class includes those of 3-4 levels. Each of these groups make up half of the entire set. Ratings significantly different from the highest score in each column are denoted with * (*p* < 0.05) and ** (*p* < 0.01). Significance was determined with two proportion Z-tests.</caption><thead><tr><th rowspan="2">System</th><th colspan="3">Fluency</th><th colspan="3">Adequacy</th><th colspan="3">Simplicity</th><th rowspan="2">Mean</th></tr><tr><th>Minor</th><th>Major</th><th>All</th><th>Minor</th><th>Major</th><th>All</th><th>Minor</th><th>Major</th><th>All</th></tr></thead><tbody><tr><td>Reference</td><td>90.9*</td><td><strong>96.0</strong></td><td>93.4</td><td>80.8</td><td>70.7*</td><td>75.8</td><td>83.8*</td><td>82.8*</td><td>83.3</td><td>84.2</td></tr><tr><td>PG<sub>Dyn</sub> (2023)</td><td>91.9*</td><td>94.9</td><td>93.4</td><td><strong>83.8</strong></td><td>73.7</td><td>78.8</td><td>88.9</td><td>85.9</td><td>87.4</td><td>86.5</td></tr><tr><td>LED<sub>para</sub></td><td><strong>98.0</strong></td><td>92.9</td><td><strong>95.5</strong></td><td>81.8</td><td>80.8</td><td>81.3</td><td><strong>92.9</strong></td><td>85.9</td><td><strong>89.4</strong></td><td><strong>88.7</strong></td></tr><tr><td>Ô → LED<sub>para</sub></td><td>90.9*</td><td><strong>96.0</strong></td><td>93.4</td><td>80.8</td><td><strong>82.8</strong></td><td><strong>81.8</strong></td><td>83.8*</td><td>90.9</td><td>87.4</td><td>87.5</td></tr><tr><td>Ô → ConBART</td><td>89.9**</td><td><strong>96.0</strong></td><td>92.9</td><td>81.8</td><td>79.8</td><td>80.8</td><td>86.9</td><td><strong>91.9</strong></td><td><strong>89.4</strong></td><td>87.7</td></tr></tbody></table>

rect simplification operation) and realisation (sim-
plifying a given sentence). While $\hat{O} \to LED_{para}$
achieves the best overall results of any system, it
is slightly outperformed by $O \to ConBART$ when
oracle plans are used, suggesting that an improved
planner would provide better simplifications when
used by $\hat{O} \to ConBART$ over $\hat{O} \to LED_{para}$.

**Human Evaluation** Results from the human evaluation are shown in Table 4. To better identify where each model excels, we report separate scores for test paragraph pairs with minor (reading-level transition of 2) and major (>2) degrees of simplification, as well as total average scores.

On fluency, all of the systems achieve very high ratings, which is unsurprising given the recognised ability of large language models (LLMs) to produce highly fluent texts. For adequacy, $\hat{O} \to LED_{para}$ achieves the highest overral score, closely followed by $LED_{para}$ and $\hat{O} \to ConBART$. In terms of simplicity, $LED_{para}$ and $\hat{O} \to ConBART$ equally achieve the highest score. Across all criteria, $LED_{para}$ achieves the highest average ratings, although very few scores are significantly better than other systems.

When considering performance differences between the minor and major simplification groups, we observe some clear trends. Systems that are not guided by a high-level document plan or do not have access to some contextual information during generation (PG<sub>Dyn</sub> and LED<sub>para</sub>) perform notably worse on examples requiring major simplification than they do on minor cases. Conversely, the models with both of these features appear to either perform equally as well or even excel on major cases. This suggests potential conservativity in the simplifications performed by PG<sub>Dyn</sub> and LED<sub>para</sub>.

Another interesting observation is the relatively

<figure><img src="image_4.png" /><figcaption>Figure 2: **Example WikiLarge simplification output extracts from $\hat{O} \to LED_{para}$ (a target reading-level of 3 was used in each case).** Note that these are small extracts from larger documents shown in Appendix E. Deletions are *underlined and in red*; rephrasings are *italicised and in green*; splitting points are *highlighted in cyan*; and factual errors are circled.</figcaption></figure>
low ratings given to the references compared to the system outputs. In particular, they receive a much lower adequacy score than any other system on major cases. This could perhaps be a result of the systems generating outputs that bear more of a resemblance to the inputs than those written by humans (see faithfulness BARTScores in Appendix D). For instance, human editors might have been able to confidently delete more content, or refer to some of the information in different paragraphs which the evaluators were not privy to. Despite this, the references still receive fluency ratings competitive with the other systems.

**Example Simplifications** Figure 2 shows some example simplification outputs from $\hat{O} \to LED_{para}$. These are paragraph-level extracts from larger document outputs, which are provided in Appendix E. Due to licensing constraints imposed by Newsela, we use out-of-domain documents from the Wiki-Large dataset (Zhang and Lapata, 2017) in these examples.

## 8 Model Efficiency

There are various others factors to consider when comparing systems, beyond their raw performance. For instance, the size of the model(s) and how much time/resources are required for each to perform inference are important practical considerations that must be made when selecting a model for real-world use. As such, we compare each system based on the time taken to simplify the test set and their total parameter counts. Table 5 shows these results.

In our case, any system that uses a plan requires a second model, approximately doubling the number of parameters that must be loaded. These pipeline setups also naturally add to overall inference time. Further, both plan pipelines and ConBART make use of dynamic context, which imposes an autoregressive bottleneck on the simplification of individual documents.

Because of the linearly scaling attention mechanism, Longformer-based models are the fastest of proposed systems. Because of this and its overall high performance, we recommend $LED_{para}$ in situations where time or computing resources are at all limited. Alternatively, $\hat{O} \to ConBART$ offers a good compromise that provides the high performance of a plan-guided system while mitigating further increases to inference time. This is because it uses the same autoregressive protocol as the planer and can therefore share the generated context

representations.

All inference processes were run on a single Nvidia A40 GPU, using a batch size of 16, 32 CPU workers for data loading, and a beam size of 5 for generation. Appendix F provides details on the specific algorithm used to handle dynamic context generation for appropriate models.

<table><caption>**Table 5: Model efficiency statistics.** All times are in milliseconds and model parameters are in millions. Inference times are calculated on the test set and normalised by the total number of sentences (i.e. # ms per sentence).</caption><thead><tr><th></th><th>Inference Time ↓</th><th># Params ↓</th></tr></thead><tbody><tr><td>BART<sub>doc</sub></td><td>182.6</td><td>140</td></tr><tr><td>BART<sub>sent</sub></td><td>54.0</td><td>140</td></tr><tr><td>BART<sub>para</sub></td><td>68.9</td><td>140</td></tr><tr><td>LED<sub>doc</sub></td><td>49.1</td><td>162</td></tr><tr><td>LED<sub>para</sub></td><td>45.9</td><td>162</td></tr><tr><td>ConBART</td><td>74.7</td><td>156</td></tr><tr><td>PG<sub>Dyn</sub> (2023)</td><td>76.6</td><td>154+140</td></tr><tr><td>hat O → BART<sub>para</sub></td><td>119.1</td><td>154+140</td></tr><tr><td>hat O → LED<sub>para</sub></td><td>103.3</td><td>154+162</td></tr><tr><td>hat O → ConBART</td><td>82.7</td><td>154+156</td></tr></tbody></table>

## 9 Conclusion

We develop a range of document simplification models that are able to use different combinations of text, context, and simplification plans as input, with several models outperforming the previous state-of-the-art both on automatic metrics and according to human judgements. Our results show that a high-level representation of the document can be useful for low-level surface realisation as well as global planning. Further, simplification models with access to local document context, either by working at the paragraph level or handling an additional input representation, lead to better meaning preservation than those that operate on individual sentences. We conclude by evaluating the model efficiency of each system and making recommendations for their selection under different circumstances.

## 10 Limitations

**Newsela Dataset** One limitation to this study is our use of the Newsela dataset. Because this requires a license to access, researchers cannot fully reproduce our work without first obtaining permission from Newsela Inc. Unfortunately there is currently no other large dataset offering high quality
aligned documents for simplification under an open
source license. The only other datasets so far used
for document-level simplification are based on Wik-
iLarge, which has very poor and inconsistent align-
ments at the document-level (Xu et al., 2015; Sun
et al., 2021; Cripwell et al., 2023).

**Paragraph-Level Human Evaluation** In order to reduce complexity, our human evaluation was performed on paragraphs rather than full documents. As a result, there is a potential limit to the accuracy of human judgements when certain discourse phenomena are present. For example, important information may be excluded from a specific output paragraph (therefore prompting a low adequacy rating), but this could actually be present in a different part of the true simplified document.

**Monolinguality** This study focused entirely on simplification for English-language documents. Re-producing the proposed systems for use on other languages would require dedicated datasets of sim-ilar scale, along with sentence/paragraph align-ments and operation labels (which likely do not currently exist). Further, the nature of simplifica-tion in other languages may differ quite a lot from English with respect to the types of operations that are performed, potentially reducing the suitability of the proposed framework.

**Generalised Target Audience** We approach this study with our definition of "simplification" being based on that of a generalised audience, following the standard set out by the assigned reading-levels of the Newsela dataset. Existing works often out-line the intent for their systems to be used to si-multaneously assist a wide array of different target users, such as those with cognitive impairments, non-native speakers, and children (Maddela et al., 2021; Garbacea et al., 2021; Sun et al., 2021). How-ever, they rarely go into any detail about which simplification strategies work for each of these dif-ferent groups or perform human evaluation with an-notators from the same target demographics (Good-ing, 2022). As such, we acknowledge that using our systems for a specific demographic might prove insufficient to enable their consumption of media without first making further revisions to support their precise needs.

References

Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of complex-simplified text pairs. In *Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 295–305, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Fernando Alva-Manchego, Louis Martin, Carolina Scarton, and Lucia Specia. 2019a. EASSE: Easi-er automatic sentence simplification evaluation. In *Proceedings of the 2019 Conference on Empiri-cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations*, pages 49–54, Hong Kong, China. Association for Computational Linguistics.

Fernando Alva-Manchego, Carolina Scarton, and Lu-cia Specia. 2019b. Cross-sentence transformations in text simplification. In *Proceedings of the 2019 Workshop on Widening NLP*, pages 181–184, Flo-rence, Italy. Association for Computational Linguis-tics.

Reinald Kim Amplayo, Peter J. Liu, Yao Zhao, and Shashi Narayan. 2022. Smart: Sentences as basic units for text evaluation.

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer.

R. Chandrasekar, Christine Doran, and B. Srinivas. 1996. Motivations and methods for text simplifica-tion. In *COLING 1996 Volume 2: The 16th Interna-tional Conference on Computational Linguistics*.

Liam Cripwell, Joël Legrand, and Claire Gardent. 2021. Discourse-based sentence splitting. In *Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 261–273, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Liam Cripwell, Joël Legrand, and Claire Gardent. 2022. Controllable sentence simplification via operation classification. In *Findings of the Association for Computational Linguistics: NAACL 2022*, pages 2091–2103, Seattle, United States. Association for Computational Linguistics.

Liam Cripwell, Joël Legrand, and Claire Gardent. 2023. Document-level planning for text simplification. In *Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin-guistics*, pages 993–1006, Dubrovnik, Croatia. As-sociation for Computational Linguistics.

Mohammad Dehghan, Dhruv Kumar, and Lukasz Go-lab. 2022. GRS: Combining generation and revision in unsupervised sentence simplification. In *Find-ings of the Association for Computational Linguis-tics: ACL 2022*, pages 949–960, Dublin, Ireland. As-sociation for Computational Linguistics.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-interpreter model for sentence simplification through explicit editing. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 3393–3402, Florence, Italy. Association for Computational Linguistics.

Cristina Garbacea, Mengtian Guo, Samuel Carton, and Qiaozhu Mei. 2021. Explainable prediction of text complexity: The missing preliminaries for text simplification. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 1086–1097, Online. Association for Computational Linguistics.

Sian Gooding. 2022. On the ethical considerations of text simplification. In *Ninth Workshop on Speech and Language Processing for Assistive Technologies (SLPAT-2022)*, pages 50–57, Dublin, Ireland. Association for Computational Linguistics.

Devamanyu Hazarika, Mahdi Namazifar, and Dilek Hakkani-Tür. 2022. Zero-shot controlled generation with encoder-decoder transformers.

David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In *Proceedings of the 13th International Conference on Natural Language Generation*, pages 169–182, Dublin, Ireland. Association for Computational Linguistics.

Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7943–7960, Online. Association for Computational Linguistics.

Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vechtomova. 2020. Iterative edit-based unsupervised sentence simplification. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7918–7928, Online. Association for Computational Linguistics.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7871–7880, Online. Association for Computational Linguistics.

Junyi Li, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, and Ji-Rong Wen. 2021.

Knowledge-based review generation by coherence enhanced text planning. In *Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR '21, page 183–192, New York, NY, USA. Association for Computing Machinery.

Junyi Jessy Li and Ani Nenkova. 2015. Detecting content-heavy sentences: A cross-language case study. In *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*, pages 1271–1281, Lisbon, Portugal. Association for Computational Linguistics.

Zhe Lin, Yitao Cai, and Xiaojun Wan. 2021. Towards document-level paraphrase generation with sentence rewriting and reordering. In *Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 1033–1044, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Mounica Maddela, Fernando Alva-Manchego, and Wei Xu. 2021. Controllable text simplification with explicit paraphrasing. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 3536–3553, Online. Association for Computational Linguistics.

Kshitij Mishra, Ankush Soni, Rahul Sharma, and Dipti Sharma. 2014. Exploring the effects of sentence simplification on Hindi to English machine translation system. In *Proceedings of the Workshop on Automatic Text Simplification - Methods and Applications in the Multilingual Society (ATS-MA 2014)*, pages 21–29, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.

Makoto Miwa, Rune Sætre, Yusuke Miyao, and Jun'ichi Tsujii. 2010. Entity-focused sentence simplification for relation extraction. In *Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)*, pages 788–796, Beijing, China. Coling 2010 Organizing Committee.

Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, and Mirella Lapata. 2023. Conditional generation with a question-answering blueprint.

Christina Niklaus, Bernhard Bermeitinger, Siegfried Handschuh, and André Freitas. 2016. A sentence simplification system for improving relation extraction. In *Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations*, pages 170–174, Osaka, Japan. The COLING 2016 Organizing Committee.

Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. Exploring neural text simplification models. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 85–91,
Vancouver, Canada. Association for Computational Linguistics.

Kostiantyn Omelianchuk, Vipul Raheja, and Oleksandr Skurzhanskyi. 2021. Text Simplification by Tagging. In *Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications*, pages 11–25, Online. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In *Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics*, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. **Sentence-BERT**: Sentence embeddings using Siamese BERT-networks. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

C. Scarton, P. Madhyastha, and L. Specia. 2020. Deciding when, how and for whom to simplify. © 2020 The Author(s) and IOS Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial Licence (http://creativecommons.org/licenses/by-nc/4.0/).

Carolina Scarton and Lucia Specia. 2018. Learning simplifications for specific target audiences. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 712–718, Melbourne, Australia. Association for Computational Linguistics.

Thomas Scialom, Louis Martin, Jacopo Staiano, Éric Villemonte de la Clergerie, and Benoît Sagot. 2021. Rethinking automatic evaluation in sentence simplification.

Advaith Siddharthan. 2003. Preserving discourse structure when simplifying text. In *Proceedings of the 9th European Workshop on Natural Language Generation (ENLG-2003) at EACL 2003*, Budapest, Hungary. Association for Computational Linguistics.

Aviv Slobodkin, Paul Roit, Eran Hirsch, Ori Ernst, and Ido Dagan. 2022. Controlled text reduction.

Eric Michael Smith, Diana Gonzalez-Rico, Emily Dihan, and Y-Lan Boureau. 2020. Controlling style in generated dialogue.

Neha Srikanth and Junyi Jessy Li. 2021. Elaborative simplification: Content addition and explanation generation in text simplification. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, pages 5123–5137, Online. Association for Computational Linguistics.

Sanja Štajner and Maja Popovic. 2016. Can text simplification help machine translation? In *Proceedings of the 19th Annual Conference of the European Association for Machine Translation*, pages 230–242.

Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021. Document-level text simplification: Dataset, criteria and baseline. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 7997–8013, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Craig Thomson and Ehud Reiter. 2020. A gold standard methodology for evaluating accuracy in data-to-text systems. In *Proceedings of the 13th International Conference on Natural Language Generation*, pages 158–168, Dublin, Ireland. Association for Computational Linguistics.

K. Woodsend and Mirella Lapata. 2011. Wikisimple: Automatic simplification of wikipedia articles. In AAAI.

Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. *Transactions of the Association for Computational Linguistics*, 3:283–297.

Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. *Transactions of the Association for Computational Linguistics*, 4:401–415.

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In *Advances in Neural Information Processing Systems*, volume 34, pages 27263–27277. Curran Associates, Inc.

Bohan Zhang, Prafulla Kumar Choubey, and Ruihong Huang. 2022. Predicting sentence deletions for text simplification using a functional discourse structure. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 255–261, Dublin, Ireland. Association for Computational Linguistics.

Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*, pages 584–594, Copenhagen, Denmark. Association for Computational Linguistics.

Yang Zhong, Chao Jiang, Wei Xu, and Junyi Jessy Li. 2020. Discourse level factors for sentence deletion in text simplification. *Proceedings of the AAAI Conference on Artificial Intelligence*, 34(05):9709–9716.
## A Training Details

For all simplification models, we used a learning rate of $2e^{-5}$, a batch size of 16, and a 0.1 dropout rate. All models were trained on a computing grid using 2x Nvidia A40 GPUs (45GB memory) until convergence or a maximum of 48 hours.

For ConBART and planning pipelines we use the same settings as Cripwell et al. (2023) for construction of the high-level document context. Specifically, this includes a fixed context window radius of size 13 and use of a dynamic context mechanism.

## B Multi-Task Systems

We also experimented with models that are explicitly trained to perform both the planning and simplification tasks using the same network. As high-level plans appear to improve the performance of simplification models, we hypothesise that learning both tasks in tandem could benefit overall performance. The motivation for this approach is to potentially produce a model that is capable of yielding similar or better simplification performance to the pipeline systems but with a more efficient single-model setup.

Specifically, these models were trained to generate the simplified text prefixed by a predicted plan in the form of operation-specific tokens. This was tested with both ConBART (ConBART<sub>prefix</sub>) and a document-level Longformer (LED<sub>prefix</sub>). In the case of the Longformer we also test a variant that generates the plan tokens as sentence separators (LED<sub>sep</sub>). Results are shown in Table 6.

Unfortunately, from our experiments none of these seemed to result in performance exceeding those of simplification-only models. Improvement could perhaps be reached given the correct tuning of hyperparameters and loss weightings, however we did not have the time or resources to pursue this further in this study.

## C Human Evaluation Details

The Newsela-auto paragraph alignments were used to identify valid references for each test paragraph. In order to align correct extracts from generated system outputs we took different steps depending on the system. For paragraph-level models (those using LED<sub>para</sub>), we simply use the full simplification output for each source paragraph. For sentence-level models (ConBART, PG<sub>Dyn</sub>), we first used the alignments to identify which paragraph the source

sentence belongs to, then concatenated their simplification results.

Human judgements were crowdsourced on the MTurk platform. We sourced workers from English speaking countries (AU, CA, GB, IE, NZ, US) and paid them $0.2 USD for each individual evaluation. We ran an initial test ourselves and timed how many evaluations could be completed within an hour. According to this, subjects should earn approximately $18 USD per hour (which is above the minimum wage in all of these countries). The form and instructions presented to human evaluators is shown in Figure 3.

## D Additional Evaluation Results

In Table 7 we provide additional results for popular automatic evaluation metrics that were not included in the main text. Specifically, we include BLEU (Papineni et al., 2002), and full operation-specific scores for SARI. In general, the results are similar to those in Table 3, with $\hat{O} \rightarrow LED_{para}$ and $\hat{O} \rightarrow ConBART$ achieving the best results.

Faithfulness BARTScore is included for clarity rather than being a direct estimation of output quality. It shows how semantically similar system outputs are to their inputs, roughly equating to a measurement of conservativity.

## E Example Simplifications

Figure 4 shows several example simplifications by the $\hat{O} \rightarrow LED_{para}$ system on full documents. Due to licensing constraints imposed by Newsela, we use out-of-domain documents from the WikiLarge dataset here. As these are Wikipedia articles they are quite different in tone than the Newsela articles as well as being much shorter in length. Regardless, we still believe this provides clarity on the types of editing performed by the model.

## F Dynamic Context Algorithm

Algorithm 1 shows the process used to handle dynamic context generation for appropriate models. As each document needs to be simplified autoregressively at the sentence level, we construct batches of sentences with the same index from different documents in order to speed up processing. Note that this could potentially be further optimised (e.g. via parallelism) and merely serves as a reasonable baseline algorithm.
<table><caption>Table 6: Results of multi-task systems on the Newsela-auto test set.</caption><thead><tr><th rowspan="2">System</th><th colspan="4">BARTScore</th><th colspan="3">SMART ↑</th><th>FKGL ↓</th><th>SARI ↑</th><th colspan="2">Length</th></tr><tr><th>Faith.<br>(s → h)</th><th>P ↑<br>(r → h)</th><th>R ↑<br>(h → r)</th><th>F1 ↑</th><th>P</th><th>R</th><th>F1</th><th></th><th></th><th>Tok.</th><th>Sent.</th></tr></thead><tbody><tr><td>Input</td><td>-0.93</td><td>-2.47</td><td>-1.99</td><td>-2.23</td><td>63.2</td><td>62.7</td><td>62.8</td><td>8.44</td><td>20.52</td><td>866.9</td><td>38.6</td></tr><tr><td>Reference</td><td>-1.99</td><td>-0.93</td><td>-0.93</td><td>-0.93</td><td>100</td><td>100</td><td>100</td><td>4.93</td><td>99.99</td><td>671.5</td><td>42.6</td></tr><tr><td>LED<sub>prefix</sub></td><td>-1.83</td><td>-1.72</td><td>-2.00</td><td>-1.86</td><td>73.5</td><td>67.6</td><td>69.8</td><td>4.97</td><td>63.14</td><td>604.0</td><td>38.0</td></tr><tr><td>LED<sub>sep</sub></td><td>-1.82</td><td>-1.80</td><td>-1.88</td><td>-1.84</td><td>72.6</td><td>70.5</td><td>71.1</td><td>5.06</td><td>62.64</td><td>640.4</td><td>40.2</td></tr><tr><td>ConBART<sub>prefix</sub></td><td>-1.96</td><td>-1.62</td><td>-1.60</td><td>-1.61</td><td>80.4</td><td>79.6</td><td>79.9</td><td>4.90</td><td>74.31</td><td>643.6</td><td>41.5</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 3: Submission form used in human evaluation.</figcaption></figure>

<table><caption>Table 7: Extra automatic evaluation results on Newsela-auto. For BARTScore, *s* is the source text and *h* is the hypothesis. Scores significantly higher than PG<sub>Dyn</sub> are denoted with * (*p* < 0.005). Significance was determined with Student's *t*-tests.</caption><thead><tr><th>System</th><th>BARTScore</th><th>BLEU ↑</th><th>SARI ↑</th><th>add</th><th>keep</th><th>delete</th></tr><tr><td></td><td colspan="6">Faith. (s → h)</td></tr></thead><tbody><tr><td>Input</td><td>-0.93</td><td>46.2</td><td>20.5</td><td>0.0</td><td>61.6</td><td>0.0</td></tr><tr><td>Reference</td><td>-1.99</td><td>100</td><td>100</td><td>100</td><td>100</td><td>100</td></tr><tr><td>BART<sub>doc</sub></td><td>-2.48</td><td>31.1</td><td>47.1</td><td>20.4</td><td>55.4</td><td>65.4</td></tr><tr><td>BART<sub>sent</sub></td><td>-1.86</td><td>70.7</td><td>73.0</td><td>55.9</td><td>83.7</td><td>79.5</td></tr><tr><td>BART<sub>para</sub></td><td>-2.11</td><td>68.6</td><td>73.7</td><td>57.8</td><td>82.6</td><td>80.8</td></tr><tr><td>LED<sub>doc</sub></td><td>-1.90</td><td>63.7</td><td>68.7</td><td>52.2</td><td>78.2</td><td>75.7</td></tr><tr><td>LED<sub>para</sub></td><td>-1.86</td><td>74.5*</td><td>76.9*</td><td>64.3*</td><td>85.0</td><td>81.5</td></tr><tr><td>ConBART</td><td>-1.89</td><td>73.7</td><td>75.8</td><td>61.4</td><td>84.9</td><td>81.2</td></tr><tr><td>PG<sub>Dyn</sub> (2023)</td><td>-1.91</td><td>72.4</td><td>75.0</td><td>58.9</td><td>84.8</td><td>81.4</td></tr><tr><td>Ō → ConBART</td><td>-1.92</td><td>76.0*</td><td>78.3*</td><td>64.6*</td><td>86.8*</td><td><strong>83.4</strong></td></tr><tr><td>Ō → BART<sub>para</sub></td><td>-2.05</td><td>71.3</td><td>74.9</td><td>58.5</td><td>84.7</td><td>81.4</td></tr><tr><td>Ō → LED<sub>para</sub></td><td>-1.87</td><td><strong>76.8*</strong></td><td><strong>78.5*</strong></td><td><strong>65.1*</strong></td><td><strong>87.3*</strong></td><td>83.0</td></tr><tr><td>PG<sub>Oracle</sub> (2023)</td><td>-1.93</td><td>78.9*</td><td>80.7*</td><td>65.2*</td><td>89.9*</td><td>87.1*</td></tr><tr><td>O → ConBART</td><td>-1.93</td><td><strong>82.6*</strong></td><td><strong>83.8*</strong></td><td><strong>70.8*</strong></td><td><strong>91.7*</strong></td><td><strong>88.7*</strong></td></tr><tr><td>O → BART<sub>para</sub></td><td>-2.09</td><td>76.1*</td><td>79.7*</td><td>64.1*</td><td>88.7*</td><td>86.3*</td></tr><tr><td>O → LED<sub>para</sub></td><td>-1.90</td><td>81.4*</td><td>82.3*</td><td>69.6*</td><td>90.6*</td><td>86.7*</td></tr></tbody></table>
<figure><img src="image_6.png" /><figcaption>Figure 4: Example simplification outputs for Ō → LED<sub>para</sub>, illustrating both strong and poor performances (a target reading-level of 3 was used for all examples). Input documents are taken from WikiLarge due to licensing constraints around sharing Newsela content. Deletions are **underlined and in red**; rephrasings are **italicised and in green**; splitting points are **highlighted in cyan**; and factual errors are circled.</figcaption></figure>
**Algorithm 1** Generation strategy for systems using a dynamic context mechanism. Inference is performed autoregressively in batches containing 1 sentence per document. At the end of each time step, the simplified sentences are encoded for use within the context of the next step. This naturally extends to the paragraph-level case by replacing sentences with paragraphs.

1: **procedure** DYNAMICGENERATION(test_set)
2:     g ← load_planner()
3:     h ← load_simplifier()
4:     max_idx ← max<sub>C∈test_set</sub> |C|
5:     **for** i ← 1 to max_idx **do**
6:         sents ← {cᵢ | C ∈ test_set}
7:         context ← load_context(sents)
8:         **if** pipeline system **then**
9:             plans ← g(sents, context)
10:         sents ← plans + sents
11:     ▷ Prepend plans to texts
12:     **end if**
13:     context ← update_context(preds)
14:     **end for**
15: **end procedure**
**A** For every submission:

☑ A1. Did you describe the limitations of your work?
10

☑ A2. Did you discuss any potential risks of your work?
10, "generalised target audience"

☑ A3. Do the abstract and introduction summarize the paper's main claims?
1

☑ A4. Have you used AI writing assistants when working on this paper?
Left blank.

**B** ☑ Did you use or create scientific artifacts?

4

☑ B1. Did you cite the creators of artifacts you used?
4

☑ B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
10

☑ B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)?
7, 10

☑ B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it?

*The dataset, Newsela, is a well established dataset that has already been used numerous times within the literature.*

☑ B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.?
4

☑ B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.
4

**C** ☑ Did you run computational experiments?

5, 8

☑ C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used?
8, Appendix A

<footnote>The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.</footnote>
<figure><img src="image_7.png" /></figure>

C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values?

5.2, Appendix A

<figure><img src="image_8.png" /></figure>

C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run?

7, 8, Appendix C

<figure><img src="image_9.png" /></figure>

C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)?

No response.

D

<figure><img src="image_10.png" /></figure>

Did you use human annotators (e.g., crowdworkers) or research with human participants?

6.2, Appendix B

<figure><img src="image_11.png" /></figure>

D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.?

Appendix C

<figure><img src="image_12.png" /></figure>

D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)?

Appendix C

<figure><img src="image_13.png" /></figure>

D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used?

We do not collect any personal data from the MTurk workers, only binary answers to concrete questions pertaining to presented text extracts.

<figure><img src="image_14.png" /></figure>

D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?

Again, no personal data was taken from the human evaluators and therefore there were no obvious ethical risks associated with the evaluation tasks.

<figure><img src="image_15.png" /></figure>

D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?

Appendix C