SlowGuess/NAACL_2022_OCR huggingface dataset:
Liam Cripwell  
Université de Lorraine  
CNRS/LORIA  
liam.cripwell@loria.fr
Joël Legrand  
Université de Lorraine  
Centrale Supélec  
CNRS/LORIA  
joel.legrand@inria.fr
Claire Gardent  
CNRS/LORIA  
Université de Lorraine  
claire.gardent@loria.fr
Different types of transformations have been used to model sentence simplification ranging from mainly local operations such as phrasal or lexical rewriting, deletion and re-ordering to the more global affecting the whole input sentence such as sentence rephrasing, copying and splitting. In this paper, we propose a novel approach to sentence simplification which encompasses four global operations: whether to rephrase or copy and whether to split based on syntactic or discourse structure. We create a novel dataset that can be used to train highly accurate classification systems for these four operations. We propose a controllable-simplification model that tailors simplifications to these operations and show that it outperforms both end-to-end, non-controllable approaches and previous controllable approaches.
Sentence simplification is a text generation task where a sentence is transformed into a simpler version of itself while preserving its core meaning. Transformations can involve several different rewrite operations such as word substitutions (lexical paraphrasing), structural modifications (e.g. sentence splitting or syntactic paraphrasing), and deletion.
Sentence simplification has been shown to aid reader comprehension (Mason, 1978; Williams et al., 2003; Kajiwara et al., 2013) and be a useful preprocessing step for downstream NLP tasks such as relation extraction (Miwa et al., 2010; Niklaus et al., 2016) and machine translation (Chandrasekar et al., 1996; Mishra et al., 2014; Li and Nenkova, 2015; Mishra et al., 2014; Štajner and Popovic, 2016).
Modern systems are data-driven, learning to perform transformations from parallel corpora of complex-simple  $\langle C,S\rangle$  pairs. Although many different approaches have been attempted in the past,
including statistical machine translation (SMT)-based methods, nearly all systems proposed in recent years follow a neural sequence-to-sequence approach. As these systems are trained in an end-to-end manner they are able to perform lexical and syntactic operations in combination and produce outputs with very high fluency.
However, given the black-box nature of these end-to-end systems, they are forced to rely on imperfect training corpora to implicitly learn rewrite operations, many of which occur infrequently (Jiang et al., 2020). As a result, neural end-to-end systems have been found to be overly conservative, often making no changes to the original text or being limited to the paraphrasing of short word sequences (Alva-Manchego et al., 2017; Maddela et al., 2021). In addition, these systems provide limited capacity for controllability and are unable to express alternative variants of the simplified text (Alva-Manchego et al., 2017; Cripwell et al., 2021).
In response, attempts have been made to produce controllable simplification systems that can constrain either the shape (length, amount of paraphrasing, lexical and syntactic complexity) of the output (Martin et al., 2020) or the type of transformation to be applied (e.g., copy, split, merge, rewrite, etc.) (Scarton and Specia, 2018; Dong et al., 2019; Scarton et al., 2020; Garbacea et al., 2021; Maddela et al., 2021).
In this work we propose a novel approach to sentence simplification which encompasses four global operations: whether to copy the input sentence (no simplification needed), rephrase it, split it based on syntax, or split it based on discourse structure. We create a novel dataset that can be used to train highly accurate classification systems for these four operations and propose a controllable-simplification model that tailors simplification to them. We compare our model with various alternatives and previous work, using both quantitative metrics and human evaluation, and show that our
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2091 - 2103 July 10-15, 2022 ©2022 Association for Computational Linguistics
model outperforms them. We also provide a qualitative analysis of the differences between the best models.
2.1 Controllable Simplification
Scarton and Specia (2018), Nishihara et al. (2019) and Scarton et al. (2020) focus on tailoring outputs to specific reader groups based on the Newsela corpus (Xu et al., 2015), a popular simplification dataset which provides versions of news articles written for audiences of different reading levels. These works propose systems that adjust their simplifications to match one of these reading levels.
Martin et al. (2020) introduce a wider array of control attributes concerning grammatical features of the desired text such as compression level, amount of paraphrasing, and lexical and syntactic complexity.
Most recently, Maddela et al. (2021) propose a system that first uses a rule-based component (Niklaus et al., 2019) to generate candidates that have undergone splitting and deletion, before ranking them and sending the top  $n$  to a neural paraphasing model. Tunable settings in both components provide control over how much of the input is changed and whether to favour deletion or splitting. Their system received higher fluency and simplicity scores from human annotators compared to existing works.
However, at inference time, these methods all require the model to be explicitly informed about which reader level to cater to or which specific grammatical features or rewrite operations to prioritise. In constraint, we develop an approach that can not only be tuned manually, but can also operate in an end-to-end manner by inferring tunable parameters from the input.
2.2 Operation Classification
Alva-Manchego et al. (2017) and Dong et al. (2019) consider sentence simplification as a sequence-labeling problem, proposing systems that predict rewrite operations at the token-level before realising them downstream. Alva-Manchego et al. (2017) showed gains over previous approaches in terms of simplicity, but at the cost of fluency and meaning preservation. Dong et al. (2019) appears to resolve this trade off by introducing an enhanced interpreter that better constructs the resulting text.
Several existing works have attempted to use a classifier to determine which rewrite operation should be performed on an input at the sentence-level. Applying a sentence-level binary classifier as an initial step to predict whether simplification should be performed has been found to yield improved SARI results, reducing conservatism and spurious transformations (Scarton et al., 2020; Garbacea et al., 2021).
Multi-class systems have been explored with limited results. Scarton and Specia (2018) and Scarton et al. (2020) predict one of 4 operations (identical, elaboration, split, and merge) and feed this into an end-to-end model alongside the  $C$  as either a control token or one-hot vector. While Scarton and Specia (2018) fail to produce an accurate classifier or show any improvement over baselines, Scarton et al. (2020) show some gains in SARI when using predicted operation labels. However, their best classifier only yields an accuracy of  $70\%$ .
In the multi-class setting, models tend to struggle to accurately predict identity cases. We believe this is partially due to the training data used. All existing works use  $C_s$  from identical  $\langle C, S \rangle$  pairs as training examples for this class, either alone or alongside standard  $S_s$ . The assumption here is that these pairs are identical because the  $C$  is already simplified. We will show that it is much more likely these items are unsimplified noise from the distribution of  $C_s$  and that excluding them from training data can dramatically improve accuracy.
We extend upon these sentence-level classification approaches by redefining the set of operations, creating comprehensive training and test data, and ultimately producing a classifier with much higher accuracy. We show that a pipeline approach which first predicts a rewrite operation outperforms existing end-to-end and controllable systems.
3 Operation Classification
We consider 4 operation types: identity, rephrase, syntax-split, and discourse-split. The identity and rephrase classes are equivalent to identical and elaboration from Scarton et al. (2020). In contrast, we split the split class into two distinct groups to capture further nuances of sentence splitting, as was explored in Cripwell et al. (2021).
Syntax-split indicates that a split should be performed based on a syntactic construct, whereas discourse-split indicates that a split should be performed based on a discourse relation. Examples of
these can be found in Appendix A.
As we focus on single sentence simplification, we exclude the merge class used in (Scarton and Specia, 2018; Scarton et al., 2020).
3.1 Training Data
We construct training data for a simplification operation classifier by combining subsets of existing English datasets. We consider simplification datasets Wiki-auto, Newsela-auto  $^{1}$  (Jiang et al., 2020), and MUSS (Martin et al., 2021) as well as dedicated splitting datasets WikiSplit (Botha et al., 2018) and D-CCNews (Cripwell et al., 2021).
Wiki-auto and Newsela-auto are automatically aligned  $\langle C, S \rangle$  pairs extracted from Wikipedia and Newsela, respectively. MUSS contains 2.7M pairs mined from Common crawl web data which are estimated paraphrases based on embedding distance. WikiSplit contains 1M split pairs mined from Wikipedia edit history, while D-CCNews contains discourse-split pairs mined from the CCNews corpus (Nagel, 2016). D-CCNews has two subsets: D-CCNews-C which contains single  $C_s$ , and D-CCNews-S which contains pairs of organic  $S_s$  and synthetic  $C_s$ . We include samples from both subsets. Table 1 provides a breakdown of the inclusions from each source.
We heuristically assign silver operation labels to sentences from these datasets as follows:
identity:  $S_{s}$  from the Wiki-auto and Newsela-auto rephrase and syntax-split sets. We can be fairly confident that  $S_{s}$  from known simplification datasets are sufficiently simplified.
rephrase:  $C_s$  from MUSS, Wiki-auto and Newsela-auto where there is no split in the output  $S$  and Levenshtein similarity between the  $C$  and  $S$  is less than 1 standard deviation above the mean  $(< 0.92)$ . This is to exclude near-identical pairs.
syntax-split:  $C_s$  from WikiSplit, MUSS, Wikiauto and Newsela-auto whose  $S$  exhibits a split and does not contain an identifiable discourse marker.
discourse-split:  $C_s$  from all datasets whose  $S$  contains a split and a discourse adverbial.
We call the resulting dataset  $\mathbf{IRSD}_4^C$ . We also
consider a 3-class subset which excludes the identity class  $(\mathbf{IRSD}_3^C)$  to explore how results change when models are trained to always simplify.
3.2 Test Data
We use two datasets for evaluation. A random sample of  $1\%$  of the training data is set aside as a large (34K examples) silver test set. We also create a smaller gold test set by randomly sampling 100 items from each of the 4 classes in our silver test set and presenting them to 3 annotators instructed to select the most appropriate operation with which to simplify the text. Further details of the annotation process are in Appendix B.
We approved all annotations that received a majority label agreement and manually adjudicated cases where all annotators disagreed (11%). The mean Cohen's Kappa agreement score between annotators is 0.246, illustrating the difficulty of this task. In many cases, several operations could feasibly apply, and so assigning a single correct label is not always a perfect solution. Appendix C lists some examples of this.
3.3 Classification Model
We fine-tune pretrained RoBERTa models (Liu et al., 2019) with classification heads on  $\mathrm{IRSD}_4^C$  and  $\mathrm{IRSD}_3^C$ . Further training details are provided in Appendix D.
Results on Silver Test Data. Results can be seen in Figure 1. Accuracy on the silver test set (98%) is much higher than previous works: Scarton and Specia (2018) and Scarton et al. (2020) achieve mean accuracies of  $51\%$  and  $70\%$  for a similar 4-class task. Garbacea et al. (2021), who only train a binary (simp, no-simp) classifier achieve  $81\%$  accuracy.
Notably, the accuracy for the identity class is much higher than the  $59\%$  achieved by Scarton et al. (2020). This is perhaps in part due to our exclusion of  $C_s$  from identical  $\langle C, S \rangle$  pairs in the identity training subset. We explored this hypothesis by using a test set containing  $C_s$  from identical pairs alongside the existing identity examples.
Figure 2 shows that doing so reduces performance on the identity class dramatically; the model only classifies  $9.8\%$  of these  $C_s$  as identity and  $82.4\%$  of them as rephrase. This suggests that
upon request after receiving a licence.
4We use the pretrained roberta-base model available at https://huggingface.co/roberta-base.
<sup>1</sup>We specifically use the aligned pairs used for simplification experiments in Jiang et al. (2020), which excludes identical pairs and those of readability levels 0-1, 1-2, and 2-3.
$^{2}$ D-CCNews is down-sampled to keep classes similar in size.
3Our data and code is available at https://github. com/liamcripwell/control_simp.Newsela data is excluded, subject to their terms of use, but can be provided
Class	WikiSplit	MUSS	Wiki-auto	Source
Newsela-auto	D-CCNews-C	D-CCNews-S	Total
Identity (0)	-	-	513,436	338,798	-	-	852,234
Rephrase (1)	-	461,702	366,382	171,508	-	-	999,592
Syntax-Split (2)	633,900	53,008	68,357	88,669	-	-	843,934
Discourse-Split (3)	269,666	1,002	5,277	2,060	250,062	249,958	778,025
Total	903,566	515,712	953,452	601,035	250,062	249,958	3,473,785
Table 1: Data source distributions for each operation class in  ${\mathrm{{IRSD}}}_{4}^{C}$  .
(a) 4-class
(b) 3class
Figure 1: Normalised confusion matrix of (a) the four-class classifier and (b) the three-class classifier, evaluated on the silver-label test set.
these examples are from a distribution more similar to the rephrase examples and are possibly complex sentences themselves that have not been fully simplified in the source data. This observation validates our decision to exclude them.
Figure 2: Normalised confusion matrix for the 4-class operation classifier, evaluated on the silver-label test set containing  $C_s$  from identical  $\langle C, S \rangle$  pairs in the identity class.
Results on Gold Test Data. As shown in Figure 3, classification accuracy on the gold test set is considerably lower than on the silver data. Identity examples are often predicted as rephrase; syntax-split often as discourse-split; and rephrase examples regularly receive predictions across all four classes.
However, this aligns with our observations with respect to manual labelling difficulties. Often it is
not immediately clear whether a particular example should be ignored or slightly rephrased. Similarly, it often seems plausible for either type of split to be performed. Rephrase is the broadest of the four classes, and so cases where any one of the other three classes could also apply should be expected.
Despite being lower than on the silver examples, we believe these results show a strong signal of performance, with common mistakes being analogous to difficulties encountered by human annotators.
4 Sentence Simplification
Training Data. For the sentence simplification task we use a modified version of  $\mathrm{IRSD}^C$  which additionally includes target simplifications, i.e.  $\langle C,o,S\rangle$  triples. We refer to this as  $\mathbf{IRSD}_4^S$  and its 3-class subset as  $\mathbf{IRSD}_3^S$ .
For the identity class, we take all inputs from  $\mathbf{IRSD}_4^S$  labelled as identity and map them to themselves. For rephrase and syntax-split, we take the rephrase and syntax-split inputs and map them to their simplifications in the source datasets. We do the same for discourse-split, but, as D-CCNews-C instances do not contain simplifications, we replace them with additional  $\langle C,S\rangle$  pairs from D-CCNews-S.
(b) 3-class
Figure 3: Normalised confusion matrix of (a) the four-class classifier and (b) the three-class classifier, evaluated on the human-annotated test set.
Test Data. We first train and test our systems on  $\mathrm{IRSD}_{3/4}^{S}$  and Newsela-auto.
Next, in order to compare with past works, we perform evaluation on the Newsela-auto test set introduced by Maddela et al. (2021). It contains 24,035 rephrases, 9,208 syntax-splits, and 148 discourse-splits. We refer to this as Newsela-M and also include results on the subset with split in their reference  $S$  (Newsela-M (Split)). We use this test set so we can leverage pre-existing system outputs from past works for comparison.
Additionally, we evaluate on the ASSET corpus (Alva-Manchego et al., 2020) which is a much smaller test set (359 examples) containing 10 human-written references per input. All test examples have at least one rephrase reference, 248 have at least one syntax-split reference, 12 have at least one discourse-split reference, and 0 have an identity reference.
Existing Systems We consider a number of past works for comparison: (i) Hybrid (Narayan and Gardent, 2014), an older system with a probabilistic splitting component combined with an MT-based lexical paraphraseer; (ii) BERT, pretrained encoder-decoder transformer  $(\mathrm{BERT}_{\mathrm{base}})$  finetuned on simplification which achieved state-of-the-art performance (Jiang et al., 2020); (iii) EditNTS (Dong et al., 2019), a recent model using operation prediction; and (iv) MadExP (Maddela et al., 2021), current state-of-the-art controllable system. We exclude other systems which require conditioning on specific reading levels.
Baseline End-to-End Model. We include end-to-end baselines that are trained to perform  $C \to S$  with no additional information. These are used to gauge whether our controllable models are competitive with a black-box approach. We use the BART architecture (Lewis et al., 2020) and fine-tune a pretrained model with a language-modelling head on  $\langle C, S \rangle$  pairs from 4 distinct datasets: IRSD $_4^S$  (BART $_4$ ), IRSD $_3^S$  (BART $_3$ ), Wiki-auto (BART $_W$ ) and Newsela-auto (BART $_N$ ).
Controllable Model. Next, we train an end-to-end generative model to perform  $\langle C,o\rangle \to S$  where  $o$  is an operation label. The  $o$  is used as a control token prepended to the input sequence for  $C$ . We use the same BART architecture as our end-to-end baselines.
From this model, we construct several systems: (i) an oracle baseline  $(\mathbf{Ctrl}_{Oracle})$  taking the silver operation label and performing generation as an end-to-end task; (ii) a pipeline system using a classifier to predict  $o$  before running the generative model.
We refer to different configurations as  $\mathbf{C}\mathbf{t}\mathbf{r}\mathbf{l}_{i,j}$ , where  $i$  is the number of classes the classifier is trained on and  $j$  is the number of classes the generator is trained on. E.g.  $\mathrm{Ctrl}_{3,4}$  uses a classifier trained on  $\mathrm{IRSD}_3^C$  and a generator trained on  $\mathrm{IRSD}_4^S$ . We expect that using the 4-class classifier will result in more conservative outputs. Using the 3-class generator could allow more model capacity to focus on simplification. Conversely, the extra training data used by the 4-class generator could improve general performance.
<sup>6</sup>We use the pretrained facebook/bart-base model available at https://huggingface.co/facebook/bart-base.
For  $\mathrm{Ctrl}_{4,3}$  any inputs classified as ignore are returned without being passed to the generator.
5We use system outputs from versions of all of these models that have been trained on Newsela-Auto.
Model	IRSD4S	IRSD3S	Newsela-auto
PBERT	SARI	RSplit	PSplit	PBERT	SARI	PBERT	SARI	RSplit	PSplit
Input	0.83	27.4	0.00	0.00	0.77	25.4	0.53	15.9	0.00	0.00
Reference	0.99	80.1	1.00	1.00	0.99	95.3	0.99	94.1	1.00	1.00
End-to-End Models
BARTW	0.81	35.0	0.18	0.85	0.76	34.7	0.54	24.6	0.05	0.64
BARTN	0.77	38.9	0.64	0.81	0.74	42.0	0.56	35.9	0.46	0.59
BART3	0.85	50.6	0.82	0.94	0.81	54.9	0.55	27.3	0.27	0.59
BART4	0.86	51.2	0.85	0.93	0.82	55.7	0.56	26.9	0.21	0.62
Controllable Models with predicted control-tokens
Ctrl3,3	0.83	50.6	0.99	0.93	0.82	58.5	0.54	33.6	0.48	0.54
Ctrl3,4	0.84	51.2	0.99	0.93	0.83	59.4	0.55	35.9	0.49	0.54
Ctrl4,3	0.86	52.9	0.99	0.98	0.83	59.5	0.55	30.7	0.45	0.56
Ctrl4,4	0.87	55.1	0.99	0.98	0.83	60.4	0.56	32.4	0.45	0.56
Controllable Models with Oracle control-tokens
CtrlOracle	0.87	55.5	1.00	1.00	0.83	60.7	0.57	38.3	0.99	1.00
Table 2: Automatic sentence simplification results on the IRSD  ${}_{4}^{S}$  ,IRSD  ${}_{3}^{S}$  and Newsela-auto test sets.
The most common evaluation metrics used in text simplification are BLEU and SARI, with SARI being viewed as the more effective at describing simplicity. Both focus primarily on lexical similarities between the reference and system output without consideration for structural simplification.
A recent meta-analysis (Alva-Manchego et al., 2021) of automated text simplification evaluation shows that the precision-based BERTScore  $(P_{BERT})$  (Zhang et al., 2019) is most highly correlated with human judgements. As  $P_{BERT}$  is very effective at identifying low quality simplifications, the authors recommend using it as a primary test of quality before referring to other metrics like SARI.
We report  $P_{BERT}$  and SARI as our primary metrics $^{8}$  and also use the split recall  $(R_{Split})$  to evaluate how often the model performs splitting in known cases. We value recall over precision as it gives a better indication of whether a model regularly performs splits, but have also included the precision  $(P_{Split})$  for clarity.
We perform a human evaluation of simplification systems by having 3 annotators evaluate outputs. In order to consider a range of structurally diverse examples we use our classifier to label the Newelam test set with predicted operations and randomly
select 25 from each of the 4 classes (further details in Appendix B). We presented the annotators with the input  $C$  from each  $\langle C, S \rangle$  pair alongside the reference  $S$  and outputs from selected systems. Judgements are made with respect to 3 criteria: fluency, adequacy, and simplicity.
Fluency refers to the grammaticality of the output; adequacy measures meaning preservation with respect to the input; and simplicity measures the overall simplicity of the result. We followed standard practice by having these criteria judged on a 1-5 Likert scale and averaging the results. For simplicity, we advised workers that a high score can be given to an output identical to the input if there is little to no obvious changes that would make the sentence simpler.
We consider the following systems for comparison: EditNTS, MadExp, BART $_N$ , BART $_4$ , and Ctrl $_{4,4}$ . This allows us to compare our systems to strong recent works and examine the effect of (i) using IRSD $^S$  vs Newsela training data and (ii) using our controllable model vs an end-to-end approach.
Automatic evaluation results are shown in Table 2.
IRSD $^S$  vs Other Data Models trained with IRSD $^S$  (BART $_{3/4}$  and Ctrl $_{*,*}$ ) greatly outperform those trained on other datasets (BART $_{N/W}$ ) across every metric on the IRSD $^S$  test sets. On the Newsela test set, IRSD $^S$  models perform at least as well as BART $_N$ . This is unsurprising as IRSD $^S$  is much larger than Newsela and contains many of
8The EASSE python library (Alva-Manchego et al., 2019) is used for calculation.