SlowGuess/NAACL_2022_OCR huggingface dataset:
Model	Training	Newsela-M	Newsela-M (Split)	ASSET
PBERT	SARI	RSplit	PSplit	PBERT	SARI	PBERT	SARI
Hybrid	Newsela-auto	0.39	30.2	0.17	0.42	0.39	31.9	0.43	30.5
BERT	Newsela-auto	0.46	32.2	0.40	0.46	0.47	34.5	0.59	35.2
EditNTS	Newsela-auto	0.49	29.3	0.32	0.45	0.53	30.8	0.54	31.4
MadExP	Newsela-auto	0.43	36.0	0.41	0.48	0.43	37.4	0.59	36.2
BARTN	Newsela-auto	0.54	34.0	0.52	0.48	0.58	37.1	0.64	36.4
BART3	IRSDS3	0.54	25.0	0.31	0.49	0.58	28.8	0.64	34.3
BART4	IRSDS4	0.55	25.3	0.25	0.51	0.58	28.6	0.64	33.7
Ctrl3,3	IRSDS3	0.54	33.4	0.54	0.43	0.58	35.9	0.64	34.1
Ctrl3,4	IRSDS4	0.55	35.6	0.54	0.43	0.59	37.8	0.64	33.8
Ctrl4,4	IRSDS4	0.54	30.4	0.51	0.45	0.59	34.5	0.64	33.5
CtrlOracle	IRSDS4	0.56	37.3	1.00	0.99	0.59	38.6	-	-
Table 3: Comparison with existing systems and baselines. Oracle labels are acquired by applying the same heuristics used in the creation of IRSD $^S$ . Note that the oracle labels for these test sets do not contain identity cases.
System	Fluency	Adequacy	Simplicity	Mean
Ref.	4.65**	3.95**	4.37*	4.32
EditNTS	3.81**	3.83**	3.91**	3.85
MadExP	3.74**	3.52**	3.97**	3.75
BARTN	4.68	4.26**	4.38*	4.44
BART4	4.71	4.74	4.14	4.53
Ctrl4,4	4.77	4.74	4.20	4.57
Table 4: Human evaluation results for selected simplification systems and baselines. Ratings significantly different from  $\mathrm{Ctrl}_{4,4}$  are denoted with  $^* (p <   0.05)$  and \*\*  $(p <   0.01)$ . Significance was determined with a Student's  $t$  -test.
the same examples. However, it shows the diversity of  $\mathrm{IRSD}^S$  does not reduce Newsela-specific performance.
On Newsela test data, using the 3-class classifier  $(\mathrm{Ctrl}_{3,*})$  yields higher SARI and  $R_{Split}$  than the 4-class case. This is likely because identity is never predicted thereby encouraging less conservative simplification on a test set where most examples are simplified (Maddela et al. (2021) excludes all examples with high or low similarity between the input and the reference from the test set).
End-to-End vs Controllable Controllable systems outperform their end-to-end counterpart on all metrics and datasets. In particular, they show a large increase in  $R_{Split}$ , suggesting that explicitly triggering splits via control tokens greatly improves a model's ability to correctly administer splits where needed. Using silver operation labels in CtrlOracle shows universally higher scores than classifier-based pipelines, indicating that there is still room for improvement in terms of classification performance.
Existing Systems Comparative results with existing systems are summarised in Table 3. All of our systems achieve much higher  $P_{BERT}$  scores than any existing system. This suggests that merely using the BART architecture yields much more fluent outputs than other models. MadExP, which receives the highest SARI scores, actually receives much lower  $P_{BERT}$  than almost any other system, indicating that although it may be simplifying the text well, outputs might be less fluent.
$\mathrm{Ctrl}_{3,4}$  achieves the highest scores on New Sala-M, except for being slightly beaten by MadExP on SARI. It is expected to be better than  $\mathrm{Ctrl}_{4,4}$  here as New Sala-M does not include any identical  $\langle C, S \rangle$  pairs and therefore penalises copying inputs.  $\mathrm{Ctrl}_{\text{Oracle}}$  universally outperforms other systems.
On ASSET,  $\mathrm{BART}_N$  actually achieves the highest SARI, closely followed by MadExP. We hypothesise that models trained on  $\mathrm{IRSD}^S$  might achieve lower SARI as the training data includes examples from splitting datasets that do not exhibit any further simplification, leading to more conservative lexical changes when splitting is performed.
Human Evaluation Human evaluation results are shown in Table 4.  $\mathrm{Ctrl}_{4,4}$  scores highest in fluency and overall. Both  $\mathrm{Ctrl}_{4,4}$  and  $\mathrm{BART}_4$  score highest in adequacy.  $\mathrm{BART}_N$  scores highest in simplicity.
All of our systems are rated highly accross all criteria and receive better average scores than even the reference. This clearly highlights the capability of pretrained generative models like BART to produce highly fluent texts. Further, we see that using a classifier trained on our data to predict operation-specific control-tokens further enhances
Output
(1) Complex Simple	An acidic goo from the stomach dissolves the snail's flesh so the worm can re-swallow both stomach and prey. An acidic goo from the stomach dissolves the snail's flesh, and the flatworm can then re-swallow both stomach and snail.
BARTN	[del] The goo dissolves the snail's flesh so the worm can re-swallow both stomach and prey.
BART4	An acidic goo from the stomach dissolves the snail's flesh so the worm can re-swallow both stomach and prey.
Ctrl4,4	An acidic goo from the stomach dissolves the snail's flesh. [split] This allows the worm to re-swallow both stomach and prey.
(2) Complex Simple	Archer's parents were Catholic in Protestant England, which became illegal.
Archer's parents were Catholic. Being Catholic was against the law in England at the time.
BARTN	Archer's parents were Catholic. They did not want the government to interfere with their religion.
BART4	Archer's parents were Catholic in Protestant England. [del]
Ctrl4,4	Archer's parents were Catholic in Protestant England, which became illegal.
(3) Complex Simple	Many people around the world are now furious at an American dentist who paid to go lion hunting in Zimbabwe.
Palmer paid to go lion hunting in Zimbabwe, which is in Africa.
BARTN	People around the world are mad at a dentist. He was paid to go lion hunting in Africa.
BART4	Many people around the world are now furious at an American dentist who paid to go lion hunting in Zimbabwe.
Ctrl4,4	Many people around the world are now furious at an American dentist who paid to go lion hunting in Zimbabwe.
(4) Complex Simple	The Senegalese capital, on the far western edge of Africa, looks nothing like the sleepy port city that Amadou Sy remembers from his youth.
Amadou Sy remembers Dakar as a quiet city.
BARTN	The Senegalese capital [del] does not look like the sleepy port city that Amadou Sy remembers from his youth.
BART4	The Senegalese capital [del] looks nothing like the sleepy port city that Amadou Sy remembers from his youth.
Ctrl4,4	The Senegalese capital is on the far western edge of Africa. [split] It looks nothing like the sleepy port city that Amadou Sy remembers from his youth.
Table 5: Example system outputs illustrating commonly seen patterns. Blue/ bold marks positive changes while red/underlined marks negative changes or errors.
performance in both fluency and simplicity. However, we also believe our training data limits simplicity at times due to examples from pure splitting datasets exhibiting no simplification but for a split.
We believe the relatively low adequacy rating given to the reference can partly be attributed to sentence alignment failures and cases where the  $S$  makes reference to terms mentioned earlier in their article that are not explicit in the  $C$ .
Qualitative Analysis We perform a qualitative analysis of system outputs from the human evaluation to get a better idea of differences between our models. Table 5 illustrates common patterns.
$\mathrm{BART}_N$  regularly produces the most simple output, but often over-simplifies to the point of removing important contextual information (e.g. item 1). It also sometimes fails to maintain the correct meaning of the input (e.g. items 2 and 3).
$\mathrm{BART}_4$  often produces outputs very similar to  $\mathrm{Ctrl}_{4,4}$ , but performs splitting much less regularly (e.g. items 1 and 4) which can uphold structural complexity.
$\mathrm{Ctrl}_{4,4}$  outputs best retain the original meaning of the input. The benefit of having a classifier pre
dict identity cases can be seen in item 2 where the other models end up rephrasing poorly or deleting important information. However, when performing splits, it fails to sufficiently rephrase, often keeping obviously complicated words (e.g. item 4).
In this work we present a new dataset for simplification operation classification and show that it can be used to produce classifiers of much higher accuracy than what has been proposed in existing studies. We show that a controllable system using such a classifier to predict control tokens outperforms end-to-end baselines and existing systems on a range of datasets and receives extremely high ratings in fluency, adequacy and simplicity from human evaluators. However, this system does result in slightly lower simplicity ratings compared to reference texts and a Newsela baseline, suggesting that further improvements can be made to the system or dataset in order to achieve the best possible results across all criteria.
We thank the anonymous reviewers for their feedback. We gratefully acknowledge the support of the French National Research Agency (Gardent; award ANR-20-CHIA-0003, XNLG "Multilingual, Multi-Source Text Generation") and of Facebook AI Research (FAIR) Paris.
Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).
Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 295-305, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoit Sagot, and Lucia Specia. 2020. ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4668-4679, Online. Association for Computational Linguistics.
Fernando Alva-Manchego, Louis Martin, Carolina Scarton, and Lucia Specia. 2019. EASSE: Easier automatic sentence simplification evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 49-54, Hong Kong, China. Association for Computational Linguistics.
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification. Computational Linguistics, pages 1-29.
Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. 2018. Learning to split and rephrase from Wikipedia edit history. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 732-737, Brussels, Belgium. Association for Computational Linguistics.
R. Chandrasekar, Christine Doran, and B. Srinivas. 1996. Motivations and methods for text simplification. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.
Liam Cripwell, Joel Legrand, and Claire Gardent. 2021. Discourse-based sentence splitting. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 261-273, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-Interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3393-3402, Florence, Italy. Association for Computational Linguistics.
Cristina Garbacea, Mengtian Guo, Samuel Carton, and Qiaozhu Mei. 2021. Explainable prediction of text complexity: The missing preliminaries for text simplification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1086-1097, Online. Association for Computational Linguistics.
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943-7960, Online. Association for Computational Linguistics.
Tomoyuki Kajiwara, Hiroshi Matsumoto, and Kazuhide Yamamoto. 2013. Selecting proper lexical paraphrase for children. In Proceedings of the 25th Conference on Computational Linguistics and Speech Processing (ROCLING 2013), pages 59-73, Kaohsiung, Taiwan. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP).
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Junyi Jessy Li and Ani Nenkova. 2015. Detecting content-heavy sentences: A cross-language case study. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1271-1281, Lisbon, Portugal. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Mounica Maddela, Fernando Alva-Manchego, and Wei Xu. 2021. Controllable text simplification with explicit paraphrasing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3536-3553, Online. Association for Computational Linguistics.
Louis Martin, Éric de la Clergerie, Benoit Sagot, and Antoine Bordes. 2020. Controllable sentence simplification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4689-4698, Marseille, France. European Language Resources Association.
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoit Sagot. 2021. Muss: Multilingual unsupervised sentence simplification by mining paraphrases. arXiv preprint arXiv:2005.00352.
Jana M Mason. 1978. Facilitating reading comprehension through text structure manipulation. Center for the Study of Reading Technical Report; no. 092.
Kshitij Mishra, Ankush Soni, Rahul Sharma, and Dipti Sharma. 2014. Exploring the effects of sentence simplification on Hindi to English machine translation system. In Proceedings of the Workshop on Automatic Text Simplification - Methods and Applications in the Multilingual Society (ATS-MA 2014), pages 21-29, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.
Makoto Miwa, Rune Sætre, Yusuke Miyao, and Jun'ichi Tsujii. 2010. Entity-focused sentence simplification for relation extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 788-796, Beijing, China. Coling 2010 Organizing Committee.
Sebastian Nagel. 2016. Cc-news.
Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 435-445, Baltimore, Maryland. Association for Computational Linguistics.
Christina Niklaus, Bernhard Bermeitinger, Siegfried Handschuh, and André Freitas. 2016. A sentence simplification system for improving relation extraction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 170-174, Osaka, Japan. The COLING 2016 Organizing Committee.
Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh. 2019. Transforming complex sentences into a semantic hierarchy. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3415-3427, Florence, Italy. Association for Computational Linguistics.
Daiki Nishihara, Tomoyuki Kajiwara, and Yuki Arase. 2019. Controllable text simplification with lexical constraint loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 260-266, Florence, Italy. Association for Computational Linguistics.
C. Scarton, P. Madhyastha, and L. Specia. 2020. Deciding when, how and for whom to simplify. © 2020 The Author(s) and IOS Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial Licence (http://creativecommons.org/licenses/by-nc/4.0/).
Carolina Scarton and Lucia Specia. 2018. Learning simplifications for specific target audiences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 712-718, Melbourne, Australia. Association for Computational Linguistics.
Sanja Štajner and Maja Popovic. 2016. Can text simplification help machine translation? In Proceedings of the 19th Annual Conference of the European Association for Machine Translation, pages 230-242.
Sandra Williams, Ehud Reiter, and Liesl Osman. 2003. Experiments with discourse-level choices and readability. In Proceedings of the 9th European Workshop on Natural Language Generation (ENLG-2003) at EACL 2003, Budapest, Hungary. Association for Computational Linguistics.
Wei Xu, Chris Callison-Burch, and Courtney Naples. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283-297.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. *Bertscore: Evaluating text generation with BERT*. CoRR, abs/1904.09675.
A Splitting Types
In the paper we distinguish two types of sentence splits, based on the findings of Cripwell et al. (2021). A syntax-split is when the split is licensed by syntactic constructs such as relative clauses, VP or sentence coordinations, gerund or apposite constructions. On the other hand, discourse-splits are licensed by the presence of a discourse relation between two discourse units. Often discourse-splits require additional rewriting in order to fully preserve the discourse semantics which would otherwise be broken by a minimal split operation. Furthermore, they can usually be represented via many equivalent variants.
Table 6 shows an example of these differences. The text in C1 contains a temporal discourse relation marked by and after this which is made explicit in the discourse-split output (S1a) by the adverbial Afterwards. A possible variant exists (S1b) where an inverse adverbial connective (Before this) is used. The seconds tier of the table shows two syntax-split examples, where minimal rephrasing is required.
B Human Annotation
In order to compile the gold-label test set for the classification task we instructed 3 human annotators to assign the labels they considered most appropriate for 400 examples. These annotators were students enrolled in a local NLP master's program and were paid slightly above the minimum wage for their work. Nine items were identified as malformed and thus removed.
Annotations were completed through a web form interface (e.g. in Figure 4). For each of the 400 items, they were presented with the input sentence and required to select one of the four class labels. They were also given the option to flag examples as being malformed or incomprehensible (which we removed from the final set). Prior to their completion of the task, they were given a detailed description of each class along with a range of examples.
For the simplification output evaluation we instructed the same 3 annotators to give their judgements on outputs from 6 systems for 100 inputs randomly sampled from the silver-label test set. Again, this was done via a web form where each input text is provided followed by the outputs from each system (e.g. in Figure 5). Below this, the annotators select 1-5 for each of the outputs on the three quality criteria: fluency, adequacy, and simplicity.
Full text instructions for both human annotation tasks are provided in the supplementary materials.
C Difficult Labelling Examples
The main paper mentions cases where it is difficult to determine a single best rewrite operation. Table 7 shows some common examples of this.
During training of the RoBERTa classification models, we used a learning rate of  $2e^{-5}$ . The network
has 12 hidden layers, a hidden size of 768, and was pretrained with the masked language modeling objective on 160GB of books and web content.
During training of the BART generative models, we used a learning rate of  $3e^{-5}$ . The network has 6 layers in each of the encoder and decoder, a hidden size of 768, and was pretrained to perform reconstruction of corrupted documents on a combination of books and Wikipedia data.
All of our finetuning experiments used a batch size of 32, performed dropout with a rate of 0.1 and early stopping as regularisation measures. All models were trained on a computing grid using 4 Nvidia RTX 2080 Ti GPUs (11GB memory) for an average of 24 hours. For each experiment we set aside  $1\%$  of the training set for validation.
For the generative models, at test time we generate output sequences by performing beam search with a beam size of 5 and restrict output to a maximum length of 128 tokens.
The use of the Newsela corpus is subject to a data sharing agreement from Newsela, Inc. This licence permits the data to be used for non-commercial research purposes.
E System Outputs
Table 8 contains example system outputs not included in the main paper which illustrate commonly seen patterns across systems.
9These examples are taken directly from Cripwell et al. (2021)
C1. The Masovians were caught by surprise, since virtually without any defense the capital, Plock, fell and after this Mindaugas crossed the Vistula river and captured the fortress of Jazdow.
S1a. The Masovians were caught by surprise, since virtually without any defense the capital, Plock, fell. Afterwards, Mindaugas crossed the Vistula river and captured the fortress of Jazdów.
S1b. Mindaugas crossed the Vistula river and captured the fortress of Jazdow. Before this, the Masovians were caught by surprise, since virtually without any defense the capital, Plock, fell.
C2. He settled in London, devoting himself chiefly to practical teaching.
S2. He settled in London. He devoted himself chiefly to practical teaching.
C3. It was a time to go back to nature, and the plastic flamingo quickly became the prototype of bad taste and anti-nature.
S3. It was a time to go back to nature. The plastic flamingo quickly became the prototype of bad taste and anti-nature.
Table 6: An example of discourse- (1) vs. syntax-based (2) sentence splitting.
Figure 4: Section of annotation form used for gold-label classification test set creation.
Figure 5: Section of annotation form used for human simplification evaluation.
C1.	He served as Mayor of The Hague from 2008 to 2017; he then took two acting positions in Drenthe and Amsterdam.
rephrase syntax-split	He was Mayor of The Hague from 2008 to 2017 then took two acting positions in Drenthe and Amsterdam. He served as Mayor of The Hague from 2008 to 2017. He then took two acting positions in Drenthe and Amsterdam.
discourse-split	He served as Mayor of The Hague from 2008 to 2017. Later, he took two acting positions in Drenthe and Amsterdam.
C2.	A bus stop is a designated place where buses stop for passengers to get on and off the bus.
identity rephrase	A bus stop is a designated place where buses stop for passengers to get on and off the bus. A bus stop is a place where buses stop for passengers.
C3.	He led Villa to victory in the inaugural League Cup in 1961 but was then sacked in 1964 on grounds of ill health.
syntax-split	He led Villa to victory in the inaugural League Cup in 1961. He was sacked in 1964 on grounds of ill health.
discourse-split	He led Villa to victory in the inaugural League Cup in 1961. However, he was sacked in 1964 on grounds of ill health.