SlowGuess/ACL_2020_OCR huggingface dataset:
We evaluate our model with a different subset of operations, i.e., removal (RM), extraction (EX), reordering (RO), and lexical substitution (LS). In our experiments, we test the following variants: RM+EX, RM+EX+LS, RM+EX+RO, and RM+EX+LS+RO.
Tables 1 and 2 present the results of the automatic evaluation on the Newsela and WikiLarge datasets, respectively.
We use the SARI metric (Xu et al., 2016) to measure the simplicity of the generated sentences. SARI computes the arithmetic mean of the  $n$ -gram F1 scores of three rewrite operations: adding, deleting, and keeping. The individual F1-scores of these operations are reported in the columns "Add," "Delete," and "Keep."
We also compute the BLEU score (Papineni et al., 2002) to measure the closeness between a candidate and a reference. Xu et al. (2016) and Sulem et al. (2018) show that BLEU correlates with human judgement on fluency and meaning preservation for text simplification.4
4This does not hold when sentence splitting is involved. In our datasets, however, sentence splitting is rare, for example,  $0.18\%$  in the Newsela validation set).
7923
Method	SARI↑	Add↑	Delete↑	Keep↑	BLEU↑	FKGL↓	Len
Baselines
Complex	27.87	-	-	-	99.39	-	22.61
Supervised Methods
PBMT-R	38.56	5.73	36.93	73.02	81.09	8.33	22.35
Hybrid	31.40	1.84	45.48	46.87	48.67	4.56	13.38
EncDecA	35.66	2.99	28.96	75.02	89.03	8.42	21.26
Dress	37.08	2.94	43.15	65.15	77.41	6.59	16.14
Dress-Ls	37.27	2.81	42.22	66.77	80.44	6.62	16.39
Edit-NTS	38.23	3.36	39.15	72.13	86.69	7.30	18.87
EntPar	37.45	-	-	-	81.49	7.41	-
Access	41.87	7.28	45.79	72.53	75.46	7.22	22.27
Models using external knowledge base
SBMT-SARI	39.96	5.96	41.42	72.52	73.03	7.29	23.44
DMass	40.45	5.72	42.23	73.41	-	7.79	-
Unsupervised Methods
UNMT	35.89	1.94	37.68	68.04	70.61	8.23	21.85
UNTS	37.20	1.50	41.27	68.81	74.02	7.84	19.05
RM+EX	36.46	1.68	35.17	72.54	88.90	6.47	18.62
RM+EX+LS	37.85	2.31	43.65	67.59	73.62	6.30	18.45
RM+EX+RO	36.54	1.73	36.10	71.79	85.07	6.89	19.24
RM+EX+LS+RO	37.58	2.30	43.97	66.46	70.15	6.69	19.54
Table 2: Results on the WikiLarge dataset.  ${}^{ \uparrow  }$  The higher,the better.  ${}^{ \downarrow  }$  The lower,the better.
In addition, we include a few intrinsic measures (without reference) to evaluate the quality of a candidate sentence: the Flesch-Kincaid grade level (FKGL) evaluating the ease of reading, as well as the average length of the sentence.
A few recent text simplification studies (Dong et al., 2019; Kriz et al., 2019) did not use BLEU for evaluation, noticing that the complex sentence itself achieves a high BLEU score (albeit a low SARI score), since the complex sentence is indeed fluent and preserves meaning. This is also shown by our Complex baseline.
For the Newsela dataset, however, we notice that the major contribution to the SARI score is from the deletion operation. By analyzing previous work such as EntPar, we find that it reduces the sentence length to a large extent, and achieves high SARI due to the extremely high F1 score of "Delete." However, its BLEU score is low, showing the lack of fluency and meaning. This is also seen from the high SARI of (Reduce-250) in Table 1. Ideally, we want both high SARI and high BLEU, and thus, we calculate the geometric mean (GM) of them as the main evaluation metric for the Newsela dataset.
On the other hand, this is not the case for WikiLarge, since none of the models can achieve high SARI by using only one operation among "Add,"
"Delete," and "Keep." Moreover, the complex sentence itself yields an almost perfect BLEU score (partially due to the multi-reference nature of WikiiLarge). Thus, we do not use GM, and for this dataset, SARI is our main evaluation metric.
Overall results on Newsela. Table 1 shows the results on Newsela. By default (without  $\dagger$ ), validation is performed using the GM score. Still, our unsupervised text simplification achieves a SARI score around 26-27, outperforming quite a few supervised methods. Further, we experiment with SARI-based validation (denoted by  $\dagger$ ), following the setting of most previous work (Dong et al., 2019; Guo et al., 2018). We achieve 30.44 SARI, which is competitive with state-of-the-art supervised methods.
Our model also achieves high BLEU scores. As seen, all our variants, if validated by GM (without  $\dagger$ ), outperform competing methods in BLEU. One of the reasons is that our model performs text simplification by making edits on the original sentence instead of rewriting it from scratch.
In terms of the geometric mean (GM), our unsupervised approach outperforms all previous work, showing a good balance between simplicity and content preservation. The readability of our generated sentences is further confirmed by the intrinsic FKGL score.
7924
Method	SARI↑	Add↑	Delete↑	Keep↑	BLEU↑	GM↑	FKGL↓	Len
RM+EX+LS+RO	27.11	2.40	71.26	7.67	26.21	26.66	3.12	12.81
- SLOR	27.63	2.22	73.20	7.49	24.14	25.83	2.61	12.37
- syntax-awareness	26.91	2.16	71.19	7.39	24.98	25.93	3.65	12.76
Table 3: Ablation test of the SLOR score based on syntax-aware language modeling.
Value	SARI↑	BLEU↑	GM↑	FRE↑	Len
Effect of threshold rop
1.0	29.20	21.69	25.17	83.75	11.75
1.1	28.38	23.59	25.87	82.83	12.17
1.2	27.45	25.54	26.48	81.98	12.62
1.3	26.60	26.47	26.53	81.47	13.07
Effect of weight α for feslor
0.75	27.04	25.75	26.39	83.46	12.46
1.25	26.91	25.96	26.43	81.26	12.96
1.50	26.74	25.20	25.96	80.94	13.06
2.0	26.83	24.29	25.53	80.11	13.15
Effect of weight β for fre
0.5	26.42	25.53	25.97	78.61	13.20
1.5	27.38	26.04	26.70	84.31	12.58
2.0	27.83	25.27	26.52	87.03	12.26
3.0	28.29	23.69	26.52	90.34	11.91
Effect of weight γ for 1/flen
0.5	24.54	25.06	24.80	80.49	14.55
2.0	29.00	21.65	25.06	82.69	10.93
3.0	29.93	19.05	23.88	82.20	10.09
4.0	30.44	17.36	22.99	80.86	9.61
Effect of weight δ for fentity
0.5	27.81	24.68	26.20	83.6	12.01
2.0	25.44	24.63	25.03	79.36	14.28
Table 4: Analysis of the threshold value of the stopping criteria and relative weights in the scoring function.
Overall results on WikiLarge. For the WikiLarge experiments in Table 2, we perform validation on SARI, which is the main metric in this experiment. Our model outperforms existing unsupervised methods, and is also competitive with state-of-the-art supervised methods.
We observe that lexical simplification (LS) is important in this dataset, as its improvement is large compared with the Newsela experiment in Table 1. Additionally, reordering (RO) does not improve performance, as it is known that WikiLarge does not focus on syntactic simplification (Xu et al., 2016). The best performance for this experiment is obtained by the  $\mathrm{RM + EX + LS}$  model.
4.5 Controllability
We now perform a detailed analysis of the scoring function described in Section 3.2 to understand the effect on different aspects of simplification. We use the  $\mathrm{RM + EX + LS + RO}$  variant and the Newsela corpus as the testbed.
The SLOR score with syntax-aware LM. We
analyze our syntax-aware SLOR score in the search objective. First, we remove the SLOR score and use the standard sentence probability. We observe that SLOR helps preserve rare words, which may be entities. As a result, the readability score (FKGL) becomes better (i.e., lower), but the BLEU score decreases. We then evaluate the importance of using a structural LM instead of a standard LM. We see a decrease in both SARI and BLEU scores. In both cases, the GM score decreases.
Threshold values and relative weights. Table 4 analyzes the effect of the hyperparameters of our model, namely, the threshold in the stopping criteria and the relative weights in the scoring function.
As discussed in Section 3.4, we use a threshold as the stopping criteria for our iterative search algorithm. For each operation, we require that a new candidate should be better than the previous iteration by a multiplicative threshold  $r_{\mathrm{op}}$  in Equation (3). In this analysis, we set the same threshold for all operations for simplicity. As seen in Table 4, increasing the threshold leads to better meaning preservation since the model is more conservative (making fewer edits). This is shown by the higher BLEU and lower SARI scores.
Regarding the weights for each individual scoring function, we find that increasing the weight  $\beta$  for the FRE readability score makes sentences shorter, more readable, and thus simpler. This is also indicated by higher SARI values. When sentences are rewarded for being short (with large  $\gamma$ ), SARI increases but BLEU decreases, showing less meaning preservation. The readability scores initially increase with the reduction in length, but then decrease. Finally, if we increase the weight  $\delta$  for the entity score, the sentences become longer and more complex since the model is penalized more for deleting entities.
In summary, the above analysis shows the controllability of our approach in terms of different simplification aspects, such as simplicity, meaning preservation, and readability.
7925
4.6 Human Evaluation
We conducted a human evaluation on the Newsela dataset since automated metrics may be insufficient for evaluating text generation. We chose 30 sentences from the test set for annotation and considered a subset of baselines. For our model variants, we chose  $\mathrm{RM + EX + LS + RO}$ , considering both validation settings (GM and SARI).
We followed the evaluation setup in Dong et al. (2019), and measure the adequacy (How much meaning from the original sentence is preserved?), simplicity (Is the output simper than the original sentence?), and fluency (Is the output grammatical?) on a five-point Likert scale. We recruited three volunteers, one native English speaker and two non-native fluent English speakers. Each of the volunteer was given 30 sentences from different models (and references) in a randomized order. Additionally, we asked the volunteers to measure the number of instances where models produce incorrect details or generate text that is not implied by the original sentence. We did this because neural models are known to hallucinate information (Rohrbach et al., 2018). We report the average count of false information per sentence, denoted as FI.
We observe that our model  $\mathrm{RM + EX + LS + RO}$  (when validated by GM) performs better than Hybrid, a combination of PBMT and discourse representation structures, in all aspects. It also performs competitively with remaining supervised NMT models.
For adequacy and fluency, Dress-Ls performs the best since it produces relatively longer sentences. For simplicity, S2S-A11-FA performs the best since it produces shorter sentences. Thus, a balance is needed between these three measures. As seen,  $\mathrm{RM + EX + LS + RO}$  ranks second in terms of the average score in the list (reference excluded). The human evaluation confirms the effectiveness of our unsupervised text simplification, even when compared with supervised methods.
We also compare our model variants  $\mathrm{RM + EX + LS + RO}$  (validated by GM) and  $\mathrm{RM + EX + LS + RO^{\dagger}}$  (validated by SARI). As expected, the latter generates shorter sentences, performing better in simplicity but worse in adequacy and fluency.
Regarding false information (FI), we observe that previous neural models tend to generate more false information, possibly due to the vagueness in
Method	A↑	S↑	F↑	Avg↑	FI↓
Hybrid	2.63	2.74	2.39	2.59	0.03
Dress-Ls	3.29	3.05	4.11	3.48	0.2
EntPar	1.92	2.97	3.16	2.68	0.47
S2S-All-FA	2.25	3.24	3.90	3.13	0.3
Edit-NTS	2.37	3.17	3.73	3.09	0.23
RM+EX+LS+RO	2.97	3.09	3.78	3.28	0.03
RM+EX+LS+RO†	2.58	3.21	3.33	3.04	0.07
Reference	2.91	3.49	4.46	3.62	0.77
Table 5: Human evaluation on Newsela, where we measure adequacy (A), simplicity (S), fluency (F), and their average score (Avg), based on 1-5 Likert scale. We also count average instances of false information per sentence (FI).
the continuous space. By contrast, our approach only uses neural networks in the scoring function, but performs discrete edits of words and phrases. Thus, we achieve high fidelity (low FI) similar to the non-neural Hybrid model, which also performs editing on discourse parsing structures with PBMT.
In summary, our model takes advantage of both neural networks (achieving high adequacy, simplicity, and fluency) and traditional phrase-based approaches (achieving high fidelity).
Interestingly, the reference of Newsela has a poor (high) FI score, because the editors wrote simplifications at the document level, rather than the sentence level.
We proposed an iterative, edit-based approach to text simplification. Our approach works in an unsupervised manner that does not require a parallel corpus for training. In future work, we plan to add paraphrase generation to generate diverse simple sentences.
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), under grant Nos. RGPIN-2019-04897, RGPIN-2020-04465, and the Canada Research Chair Program. Lili Mou is also supported by AltaML, the Amii Fellow Program, and the Canadian CIFAR AI Chair Program. This research was supported in part by Compute Canada (www.computecanada.ca).
7926
Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJCNLP), pages 295-305.
Joachim Bingel and Anders Søgaard. 2016. Text simplification as tree labeling. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Short Papers, pages 337-343.
John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In Ninth Conference of the European Chapter of the Association for Computational Linguistics (EACL).
R. Chandrasekar, Christine Doran, and B. Srinivas. 1996. Motivations and methods for text simplification. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-Interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3393-3402.
Richard Evans, Constantin Orăsan, and Justin Dornescu. 2014. An evaluation of syntactic simplification rules for people with autism. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 131-140.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Dynamic multi-level multi-task learning for sentence simplification. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 462-476.
Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800.
Katharina Kann, Sascha Rothe, and Katja Filippova. 2018. Sentence-level fluency evaluation: References help, but can be spared! In Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL), pages 313-323.
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.
Beata Beigman Klebanov, Kevin Knight, and Daniel Marcu. 2004. Text simplification for information-seeking applications. In OTM Confederated International Conferences "On the Move to Meaningful Internet Systems", pages 735-747. Springer.
Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Burch. 2019. Complexity-weighted loss and diverse reranking for sentence simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 3137-3147.
Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202-1241.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies NAACL-HLT, pages 1865-1874.
Xianggen Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song. 2020. Unsupervised paraphrasing by simulated annealing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics (ACL): System Demonstrations, pages 55-60.
Louis Martin, Benoit Sagot, Éric de la Clergerie, and Antoine Bordes. 2019. Controllable sentence simplification. arXiv preprint arXiv:1910.02677.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. CGMH: Constrained sentence generation by Metropolis-Hastings sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6834-6842.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
George A. Miller. 1995. Wordnet: a lexical database for english.
Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 435-445.
7927
Shashi Narayan and Claire Gardent. 2016. Unsupervised sentence simplification using deep semantics. In Proceedings of the 9th International Natural Language Generation conference (INLG), pages 111-120.
Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P Dinu. 2017. Exploring neural text simplification models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Short Papers, pages 85-91.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311-318.
Adam Pauls and Dan Klein. 2012. Large-scale syntactic language modeling with treelets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL), pages 959-968.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Boris T Polyak and Anatoli B Juditsky. 1992. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-855.
Luz Rello, Clara Bayarri, Azuki Gorriz, Ricardo Baeza-Yates, Saurabh Gupta, Gaurang Kanvinde, Horacio Saggion, Stefan Bott, Roberto Carlini, and Vasile Topac. 2013. Dyswebxia 2.0!: more accessible text for people with dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, page 25.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4035-4045.
Raphael Schumann, Lili Mou, Yao Lu, Olga Vechtomova, and Katja Markert. 2020. Discrete optimization for unsupervised sentence summarization with word-level extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Advaith Siddharthan. 2002. An architecture for a text simplification system. In *Language Engineering Conference*, 2002. Proceedings, pages 64-71. IEEE.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018. BLEU is not suitable for the evaluation of text simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 738-744.
Sai Surya, Abhijit Mishra, Anirban Laha, Parag Jain, and Karthik Sankaranarayanan. 2019. Unsupervised neural text simplification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2058-2068.
I Sutskever, O Vinyals, and QV Le. 2014. Sequence to sequence learning with neural networks. Advances in NIPS.
Willian Massami Watanabe, Arnaldo Candido Junior, Vinicius Rodriguez Uzêda, Renata Pontin de Mattos Fortes, Thiago Alexandre Salgueiro Pardo, and Sandra Maria Aluísio. 2009. Facilita: reading assistance for low-literacy readers. In Proceedings of the 27th ACM international conference on Design of communication, pages 29-36.
Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1015-1024.
Wei Xu, Chris Callison-Burch, and Courtney Naples. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics (TACL), 3:283-297.
Wei Xu, Courtney Naples, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415.
Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 584-594.
Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018a. Integrating transformer and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3164-3173.
Yang Zhao, Zhiyuan Luo, and Akiko Aizawa. 2018b. A language model based evaluator for sentence compression. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), Short Papers, pages 170-175.
Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), pages 1353-1361.
7928