SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 3: Automatic evaluation results on NewSELA-TURK that focuses on paraphrasing (500 complex sentences with 4 human written paraphrases). We control the extent of paraphrasing of our models by specifying the percentage of words to be copied  $(cp)$  from the input as a soft constraint.
Models	SARI	add	keep	del	FK	SLen	OLen	CR	%split	s-BL	%new	%eq
Complex (input)	17.0	0.0	51.1	0.0	14.6	30.0	30.2	1.0	0.0	100.0	0.0	100.0
Simple (reference)	93.0	89.9	91.6	97.5	7.0	13.4	28.6	0.98	100.0	36.8	29.7	0.0
Hybrid-NG	37.1	2.2	44.9	64.1	11.6	25.5	30.1	1.0	17.3	57.7	8.7	1.6
\( Transformer_{bert} \)	39.5	4.2	47.3	67.0	8.8	17.1	25.3	0.85	39.7	57.7	11.9	5.2
EditNTS	38.9	1.5	49.1	66.2	9.1	16.9	26.2	0.88	50.3	71.2	7.2	0.2
Our Model	39.4	4.0	46.6	67.6	8.7	17.5	25.5	0.85	40.6	48.3	15.6	0.1
Our Model (w/ split)	42.1	5.6	50.6	70.1	8.1	15.3	30.3	1.02	93.5	60.7	12.4	1.1
Table 4: Automatic evaluation results on a splitting-focused subset of the NEWSELA-AUTO test set (9,356 sentence pairs with splitting). Our model chooses only candidates that have undergone splitting during the ranking step.
Models	SARI	add	keep	del	FK	SLen	OLen	CR	%split	s-BL	%new	%eq
Complex (input)	9.6	0.0	28.8	0.0	12.9	25.8	26.0	1.0	0.0	100.0	0.0	100.0
Simple (reference)	85.7	82.7	76.0	98.6	6.7	12.6	12.6	0.5	0.0	19.6	32.6	0.0
Hybrid-NG	35.8	1.4	27.0	79.1	10.6	22.7	25.9	1.0	13.3	58.9	8.7	3.6
\( \text{Transformer}_{bert} \)	36.8	2.2	29.6	78.7	8.4	16.2	21.7	0.85	27.7	57.9	12.3	8.2
EditNTS	37.1	1.0	29.7	80.7	8.8	16.6	23.1	0.91	36.6	71.8	7.8	0.6
Our Model	39.2	2.4	29.8	85.3	8.2	16.4	21.9	0.85	29.1	48.8	15.6	0.4
Our Model (no split; CR<0.7)	38.2	2.0	28.5	84.1	8.6	16.8	17.5	0.68	0.1	42.0	12.5	0.2
Table 5: Automatic evaluation results on a deletion-focused subset of the NEWSELA-AUTO test set (9,511 sentence pairs with compression ratio  $< 0.7$  and no sentence splits). Our model selects only candidates with similar compression ratio and no splits during ranking.
Metrics. We report SARI (Xu et al., 2016), which averages the F1/precision of n-grams ( $n \in \{1,2,3,4\}$ ) inserted, deleted and kept when compared to human references. More specifically, it computes the F1 score for the n-grams that are added (add), which is an important indicator if a model is good at paraphrasing. The model's deletion capability is measured by the F1 score for n-grams that are kept (keep) and precision for those deleted (del). To evaluate a model's para
phrasing capability and diversity, we calculate the BLEU score with respect to the input (s-BL), the percentage of new words (\%new) added, and the percentage of system outputs identical to the input (\%eq). Low s-BL,  $\%eq$ , or high  $\%$  new indicate that the system is less conservative. We also report Flesch-Kincaid (FK) grade level readability (Kincaid and Chissom, 1975), average sentence length (SLen), the percentage of splits (\%split), compression ratio (CR), and average output length (OLen). We do not report BLEU because it often does not correlate with simplicity (Sulem et al., 2018a,b; Xu et al., 2016).
Results. Table 2 shows the results on NEWSELA-AUTO test set. Our model outperforms the state-of-the-art Transformer<sub>bert</sub> and EditNTS models with respect to SARI.<sup>10</sup> EditNTS and LSTM focus on
10 According to Jiang et al. (2020), a BERT-initialized Transformer performs better than EditNTS. We see a different behavior here because we retained sentence splits from 0-1, 1-2,
<sup>8</sup>We slightly improved the SARI implementation by Xu et al. (2016) to exclude the spurious ngrams while calculating the F1 score for add. For example, if the input contains the phrase "is very beautiful", the phrase "is beautiful" is treated as a new phrase in the original implementation even though it is caused by the delete operation.
${}^{9}$  SARI score of a reference with itself may not always be 100 as it considers 0 divided by 0 as 0, instead of 1 , when calculating n-gram precision and recall. This avoids the inflation of del scores when the input is same as the output.
Model	Overall Simplification	Structural Simplification
Fluency	Adequacy	Simplicity	Average	Fluency	Adequacy	Has Split	Correct Split
Hybrid-NG	3.23*	2.96*	3.40*	3.19*	3.25*	3.53*	42%	15%
EditNTS	3.88*	3.70	3.67	3.75	4.08	3.81*	41%	18%
\(Transformer_{bert}\)	3.91	3.63	3.65*	3.73	4.15	3.65*	53%	49%
Our Model	4.02	3.65	3.77	3.81	4.19	4.13	97%	90%
Simple (reference)	4.12	3.64	3.97	3.84	4.41	4.10	100%	100%
Table 6: Human evaluation of 100 random simplifications from the NEWSELA-AUTO test set and the split-focused subset of the same test set. Has Split and Correct Split denote the percentage of the output sentences that have undergone splitting and the percentage of coherent splits respectively. * denotes that our model is significantly better than the corresponding baseline (according to a  $t$ -test with  $p < 0.05$ ).
deletion as they show high self-BLEU ( $>66.5$ ) and FK ( $>8.8$ ) scores despite having compression ratios similar to other systems. Transformer model alone is rather conservative and copies  $10.2\%$  of the sentences directly to the output. Although HybridNG makes more changes than any other baselines, its SARI and add scores are 3.7 and 1.7 points lower than our model indicating that it generates more errors. Our model achieves the lowest self-BLEU (48.7), FK (7.9), and percentage of sentences identical to the input (0.4), and the highest add (3.3) score and percentage of new words ( $16.2\%$ ). In other words, our system is the least conservative, generates more good paraphrases, and mimics the human references better. We provide examples of system outputs in Table 9 and Appendix C.
Tables 3, 4, and 5 show the results on NEWSELA-TURK, split-focused, and delete-focused subsets of NEWSELA-AUTO test set respectively. For these experiments, we configure our model to focus on specific operations (details in §2.4). Our model again outperforms the existing systems according to SARI, add score, and percentage of new words, which means that our model is performing more meaningful paraphrasing. We show that we can control the extent of paraphrasing by varying the copy ratio  $(cp)$ . Our model splits  $93.5\%$  of the sentences, which is substantially better than the other models.
We performed two human evaluations: one to measure the overall simplification quality and the other to specifically capture sentence splitting. For the first one, we asked five Amazon Mechanical Turk workers to evaluate fluency, adequacy and simplicity of 100 random simplifications from the NEWSELA-AUTO test set. We supplemented the
2-3 readability levels in NEWSELA-AUTO, which contained more lexical overlaps and inflated the scores for EditNTS.
<sup>11</sup>We provide instructions in Appendix E.
fluency and adequacy ratings with binary questions described in Zhang et al. (2020a) for the second evaluation over another 100 simplifications from the NEWSELA-AUTO split-focused test set. We asked if the output sentence exhibits spitting and if the splitting occurs at the correct place. While fluency measures the grammaticality of the output, adequacy captures the extent of meaning preserved when compared to the input. Simplicity evaluates if the output is simpler than the input. Each sentence was rated on a 5-point Likert scale and we averaged the ratings from the five workers. We chose the majority value for the binary ratings. We used the output of our model that is tailored for sentence splitting for the second evaluation.
Table 6 demonstrates that our model achieves the best fluency, simplicity, and overall ratings. The adequacy rating is also very close to that of Transformer<sub>bert</sub> and EditNTS even though our model is performing more paraphrasing (Table 2), which verifies that the changes made by our system are meaningful. Our model achieves the most number of correct sentence splits (90%), and the highest fluency (4.19) for syntactic simplification, showing that it can generate more number of coherent sentence splits when compared to other models.
4 Model Analysis
In this section, we analyze the contribution of each model component and examine the system errors.
4.1 System Ablations
We evaluate our key design choices, namely candidate ranking that is based on length-penalized BERTScore and paraphrase generation that uses data augmentation and copy attention. Table 8 summarizes the results. Our pairwise ranking model (BERTScorelen) achieves an increase of 3.2 points in SARI when compared to choosing a random (Random) candidate. Randomly selecting a candidate also performs fairly well, indicating that the
Examples
Good (49%)
Complex	The Seattle kids petitioned Washington state last year to adopt stricter science-based regulations to protect them against climate change.
Simple	The eight kids from Seattle who petitioned Washington state last year are fighting for the state to adopt stricter laws to protect their future against climate change.
Our Model	The Seattle kids asked the Washington state last year to take tougher rules against climate change.
Hallucinations (13%)
Complex	"It's more of a family than living outside," said Jessica Konczal, 33, whose husband is Sgt. Matthew Konczal.
Simple	Jessica Konczal, age 33, says the base is its own type of family.
Our Model	"It's more of a family than living outside," said Jessica Konczal, one of the protesters.
Fluency Errors (14%)
Complex	Parental feedback on the menu additions so far, from some of the early adopter markets, has been "phenomenal," Leverton said.
Simple	Leverton said parents love the new food.
Our Model	So far, parents parents have feedback on the menu changes has been a great deal, he added.
Bad Substitution (15%)
Complex	One of the device's inventors explained to the president that the machine was a prototype.
Simple	One of the girls explained it was just a prototype.
Our Model	One of the inventors told the president that the first design was a design.
Anaphora Resolution / Entailment (12%)
Complex	Sea slugs dubbed sacoglossans are some of the most remarkable biological burglars on the planet.
Simple	The sea slugs known as sacoglossans are some of the most interesting burglars on the planet.
Our Model	Scientists say these are some of the most interesting creatures on the planet.
Human Reference Errors (4%)
Complex	They float in and out of formations that combine the underwater world with the stage.
Simple	In the dance on a round stage on the seabed, the dancers swim together and apart.
Our Model	They float in and out of places that combine stage with the underwater.
Table 7: Error analysis of 200 simplifications from NEWSELA-AUTO test set generated by our model. Red marks the errors; blue marks good paraphrases.
SARI	FK	CR	%split	%new
Complex (input)	15.9	12.2	1.0	0.0	0.0
Simple (reference)	90.5	7.5	0.83	28.9	32.8
Random Candidate	33.7	8.1	0.81	34.4	5.5
BERTScorelen	36.9	9.0	0.87	25.9	5.9
Our Model	38.6	8.4	0.88	26.1	18.9
- augmentation	37.6	7.9	0.86	29.5	12.7
- copy attn	36.0	8.1	0.87	26.2	15.9
- only Transformer	37.9	7.7	0.78	26.3	16.5
- only DisSim	37.2	8.3	0.84	27.1	18.0
Table 8: Model ablation study on dev set
sentence splitting and deletion models we chose are of good quality.
Compared to our final model (Our Model), its variants without data augmentation  $(- \text{augmentation})$  and copy mechanism  $(- \text{copy attn})$  suffer a drop of 1.0 and 2.6 points in SARI respectively and a decrease of at least  $3.0\%$  of new words, which demonstrates that these components encourage the system to paraphrase. Our model trained on only DisSim  $(- \text{only DisSim})$  and Transformer  $(- \text{only Transformer})$  candidates performs close to our best model (Our Model) in terms of SARI.
4.2 Error Analysis
To understand the errors generated by our model, we manually classified 200 simplifications from the
NEWSELA-AUTO test set into the following categories: (a) Good, where the model generated meaningful simplifications, (b) Hallucinations, where the model introduced information not in the input, (c) Fluency Errors, where the model generated ungrammatical output, (d) Anaphora Resolution, where it was difficult to resolve pronouns in the output. (e) Bad substitution, where the model inserted an incorrect simpler phrase, and (e) Human Reference Errors, where the reference does not reflect the source sentence. Note that a simplification can belong to multiple error categories. Table 7 shows the examples of each category.
Before the advent of neural networks, text simplification approaches performed each operation separately in a pipeline manner using either handcrafted rules (Carroll et al., 1999; Siddharthan, 2002; Siddharthan et al., 2004) or data-driven methods based on parallel corpora (Zhu et al., 2010; Woodsend and Lapata, 2011; Narayan and Gardent, 2014). Following neural machine translation, the trend changed to performing all the operations together end-to-end (Zhang and Lapata, 2017; Nisioi et al., 2017; Zhao et al., 2018; Alva-Manchego et al., 2017; Vu
System Outputs
Complex	Since 2010, project researchers have uncovered documents in Portugal that have revealed who owned the ship.
Simple	Since 2010, experts have been figuring out who owned the ship.
Hybrid-NG	since 2010, the project scientists have uncovered documents in portugal that have about who owns the ship.
LSTM	since 2010, scientists have uncovered documents in portugal that have revealed who owned the ship.
Transformerbert	they discovered that the ship had been important.
EditNTS	since 2010, project researchers have uncovered documents in portugal. have revealed who owned the ship.
Our Model (cp = 0.6)	scientists have found a secret deal. they have discovered who owned the ship.
Our Model (cp = 0.7)	scientists have found documents in portugal. they have also found out who owned the ship.
Our Model (cp = 0.8)	scientists have found documents in portugal. they have discovered who owned the ship.
Complex	Experts say China's air pollution exacts a tremendous toll on human health.
Simple	China's air pollution is very unhealthy.
Hybrid-NG	experts say the government's air pollution exacts a toll on human health.
LSTM	experts say china's air pollution exacts a tremendous toll on human health.
Transformerbert	experts say china's pollution has a tremendous effect on human health.
EditNTS	experts say china's air pollution can cause human health.
Our Model (cp = 0.6)	experts say china's air pollution is a big problem for human health.
Our Model (cp = 0.7)	experts say china's air pollution can cause a lot of damage on human health.
Our Model (cp = 0.8)	experts say china's air pollution is a huge toll on human health.
Table 9: Examples of system outputs. Red marks the errors; blue marks good paraphrases.  ${cp}$  is a soft constraint that denotes the percentage of words that can be copied from the input.
et al., 2018; Kriz et al., 2019; Dong et al., 2019; Jiang et al., 2020) at the cost of controllability and performance as shown in this paper.
Controllable text simplification has been attempted before, but only with limited capability. Scarton and Specia (2018) and Martin et al. (2020) added additional tokens to the input representing grade level, length, lexical, and structural complexity constraints. Nishihara et al. (2019) proposed a loss which controls word complexity, while Mallinson and Lapata (2019) concatenated constraints to each word embedding. Kumar et al. (2020) proposed a linguistic scoring function to control the edits to the input.
Another long body of research focuses on a single simplification operation and can be broadly divided into three categories: (1) Lexical Simplification (Specia et al., 2012; Horn et al., 2014; Glavaš and Šajner, 2015; Paetzold and Specia, 2017, 2015; Maddela and Xu, 2018; Qiang et al., 2020), where complex words are substituted with simpler words. (2) Syntactic Simplification (Siddharthan, 2006; Aharoni and Goldberg, 2018; Botha et al., 2018; Niklaus et al., 2019), which deals exclusively with sentence splitting, and (3) Sentence Compression (Filippova et al., 2015; Rush et al., 2015; Nallapati et al., 2016; See et al., 2017; Baziotis et al., 2019), where the goal is to shorten the input sentence by removing its irrelevant content.
We proposed a novel hybrid approach for sentence simplification that performs better and produces more diverse outputs than the existing systems. We designed a new data augmentation method to encourage the model to paraphrase. We created a new dataset, NEWSELA-TURK, to evaluate paraphrasing-focused simplifications. We showed that our model can control various attributes of the simplified text, such as number of sentence splits, length, and number of words copied from the input.
We thank the anonymous reviewers for their valuable feedback. We thank Newsela for sharing the data and NVIDIA for providing GPU computing resources. This research is supported in part by the NSF award IIS-1822754, ODNI and IARPA via the BETTER program contract 19051600004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Roee Aharoni and Yoav Goldberg. 2018. Split and rephrase: Better evaluation and stronger baselines. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Natural Language Processing.
Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoit Sagot, and Lucia Specia. 2020. ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In Proceedings of the Association for Computational Linguistics.
Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. 2019. SEQ^3: Differentiable sequence-to-sequence-to-sequence autoencoder for unsupervised abstractive sentence compression. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. 2018. Learning to split and rephrase from Wikipedia edit history. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In Proceedings of the 9th Conference of the European Chapter of the ACL.
R. Chandrasekar, Christine Doran, and B. Srinivas. 1996. Motivations and methods for text simplification. In The 16th International Conference on Computational Linguistics.
Han-Bin Chen, Hen-Hsen Huang, Hsin-Hsi Chen, and Ching-Ting Tan. 2012. A simplification-translation-restoration framework for cross-domain SMT applications. In Proceedings of the 24th International Conference on Computational Linguistics.
William Coster and David Kauchak. 2011. Simple English Wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-Interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Katja Filippova, Enrique Alfonseca, Carlos Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with lstms. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Goran Glavaš and Sanja Štajner. 2015. Simplifying lexical simplification: Do we need simplified corpora? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
Colby Horn, Cathryn Manduca, and David Kauchak. 2014. Learning a lexical simplifier using wikipedia. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF sentence alignment model for text simplification. In Proceedings of the Association for Computational Linguistics.
Tomoyuki Kajiwara, Hiroshi Matsumoto, and Kazuhide Yamamoto. 2013. Selecting proper lexical paraphrase for children. In Proceedings of the 25th Conference on Computational Linguistics and Speech Processing.
Robert P. Jr.; Rogers Richard L.; Kincaid, J. Peter; Fishburne and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. research branch report.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Burch. 2019. Complexity-weighted loss and diverse reranking for sentence simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics.
Dhruv Kumar, Lili Mou, Lukasz Golab, and Vechtomova Olga. 2020. Iterative edit-based unsupervised sentence simplification. In Proceedings of the Association for Computational Linguistics.