SlowGuess/NAACL_2021_OCR huggingface dataset:
John Lee and Chak Yan Yeung. 2018. Personalizing lexical simplification. In Proceedings of the 27th International Conference on Computational Linguistics.
Dan Liu, Wei Lin, Shiliang Zhang, Si Wei, and Hui Jiang. 2016. Neural networks models for entity discovery and linking. CoRR, abs/1611.03558.
Mounica Maddela and Wei Xu. 2018. A word-complexity lexicon and a neural readability ranking model for lexical simplification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Jonathan Mallinson and Mirella Lapata. 2019. Controllable sentence simplification: Employing syntactic and lexical constraints.
Louis Martin, Éric de la Clergerie, Benoit Sagot, and Antoine Bordes. 2020. Controllable sentence simplification. In Proceedings of The 12th Language Resources and Evaluation Conference.
Makoto Miwa, Rune Sætre, Yusuke Miyao, and Jun'ichi Tsujii. 2010. Entity-focused sentence simplification for relation extraction. In Proceedings of the 23rd International Conference on Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulíçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning.
Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.
Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh. 2019. Transforming complex sentences into a semantic hierarchy. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Daiki Nishihara, Tomoyuki Kajiwara, and Yuki Arase. 2019. Controllable text simplification with lexical constraint loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop.
Sergiu Nisioi, Sanja Šajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. Exploring neural text simplification models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
Gustavo Paetzold and Lucia Specia. 2015. LEXenstein: A framework for lexical simplification. In Proceedings of ACL-IJCNLP 2015 System Demonstrations.
Gustavo Paetzold and Lucia Specia. 2017. Lexical simplification with neural ranking. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics.
Gustavo Henrique Paetzold. 2016. Lexical Simplification for Non-Native English Speakers. Ph.D. thesis, University of Sheffield.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics.
David Pellow and Maxine Eskenazi. 2014. An open corpus of everyday documents for simplification tasks. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations.
Sarah E Petersen and Mari Ostendorf. 2007. Text simplification for language learners: A corpus analysis. In Proceedings of Workshop on Speech and Language Technology for Education.
Jipeng Qiang, Yun Li, Zhu Yi, Yunhao Yuan, and Xindong Wu. 2020. Lexical simplification with pretrained encoders. Association for the Advancement of Artificial Intelligence.
Scott E. Reed and Nando de Freitas. 2016. Neural programmer-interpreters. In 4th International Conference on Learning Representations.
Luz Rello, Ricardo Baeza-Yates, and Horacio Saggion. 2013. The impact of lexical simplification by verbal paraphrases for people with and without dyslexia. In Proceedings of the 14th International Conference on Computational Linguistics and Intelligent Text Processing.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Horacio Saggion. 2017. Automatic text simplification. Synthesis Lectures on Human Language Technologies.
Carolina Scarton and Lucia Specia. 2018. Learning simplifications for specific target audiences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
Advaith Siddharthan. 2002. An architecture for a text simplification system. In Proceedings of the Language Engineering Conference.
Advaith Siddharthan. 2006. Syntactic simplification and text cohesion. Research on Language and Computation.
Advaith Siddharthan and Angrosh Mandya. 2014. Hybrid text simplification using synchronous dependency grammars with hand-written and automatically harvested rules. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics.
Advaith Siddharthan, Ani Nenkova, and Kathleen McKeown. 2004. Syntactic simplification for improving content selection in multi-document summarization. In Proceedings of the 20th International Conference on Computational Linguistics.
Avirup Sil, Gourab Kundu, Radu Florian, and Wael Hamza. 2017. Neural cross-lingual entity linking. In Proceedings of the 30th Conference on Association for the Advancement of Artificial Intelligence.
Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea. 2012. Semeval-2012 task 1: English lexical simplification. In Proceedings of the First Joint Conference on Lexical and Computational Semantics: Proceedings of the Main Conference and the Shared Task and Proceedings of the Sixth International Workshop on Semantic Evaluation.
Sanja Štajner, Hannah Bechara, and Horacio Saggion. 2015. A deeper exploration of the standard PB-SMT approach to text simplification and its evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.
Sanja Štajner and Maja Popovic. 2016. Can text simplification help machine translation? In Proceedings of the 19th Annual Conference of the European Association for Machine Translation.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018a. BLEU is not suitable for the evaluation of text simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018b. Semantic structural evaluation for text simplification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018c. Simple and effective text simplification using semantic and neural methods. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
David Vickrey and Daphne Koller. 2008. Sentence simplification for semantic role labeling. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics.
Tu Vu, Baotian Hu, Tsendvarsen Munkhdalai, and Hong Yu. 2018. Sentence simplification with memory-augmented neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Wei Xu, Chris Callison-Burch, and Courtney Naples. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics.
Wei Xu, Courtney Naples, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics.
Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013. A lightweight and high performance monolingual word aligner. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics.
Li Zhang, Huaiyu Zhu, Siddhartha Brahma, and Yun-yao Li. 2020a. Small but mighty: New benchmarks for split and rephrase.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018. Integrating transformer and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Yang Zhong, Chao Jiang, Wei Xu, and Junyi Jessy Li. 2020. Discourse level factors for sentence deletion in text simplification. In Proceedings of the 2020 Conference on Association for the Advancement of Artificial Intelligence.
Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics.
A Implementation and Training Details
We implemented two separate Transformer models for neural deletion and split component (§2.1) and paraphrase generation (§2.3) using the Fairseq $^{12}$  toolkit. Both the encoder and decoder follow BERT $_{\text{base}}$  architecture, while the encoder is also initialized with BERT $_{\text{base}}$  checkpoint. For neural deletion and split component, we used a beam search of width 10 to generate candidates. The copy attention mechanism is a feedforward network containing 3 hidden layers, 1000 nodes in each layer with tanh activation, and a single linear output node with sigmoid activation. We used Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.0001, linear learning rate warmup of 40k steps, and 100k training steps. We used a batch size of 64. We used BERT WordPiece tokenizer. During inference, we constrained the beam-search to not repeat trigrams and emitted sentences that avoided aggressive deletion (compression ratio  $\in$  [0.9, 1.2]. We chose the best checkpoint based on the SARI score (Xu et al., 2016) on the dev set. We saved a checkpoint after every epoch. We did not perform any hyperparameter search and directly used the hyperparameters of the BERT-initialized Transformer described in? . The model takes 10 hours to train on 1 NVIDIA GeForce GPU.
Our pairwise ranking model, implemented using the PyTorch framework, consists of 3 hidden layers, 100 nodes in each layer, tanh activation, and a single linear output node. We used Adam optimizer with a learning rate of 0.01 and 10 epochs. We applied a dropout of 0.2. For Gaussian binning, we vectorized the numerical features into 10 dimensional vectors. The model takes half hour to train on 1 NVIDIA GeForce GPU. We do not perform any extensive hyperparameter tuning. We just examined few values for learning rate (0.001, 0.01 and 0.1) and chose the best based on the SARI score on the dev set. We used the original code for DisSim. $^{14}$
$^{12}$ https://github.com/pytorch/fairseq  
 $^{13}$ https://github.com/google-research/bert  
 $^{14}$ https://github.com/Lambda-3/  
DiscourseSimplification
Detailed Instructions
You are given three sentences that need to be rewritten so that they use simpler English. This means that you should reduce the number of difficult words or idioms and make the sentence more straightforward. We ask you to accomplish this through lexical paraphrasing only.
Lexical paraphrasing involves changing complex words or phrases for simpler synonyms. For example:
Original	Simplification
From its inception, it was designated a duty-free port and vied with the neighboring Sultanate of Pattani for trade.	From the start, it was chosen to be a duty-free port to compete with the neighboring Sultanate of Pattani for trade.
People are constantly tossing out stuff to make more space.	People are always throwing out things to make more space.
Catania decided to design an experiment to find out if his hunch was correct.	Catania decided to design an experiment to test his idea.
*the bold facing here is for ease of readability and will not be used in the task.
When paraphrasing, try to preserve as much of the original meaning as possible. Proper names (e.g. John, Microsoft, Apple), geographical locations (e.g. Northern Europe, Sultanate of Pattani), specialized technical terms (e.g. polymorphism, electropneumatic) or any word you don't know should be kept whenever possible. You should use simpler concepts/words/phrases in your paraphrasing. This means your sentence could potentially have more words than the original.
Since you will be simplifying sentences in isolation, sometimes it could be possible to delete words that do not contribute to the meaning of the sentence by itself. For example:
Original	Simplification
The automotive industry in California is still small compared with Detroit, however.	The vehicle industry in California is smaller than the one in Detroit.
*the bold facing here is for ease of readability and will not be used in the task.
In the previous example, "however" was deleted since it does not contribute to the meaning of the sentence presented in this way. Your focus should be on lexical paraphrasing, and deletion should be limited to cases where it is necessary to remove words in order to keep the sentence fluent.
We will manually check a fraction of your answers and spammers will be rejected. Read the provided instructions carefully and make sure you understand the task before beginning.
View Instructions
We ask you to rewrite each original sentence in order to make it easier to understand by non-native speakers of English. You can do so by mainly replacing complex words with simpler synonyms (i.e. lexical paraphrasing). You are allowed to delete a few words, if needed, to keep the sentence fluent after performing the lexical paraphrasing. The final simplified sentences need to be grammatical, fluent, and retain the main ideas of their original counterparts without altering their meanings. More detailed instructions and examples can be found by clicking the "View Instructions" button.
Carefully read the instructions provided and then simplify the following sentences:
Sentence 1 of 2
Original: When the brain does not have enough information, it interprets the image and makes guesses.
Type the Simplification here...
Sentence 2 of 2
Original: The researchers found that the amount of friction depended strongly on whether the tip was dragged along the direction in which the pairs were oriented, or against it.
Type the Simplification here..
Figure 2: Annotation guidelines for our NEWSELA-TURK corpus along with example questions.
C System Outputs
System Outputs
Complex	This year, the FAA has approved dozens of permits for agricultural drone businesses.
Simple	This year, it approved dozens of permits for agricultural drone businesses.
Hybrid-NG	this year, the government has approved dozens of drone permits for agricultural businesses.
LSTM	this year, the faa has approved dozens of permits for agricultural drone businesses.
Transformerbert	this year, the faa has approved dozens of permits for agricultural businesses.
EditNTS	this year, the government has approved dozens of permits for drone businesses for no permission.
Our Model (cp = 0.6)	this year, the faa has allowed many businesses to use drones.
Our Model (cp = 0.7, 0.8)	this year, the faa has approved dozens of permits for drones.
Complex	The room echoed with the sounds of song, the beat of drums, the voices of young men.
Simple	As she spoke, the building echoed with music and the beat of drums.
Hybrid-NG	echoed the room.
LSTM	the room echoed with the sounds of song, the voices of young men.
Transformerbert	the room echoed with the sound of song, the beat of drums, the voices of young men.
EditNTS	the room echoed with the sounds of song, the beat of drums, the voices of young men who are hungry and legs.
Our Model (cp = 0.6)	the sound of the room was full of sounds of young men and the voices of cellos.
Our Model (cp = 0.7)	the sound of the room sounded like a lot of music, and the voices of young men.
Our Model (cp = 0.8)	the sound of the room sounded like a song, the beat of drums, and the voices of young men.
Table 10: Examples of system outputs by our paraphrase generation model and other baselines. Our model generates paraphrase-focused simplifications by skipping the splitting and deletion steps and running only the neural paraphrase generation component. (red marks the errors; blue marks good paraphrases).  $cp$  is a soft constraint that denotes the extent of paraphrasing in terms of number of words that can be copied from the input.
D Additional Evaluation on Newsela
Models	SARI	add	keep	del	FK	SLen	OLen	CR	%split	s-BL	%new	%eq
Complex (input)	20.6	0.0	61.7	0.0	9.2	16.9	17.0	1.0	0.0	100.0	0.0	100.0
Simple (reference)	94.6	93.6	91.4	98.8	8.7	17.9	17.9	1.06	0.0	48.0	29.7	0.0
Hybrid-NG	35.0	2.3	52.7	50.1	7.8	16.1	17.0	1.0	5.6	61.7	9.3	9.1
\(Transformer_{bert}\)	35.3	3.4	52.9	49.6	7.0	13.5	15.2	0.91	10.4	60.2	14.4	15.7
EditNTS	37.7	2.0	56.4	54.5	7.6	14.2	15.5	0.93	8.7	69.0	7.1	3.5
Our Model	37.9	4.4	51.3	58.0	6.7	13.6	15.3	0.92	9.7	49.2	19.2	0.9
Our Model (no split; cp = 0.6)	38.3	3.9	53.8	57.3	7.9	16.1	16.7	1.0	0.0	53.4	20.8	3.6
Our Model (no split; cp = 0.7)	39.1	3.7	58.5	55.2	8.3	16.2	16.8	1.0	0.0	67.6	12.4	11.0
Our Model (no split; cp = 0.8)	38.0	3.3	60.3	50.4	8.5	16.4	16.9	1.0	0.0	76.5	8.2	20.3
Table 11: Automatic evaluation results on a subset of Newsela test set that focuses on paraphrasing (8371 complex-simple sentence with compression ratio  $>0.9$  and no splits). We control the extent of paraphrasing of our models by specifying the percentage of words to be copied  $(cp)$  from the input as a soft constraint.
E Human Evaluation Interface
Original sentence:
Financial markets had anticipated Portugal's need for assistance as its costs of financing had risen to unsustainable levels, and investors generally shrugged off the news on Thursday.
Simplifications	Meaning	Grammar	Simplicity
1. Financial markets had expected Portugal's need for help because costs had become unbearable and investors dismissed the news on Thursday.	5	5	5
2. Financial markets had expected Portugal's need for help. Its financial costs had reached impossible levels and investors disregarded the news on Thursday.	4	5	5
3. Financial markets had expected Portugal's need for help as its costs of financing had risen to unsustainable levels, and investors generally shrugged off the news on Thursday.	5	5	2
4. Financial markets the need need for assistance had anticipated, costs of financing unsustainable shrugged off the news Thursday.	1	1	1
5. Financial markets had anticipated Portugal's need for assistance.	2	5	5
Sentence (1) receives a high rating with respect to simplicity since the long and complex sentence had been simplified considerably. Few words (e.g., generally, of financing) have been dropped, whereas others have been substituted with more familiar ones (e.g. anticipated). It also gets high rating with respect to grammar and meaning because it is grammatically correct and preserves most of the meaning of the original sentence.
In sentence (2), the original sentence is split into two sentences and the unfamiliar words have been substituted with simpler words. Therefore, it receives a high simplicity rating. It also receives high meaning and grammar ratings.
Sentence (3) also rates high in terms of grammar and meaning. However, it is not as easy to understand as sentence (1) or (2) and contains difficult words. Therefore, it gets a modest simplicity rating.
Sentence (4) makes little sense and is rather difficult to read. Therefore, it gets a low rating for grammar, simplicity and meaning.
Sentence (5) is fluent and easier to understand. So, it gets high rating in terms of grammar and simplicity. Although it is simpler than the original, it has omitted a large part of the sentence content.
Simplifications that drastically change the meaning of the original sentence should be rated low in terms of meaning.
In some cases, the computer program will choose not to change the original sentence at all. In such cases, try to think if you could make the sentence simpler. If this is the case then you should probably rate the computer-generated sentence low in terms of simplicity. Otherwise you can give high rating.
These sentences have been preprocessed by converting all letters to lowercase, separating punctuation, and spitting conjunctions. Please ignore this in your work and do not allow it to affect your judgments.
Figure 3: Guidelines provided to the Amazon Mechanical Turk workers for evaluating simplified sentences. Our interface is based on the one proposed by Kriz et al. (2019).
For this task, you are given one sentence and its simplified version generated by different computer programs. The goal is to use the slider to indicate how much you agree with the following statements (1 = Strongly disagree, 5 = Strongly agree). For the last question, select one of the multiple choices (Yes, No or No sentence splits).
1. The simplified sentence adequately expresses the meaning of the original sentence.
2. The simplified sentence is fluent
3. The simplified sentence undergoes correct sentence splitting.
Simplifications	Meaning	Fluency	Correct Split
1. financial markets had expected portugal's need for help because costs had become unbearable and investors dismissed the news on thursday .	5	5	Sentence splitting does not occur
2. financial markets had expected portugal's need for assistance . its financial costs had reached unsustainable levels and investors disregarded the news on thursday .	5	5	Yes
3. financial markets had expected portugal's need for help as its costs of financing . had risen to unsustainable levels investors generally shrugged off the news on thursday .	4	3	No
4. financial markets had anticipated portugal's need need for assistance . its cost of financing has increased .	3	2	Yes
red and green colors indicate errors and good changes respectively. They will not be used in the task. The simplifications have been preprocessed by converting all letters to lowercase and separating punctuation. Please do not let it affect your judgements.
Sentence (1) is fluent and preserves most of the meaning. But it does not undergo sentence splitting.
Sentence (2) is also fluent and preserves most of the meaning. The split sentences are coherent and split is grammatically correct.
Sentence (3) shows issues with sentence splitting.
Sentence (4) has few fluency issues within the shorter sentences. However, the sentence splitting is grammatically correct and easy to read.
Figure 4: Guidelines provided to the Amazon Mechanical Turk workers for evaluating simplified sentences specifically for sentence splitting.
F Evaluation on Wikipedia
Models	SARI	add	keep	del	FK	SLen	OLen	CR	%split	s-BL	%new	%eq
Complex (input)	25.9	0.0	77.8	0.0	13.4	22.4	22.6	1.0	0.0	100.0	0.0	100.0
Simple (reference)	42.0	20.6	59.9	45.5	10.9	19.1	19.3	0.88	1.1	55.2	15.3	7.8
Hybrid-NG	25.4	0.1	42.7	33.5	9.0	13.3	13.4	0.6	0.8	38.2	1.4	3.1
LSTM	32.6	2.1	59.8	36.0	10.0	17.8	17.8	0.84	0.8	60.0	10.7	15.0
\(Transformer_{bert}\)	35.1	4.3	61.8	39.2	10.4	16.7	18.8	0.85	10.9	62.1	11.1	11.1
EditNTS	36.1	2.5	67.4	38.5	11.7	20.9	22.4	1.02	6.4	63.5	13.5	0.0
Our Model	35.9	4.7	63.6	39.6	9.2	14.7	19.8	0.9	33.7	63.2	12.9	9.2
Our Model (no split; cp = 0.6)	36.5	4.9	63.2	41.4	10.8	18.6	19.9	0.89	6.7	61.9	12.4	3.9
Our Model (no split; cp = 0.7)	37.5	4.3	68.8	39.4	11.2	19.1	20.9	0.94	8.9	72.6	8.6	12.3
Our Model (no split; cp = 0.8)	37.0	3.8	72.0	35.3	11.7	19.8	21.7	0.97	8.4	80.4	6.6	24.5