SlowGuess/ACL_2020_OCR huggingface dataset:
Table 10: The thresholds in paragraph alignment Algorithm 2 for Newsela data.
For Wikipedia data, we tailored our paragraph alignment algorithm (Algorithm 3 and 4). Table 11 provides the thresholds for Algorithm 4, which were chosen based on WIKI-MANUAL dev data.
Threshold	Value
τ1	0.991775706637882
τ2	0.8
τ3	0.5
τ4	5
τ5	0.9958
Table 11: The thresholds in paragraph alignment Algorithm 4 for Wikipedia data.
B Sentence Aligned Wikipedia Corpus
We present more details about our pre-processing steps for creating the WIKI-MANUAL and WIKI-AUTO corpora here. In Wikipedia, Simple English
is considered as a language by itself. When extracting articles from Wikipedia dump, we removed the meta-page and disambiguation pages. We also removed sentences with less than 4 tokens and sentences that end with a colon.
After the pre-processing and matching steps, there are 13,036 article pairs in which the simple article contains only one sentence. In most cases, that one sentence is aligned to the first sentence in the complex article. However, we find that the patterns of these sentence pairs are very repetitive (e.g., XXX is a city in XXX. XXX is a football player in XXX.). Therefore, we use regular expressions to filter out the sentences with repetitive patterns. Then, we use a BERT model fine-tuned on the WIKI-MANUAL dataset to compute the semantic similarity of each sentence pair and keep the ones with a similarity larger than a threshold
$^{9}$ https://pytorch.org/
7955
tuned on the dev set. After filtering, we ended up with 970 aligned sentence pairs in total from these 13,036 article pairs.
C Sentence Alignment on Wikipedia
In this section, we compare different approaches for sentence alignment on the WIKI-MANUAL dataset. Tables 12 and 13 report the performance for Task 1 (aligned + partially-aligned vs. not-aligned) on dev and test set. To generate prediction for MASAlign, CATS and two BERT_finetune methods, we first utilize the method in §3.2 to select candidate sentence pairs, as we found this step helps to improve their accuracy. Then we apply the similarity metric from each model to calculate the similarity of each candidate sentence pair. We tune a threshold for max f1 on the dev set and apply it to the test set. Candidate sentence pairs with a similarity larger than the threshold will be predicted as aligned, otherwise not-aligned. Sentence pairs that are not selected as candidates will also be predicted as not-aligned.
Dev set
P	R	F
MASSAlign (Paetzold et al., 2017)	72.9	79.5	76.1
CATS (Šajner et al., 2018)	65.6	82.7	73.2
BERTfinetune (NEWSELA-MANUAL)	82.6	83.9	83.2
BERTfinetune (WIKI-MANUAL)	87.9	85.4	86.6
+ ParaAlign	88.6	85.4	87.0
Our CRF Aligner (WIKI-MANUAL)	92.4	85.8	89.0
Table 12: Performance of different sentence alignment methods on the WIKI-MANUAL dev set for Task 1.
Test set
P	R	F
MASSAlign (Paetzold et al., 2017)	68.6	72.5	70.5
CATS (Šajner et al., 2018)	68.4	74.4	71.3
BERTfinetune (NEWSELA-MANUAL)	80.6	78.8	79.6
BERTfinetune (WIKI-MANUAL)	86.3	82.4	84.3
+ ParaAlign	86.6	82.4	84.5
Our CRF Aligner (WIKI-MANUAL)	89.3	81.6	85.3
D Sentence Simplification
D.1 Implementation Details
We used Fairseq $^{11}$  toolkit to implement our Transformer (Vaswani et al., 2017) and LSTM (Hochreiter and Schmidhuber, 1997) baselines. For the Transformer baseline, we followed  $\mathrm{BERT}_{\text {base }}$
Table 13: Performance of different sentence alignment methods on the WIKI-MANUAL test set for Task 1.
Parameter	Value	Parameter	Value
hidden units	768	batch size	32
filter size	3072	max len	100
# of layers	12	activation	GELU
attention heads	12	dropout	0.1
loss	CE	seed	13
Table 14: Parameters of our Transformer model.
Parameter	Value	Parameter	Value
hidden units	256	batch size	64
embedding dim	300	max len	100
# of layers	2	dropout	0.2
lr	0.001	optimizer	Adam
clipping	5	epochs	30
min vocab freq	3	seed	13
Table 15: Parameters of our LSTM model.
architecture for both encoder and decoder. We initialized the encoder using  $\mathrm{BERT}_{\text {base }}$  uncased checkpoint. Rothe et al. (2020) used a similar model for sentence fusion and summarization. We trained each model using Adam optimizer with a learning rate of 0.0001, linear learning rate warmup of 40k steps and 200k training steps. We tokenized the data with BERT WordPiece tokenizer. Table 14 shows the values of other hyperparameters.
For the LSTM baseline, we replicated the LSTM encoder-decoder model used by Zhang and Lapata (2017). We preprocessed the data by replacing the named entities in a sentence using  $\mathsf{spaCy}^{13}$  toolkit. We also replaced all the words with frequency less than three with  $<\mathsf{UNK}>$ . If our model predicted  $<\mathsf{UNK}>$ , we replaced it with the aligned source word (Jean et al., 2015). Table 15 summarizes the hyperparameters of LSTM model. We used 300-dimensional GloVe word embeddings (Pennington et al., 2014) to initialize the embedding layer.
<sup>11</sup>https://github.com/pytorch/fairseq
12 https://github.com/google-research/bert
<sup>13</sup>https://spacy.io/
7956
D.2 Human Evaluation
For this task you are given one source sentence and five (5) simplifications of the original sentence generated by different computer programs. The goal is to judge whether each simplified sentence
- is grammatically correct i.e. whether it is well-formed
- is simpler than the original source sentence.
- preserves meaning of the original sentence.
You will do this using a 1-5 rating scale, where 5 is best and 1 is worst. There are no "correct" answers and whatever choice is appropriate for you is a valid response. For example, if you are given the following complex sentence and simplifications:
Original sentence:
Financial markets had anticipated Portugal's need for assistance as its costs of financing had risen to unsustainable levels, and investors generally shrugged off the news on Thursday.
Simplifications	Meaning	Grammar	Simplicity
1. Financial markets had expected Portugal's need for help because costs had become unsustainable and investors dismissed the news on Thursday.	5	5	5
2. Financial markets had expected Portugal's need for help as its costs of financing had risen to unsustainable levels, and investors generally shrugged off the news on Thursday.	5	5	2
3. Financial markets the need for assistance had anticipated, costs of financing unsustainable shrugged off the news Thursday.	1	1	1
4. Financial markets had anticipated Portugal's need for assistance.	2	5	5
5. Financial markets dismissed the news on Thursday.	1	5	4
Sentence (1) gets a high rating with respect to simplicity since the long and complex sentence had been simplified considerably. Few words (e.g., generally, of financing) have been dropped, whereas others have been substituted with what more familiar ones (e.g. anticipated). It also gets high rating with respect to grammar and meaning because it is grammatically correct and preserves most of the meaning of the original. Sentence (2) also rates high in terms of grammar and meaning. However, it is not as simple as sentence (1) although some unfamiliar words have been substituted with simpler alternatives. Therefore, it gets a modest simplicity rating. Simplified sentence (3) makes little sense and is rather difficult to read. Therefore, it gets a low rating for grammar, simplicity and meaning. Simplified sentence (4) is fluent and easier to understand. So, it gets high rating in terms of grammar and simplicity. Although it is simpler than the original, it has omitted a large part of the sentence content. Simplifications that drastically change the meaning of the original sentence should be rated low in terms of meaning. Simplified sentence (5) changes the meaning but is easier to understand and well-formed. So, its gets low rating for meaning and high rating for simplicity and grammar. Simplifications that are grammatically correct should be rated high in terms of grammar even though they change the meaning of the original sentence.
In some cases, the computer program will choose not to change the original sentence at all. In such cases, try to think if you could make the sentence simpler. If this is the case then you should probably rate the computer-generated sentence low in terms of simplicity. Otherwise you can give high rating.
These sentences have been preprocessed by converting all letters to lowercase, separating punctuation, and spitting conjunctions. Please ignore this in your work and do not allow it to affect your judgments.
Figure 4: Instructions provided to Amazon Mechanical Turk workers to evaluate generated simplified sentences. We used the same instructions as described in Kriz et al. (2019).
7957
D.3 Example System Outputs
Examples
Generated by LSTM baseline
Complex (input)	In Seattle, eight activists between ages 10 and 15 petitioned Washington state last year to adopt stricter science-based regulations to protect them against climate change.
Simple (reference)	In Seattle, eight youths between 10 to 15 years old petitioned the state of Washington to change the law.
New (this work)	in seatale, eight activists between ages 10 and 15 asked washington state last year to keep the environment safe. (Phrasal Praphrase + Deletion)
Old (Xu et al., 2015)	in seatale, eight activists between ages 10 and 15 asked washington state last year to adopt stricter science - based rules to protect them against climate change. (Lexical Paraphrase)
Complex (input)	He recognized that another recommendation would be controversial with police groups: independent investigations after police shootings.
Simple (reference)	He admitted that police would not like one of the recommendations.
New (this work)	he thought another suggestion would be against the police. (Phrasal Paraphrase + Deletion)
Old (Xu et al., 2015)	he recognized that another suggestion would be controversial with police groups. (Lexical Paraphrase + Deletion)
Complex (input)	The Philadelphia Museum of Art has two famous selfie spots, both from the movie " Rocky."
Simple (reference)	The Philadelphia Museum of Art has two big selfie spots.
New (this work)	the philadelphia museum of art has two picture spots. (Lexical Paraphrase + Deletion)
Old (Xu et al., 2015)	the philadelphia museum of art has two famous spots. (Deletion)
Generated by TransformerBERT
Complex (input)	Some Chicago residents got angry about it.
Simple (reference)	The plan made some people angry.
New (this work)	some people in chicago were angry. (Phrasal Paraphrase)
Old (Xu et al., 2015)	some chicago residents got angry. (Deletion)
Complex (input)	Emissions standards have been tightened, and the government is investing money in solar, wind and other renewable energy.
Simple (reference)	China has also put a great deal of money into solar, wind and other renewable energy.
New (this work)	the government is putting aside money for new types of energy. (Phrasal Paraphrase + Deletion)
Old (Xu et al., 2015)	the government is investing in money, wind and other equipment. (Lexical Paraphrase + Deletion)
Complex (input)	On Feb. 9, 1864, he was sitting for several portraits, including the one used for the $5 bill.
Simple (reference)	On Feb. 9, 1864, several artists painted pictures of him.
New (this work)	on feb. 9, 1864, he was sitting for several portraits. (Deletion)
Old (Xu et al., 2015)	on feb. 9, 1864, he was sitting for several, including the $5 bill for the bill. (Deletion)
Table 16: Examples of simplified sentences generated by LSTM and Transformer<sub>bert</sub> models trained on our new NEWSELA-AUTO (this work) and old existing NEWSELA (Xu et al., 2015) datasets. The source sentences are from our new NEWSELA-AUTO test set. Models trained on our new data rephrase the input sentence more often than the models trained on old data. **Bold** indicates deletions or paraphrases.
Examples
Complex (input)	Now at age 9, his teachers say Richie reads at the level of a student in high school, and his vocabulary is well above those of his classmates.
Simple (reference)	He reads like a high school student.
LSTM	now he is age 9.
EditNTS (Dong et al., 2019)	he say his classmates are using a special job.
Rerank (Kriz et al., 2019)	but it is well above those of his classmates.
Transfomerbert(this work)	now at age 9, his teachers say that richie reads high schoolwork.
Complex (input)	He can recall the special feeling when, at age 7, he built his first kite and saw it waft into the air.
Simple (reference)	He can remember the special feeling when he built his first kite .
LSTM	he can remember the people when he was age 7.
EditNTS (Dong et al., 2019)	, at age 7, he built his first kite.
Rerank (Kriz et al., 2019)	he could remember the special feeling when.
Transfomerbert(this work)	he can remember the special feeling when he was 7 years old.
Complex (input)	Following the action , two middle-aged brothers slid down a hill holding signs.
Simple (reference)	For example , two grownup brothers slid down a hill holding signs .
LSTM	middle - aged brothers slid down a hill holding signs.
EditNTS (Dong et al., 2019)	two middle-aged brothers , 14 , heard down a hill signs.
Rerank (Kriz et al., 2019)	he made a hill holding signs.
Transfomerbert(this work)	two middle-aged brothers slid down a hill holding signs.
Table 17: Examples of simplifications generated by our best model,  $\text{Transformer}_{bert}$ , and other baselines, namely, EditNTS (Dong et al., 2019), Rerank (Kriz et al., 2019) and LSTM on the old NEWSELA test set. Both LSTM and  $\text{Transformer}_{bert}$  are trained on NEWSELA-AUTO. For EditNTS and Rerank, we use the system outputs shared by their original authors. Bold indicates new phrases introduced by the model.
7958
E Annotation Interface
E.1 Crowdsourcing Annotation Interface
A and B are equivalent
- Case 1: A simplify B or B simplify A (equivalent in meaning, though differ in length):
Figure 5: Instructions and an example question for our crowdsourcing annotation on the Figure Eight platform.
Please fully understand this example! This is the most crucial part of this task!
A: They could be killed by the terrorists if they come down from the mountain.  
B: The people risk death if they descend.
Two sentences convey the same meaning, while one sentence is simpler than the other one.
Don't judge by sentence length! Instead, judge by readability of the sentence.
- Case 2: A and B are equivalent in both meaning and readability:
A: They were trying to gather information and watch as the situation gets worse.
B: They were trying to gather information and monitor the worsening situation.
Two sentences are completely equivalent, as they mean the same thing.
Differing in some very unimportant information is acceptable.
- A and B are partially overlapped:
-Case 1:
A: The trip was disastrous, and Bishop promised herself she'd never fly with Nathaniel again.
B: The trip was very hard.
One sentence contains most of the information of the other one. It also contains important extra information.
The length of extra information should be equal or longer than a long phrase.
-Case 2:
A: Some Republicans have called for the president to take action and have said he doesn't need the approval of lawmakers.
B: Some Republicans have asked the president to take action, but the White House was waiting for more information to make decision.
Two sentences share some information in common.
And each of them also contains extra information.
A and B are mismatched:
A: The technology is new and very advanced.
B: The scientists hope it will also work on existing smartphones.
The two sentences are completely dissimilar in meaning.
Sentence A
The competition with West Point, which is now an annual affair, has grown into a rivalry.
Sentence B
The inmates have formed a popular debate club.
What's the relationship between Sentence A and Sentence B?
- A and B are equivalent (convey the same meaning, though one sentence can be much shorter or simpler than the other sentence)
$\bigcirc$  A,B are partially overlapped
- A and B are partially overlap (share information in common, while some important information differs/missing).
A and B are mismatched
The two sentences are completely dissimilar in meaning.
7959
E.2 In-house Annotation Interface
Sentence Alignment Viewer
Step 1: Setup Alignment File Path
Alignment File Path: Choose File current...csv
Step 2: Setup Article and Readability (Please click load)
Article Name: marktwain-newspaper.  $\Leftrightarrow$  Article 1 Readability: 1  $\downarrow$  Article 2 Readability: 0
VIRGINIA CITY, Nev. — One wonders what Mark Twain himself would make of the news: The Gold Rush-era newspaper for which he once wrote stories and witticisms on frontier life as a young journalist is once again in print after a decadeslong break.
The Territorial Enterprise, once the region's premier recorder of gossip, scandal, humor and tall tales — before Nevada was even a state — is back. The newspaper, which has run out of money on several occasions, is now a traditional monthly magazine. There is also an online edition, territorialenterprise.com.
Would Twain use Twitter to complain about the sad state of the press, as he once did with pen and ink? "If you don't read the newspaper, you're uninformed. If you do read the newspaper, you're misinformed."
Or would he gnash his teeth at the leaders of the media today? "I am not the editor of a newspaper and shall always try to do the right thing and be good so that God will not make me one."
Even the Enterprise's new editor, Elizabeth Thompson, guesses that Samuel Clemens - Twain's real name - would have a field day.
"He'd have something to say," she said. "He'd get a kick out of it."
Submit Change
VIRGINIA CITY, Nev. — One wonders what Mark Twain himself would make of the news: The Gold Rush-era newspaper for which he once penned stories and witticisms on frontier life as a fledgling journalist is once again in print after a decadeslong hiatus.
Following numerous attempts at solvency, the Territorial Enterprise, once the region's premier recorder of gossip, scandal, satire and irreverent tall tales — before Nevada was even a state — is back, this time as a traditional glossy monthly magazine and online edition, territorialenterprise.com.
Would Twain use Twitter to bemoan the deplorable state of the press, as he once did by pen? "If you don't read the newspaper, you're uninformed. If you do read the newspaper, you're misinformed."
Or gnash his teeth at media leadership? "I am not the editor of a newspaper and shall always try to do the right thing and be good so that God will not make me one."
Even the Enterprise's new editor, Elizabeth Thompson, guesses that Samuel Clemens would have a field day.
"I don't think he could resist with some witticism about the many attempts to resurrect the paper over the years," she said. "He'd have something to say. He'd get a kick out of it."
Figure 6: Annotation interface for correcting the crowdsourced alignment labels.