SlowGuess/ACL_2020_OCR huggingface dataset:
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, Wei Xu
{jiang.1530, maddela.4, lan.105, zhong.536, xu.1265}@osu.edu
The success of a text simplification system heavily depends on the quality and quantity of complex-simple sentence pairs in the training corpus, which are extracted by aligning sentences between parallel articles. To evaluate and improve sentence alignment quality, we create two manually annotated sentence-aligned datasets from two commonly used text simplification corpora, Newsela and Wikipedia. We propose a novel neural CRF alignment model which not only leverages the sequential nature of sentences in parallel documents but also utilizes a neural sentence pair model to capture semantic similarity. Experiments demonstrate that our proposed approach outperforms all the previous work on monolingual sentence alignment task by more than 5 points in F1. We apply our CRF aligner to construct two new text simplification datasets, NEWSELA-AUTO and WIKI-AUTO, which are much larger and of better quality compared to the existing datasets. A Transformer-based seq2seq model trained on our datasets establishes a new state-of-the-art for text simplification in both automatic and human evaluation.
Text simplification aims to rewrite complex text into simpler language while retaining its original meaning (Saggion, 2017). Text simplification can provide reading assistance for children (Kajiwara et al., 2013), non-native speakers (Petersen and Ostendorf, 2007; Pellow and Eskenazi, 2014), non-expert readers (Elhadad and Sutaria, 2007; Siddharthan and Katsos, 2010), and people with language disorders (Rello et al., 2013). As a preprocessing step, text simplification can also improve
the performance of many natural language processing (NLP) tasks, such as parsing (Chandrasekar et al., 1996), semantic role labelling (Vickrey and Koller, 2008), information extraction (Miwa et al., 2010), summarization (Vanderwende et al., 2007; Xu and Grishman, 2009), and machine translation (Chen et al., 2012; Stajner and Popovic, 2016).
Automatic text simplification is primarily addressed by sequence-to-sequence (seq2seq) models whose success largely depends on the quality and quantity of the training corpus, which consists of pairs of complex-simple sentences. Two widely used corpora, NEWSELA (Xu et al., 2015) and WIKILARGE (Zhang and Lapata, 2017), were created by automatically aligning sentences between comparable articles. However, due to the lack of reliable annotated data, sentence pairs are often aligned using surface-level similarity metrics, such as Jaccard coefficient (Xu et al., 2015) or cosine distance of TF-IDF vectors (Paetzold et al., 2017), which fails to capture paraphrases and the context of surrounding sentences. A common drawback of text simplification models trained on such datasets is that they behave conservatively, performing mostly deletion, and rarely paraphrase (Alva-Manchego et al., 2017). Moreover, WIKILARGE is the concatenation of three early datasets (Zhu et al., 2010; Woodsend and Lapata, 2011; Coster and Kauchak, 2011) that are extracted from Wikipedia dumps and are known to contain many errors (Xu et al., 2015).
To address these problems, we create the first high-quality manually annotated sentence-aligned datasets: NEWSELA-MANUAL with 50 article sets, and WIKI-MANUAL with 500 article pairs. We design a novel neural CRF alignment model, which utilizes fine-tuned BERT to measure semantic similarity and leverages the similar order of content be
1Code and data are available at: https://github. com/choojiang06/wiki-auto.Newsela data need to be requested at: https://newsela.com/data/.
Hwang et al. (2015) annotated 46 article pairs from Simple-Normal Wikipedia corpus; however, its annotation is noisy, and it contains many sentence splitting errors.
7943
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943-7960 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: An example of sentence alignment between an original news article (right) and its simplified version (left) in Newsela. The label  $a_{i}$  for each simple sentence  $s_{i}$  is the index of complex sentence  $c_{a_{i}}$  it aligns to.
tween parallel documents, combined with an effective paragraph alignment algorithm. Experiments show that our proposed method outperforms all the previous monolingual sentence alignment approaches (Šajner et al., 2018; Paetzold et al., 2017; Xu et al., 2015) by more than 5 points in F1.
By applying our alignment model to all the 1,882 article sets in Newsela and 138,095 article pairs in Wikipedia dump, we then construct two new simplification datasets, NEWSELA-AUTO (666,645 sentence pairs) and WIKI-AUTO (488,332 sentence pairs). Our new datasets with improved quantity and quality facilitate the training of complex seq2seq models. A BERT-initialized Transformer model trained on our datasets outperforms the state-of-the-art by  $3.4\%$  in terms of SARI, the main automatic metric for text simplification. Our simplification model produces  $25\%$  more rephrasing than those trained on the existing datasets. Our contributions include:
1. Two manually annotated datasets that enable the first systematic study for training and evaluating monolingual sentence alignment;
2. A neural CRF sentence alinger and a paragraph alignment algorithm that employ fine-tuned BERT to capture semantic similarity and take advantage of the sequential nature of parallel documents;
3. Two automatically constructed text simplification datasets which are of higher quality and 4.7 and 1.6 times larger than the existing datasets in their respective domains;
4. A BERT-initialized Transformer model for automatic text simplification, trained on our datasets, which establishes a new state-of-the-art in both automatic and human evaluation.
2 Neural CRF Sentence Aligner
We propose a neural CRF sentence alignment model, which leverages the similar order of content presented in parallel documents and captures editing operations across multiple sentences, such as splitting and elaboration (see Figure 1 for an example). To further improve the accuracy, we first align paragraphs based on semantic similarity and vicinity information, and then extract sentence pairs from these aligned paragraphs. In this section, we describe the task setup and our approach.
Given a simple article (or paragraph)  $S$  of  $m$  sentences and a complex article (or paragraph)  $C$  of  $n$  sentences, for each sentence  $s_i$  ( $i \in [1, m]$ ) in the simple article, we aim to find its corresponding sentence  $c_{a_i}$  ( $a_i \in [0, n]$ ) in the complex article. We use  $a_i$  to denote the index of the aligned sentence, where  $a_i = 0$  indicates that sentence  $s_i$  is not aligned to any sentence in the complex article. The full alignment  $a$  between article (or paragraph) pair  $S$  and  $C$  can then be represented by a sequence of alignment labels  $\mathbf{a} = (a_1, a_2, \dots, a_m)$ . Figure 1 shows an example of alignment labels. One specific aspect of our CRF model is that it uses a varied number of labels for each article (or paragraph) pair rather than a fixed set of labels.
2.2 Neural CRF Sentence Alignment Model
We learn  $P(\mathbf{a}|S,C)$ , the conditional probability of alignment a given an article pair  $(S,C)$ , using
7944
linear-chain conditional random field:
$$
\begin{array}{l} P (\mathbf {a} | S, C) = \frac {\exp (\Psi (\mathbf {a} , S , C))}{\sum_ {\mathbf {a} \in \mathcal {A}} \exp (\Psi (\mathbf {a} , S , C))} \\ = \frac {\exp \left(\sum_ {i = 1} ^ {| S |} \psi \left(a _ {i} , a _ {i - 1} , S , C\right)\right)}{\sum_ {a \in \mathcal {A}} \exp \left(\sum_ {i = 1} ^ {| S |} \psi \left(a _ {i} , a _ {i - 1} , S , C\right)\right)} \tag {1} \\ \end{array}
$$
where  $|S| = m$  denotes the number of sentences in article  $S$ . The score  $\sum_{i=1}^{|S|} \psi(a_i, a_{i-1}, S, C)$  sums over the sequence of alignment labels  $\mathbf{a} = (a_1, a_2, \ldots, a_m)$  between the simple article  $S$  and the complex article  $C$ , and could be decomposed into two factors as follows:
$$
\psi \left(a _ {i}, a _ {i - 1}, S, C\right) = \operatorname {s i m} \left(s _ {i}, c _ {a _ {i}}\right) + T \left(a _ {i}, a _ {i - 1}\right) \tag {2}
$$
where  $\text{sim}(s_i, c_{a_i})$  is the semantic similarity score between the two sentences, and  $T(a_i, a_{i-1})$  is a pairwise score for alignment label transition that  $a_i$  follows  $a_{i-1}$ .
Semantic Similarity A fundamental problem in sentence alignment is to measure the semantic similarity between two sentences  $s_i$  and  $c_j$ . Prior work used lexical similarity measures, such as Jaccard similarity (Xu et al., 2015), TF-IDF (Paetzold et al., 2017), and continuous n-gram features (Šajner et al., 2018). In this paper, we fine-tune BERT (Devlin et al., 2019) on our manually labeled dataset (details in §3) to capture semantic similarity.
Alignment Label Transition In parallel documents, the contents of the articles are often presented in a similar order. The complex sentence  $c_{a_i}$  that is aligned to  $s_i$ , is often related to the complex sentences  $c_{a_{i-1}}$  and  $c_{a_{i+1}}$ , which are aligned to  $s_{i-1}$  and  $s_{i+1}$ , respectively. To incorporate this intuition, we propose a scoring function to model the transition between alignment labels using the following features:
$$
g _ {1} = \left| a _ {i} - a _ {i - 1} \right|
$$
$$
g _ {2} = \mathbb {1} (a _ {i} = 0, a _ {i - 1} \neq 0) \tag {3}
$$
$$
g _ {3} = \mathbb {1} (a _ {i} \neq 0, a _ {i - 1} = 0)
$$
$$
g _ {4} = \mathbb {1} \left(a _ {i} = 0, a _ {i - 1} = 0\right)
$$
where  $g_{1}$  is the absolute distance between  $a_{i}$  and  $a_{i - 1}$ ,  $g_{2}$  and  $g_{3}$  denote if the current or prior sentence is not aligned to any sentence, and  $g_{4}$  indicates whether both  $s_i$  and  $s_{i - 1}$  are not aligned to
any sentences. The score is computed as follows:
$$
T \left(a _ {i}, a _ {i - 1}\right) = \operatorname {F F N N} \left(\left[ g _ {1}, g _ {2}, g _ {3}, g _ {4} \right]\right) \tag {4}
$$
where  $[,]$  represents concatenation operation and FFNN is a 2-layer feedforward neural network. We provide more implementation details of the model in Appendix A.1.
2.3 Inference and Learning
During inference, we find the optimal alignment  $\hat{\mathbf{a}}$ :
$$
\hat {\mathbf {a}} = \underset {\mathbf {a}} {\operatorname {a r g m a x}} P (\mathbf {a} | S, C) \tag {5}
$$
using Viterbi algorithm in  $\mathcal{O}(mn^2)$  time. During training, we maximize the conditional probability of the gold alignment label  $\mathbf{a}^{*}$ :
$$
\begin{array}{l} \log P (\mathbf {a} ^ {*} | S, C) = \Psi (\mathbf {a} ^ {*}, S, C) - \\ \log \sum_ {\mathbf {a} \in \mathcal {A}} \exp (\Psi (\mathbf {a}, S, C)) \tag {6} \\ \end{array}
$$
The second term sums the scores of all possible alignments and can be computed using forward algorithm in  $\mathcal{O}(mn^2)$  time as well.
2.4 Paragraph Alignment
Both accuracy and computing efficiency can be improved if we align paragraphs before aligning sentences. In fact, our empirical analysis revealed that sentence-level alignments mostly reside within the corresponding aligned paragraphs (details in §4.4 and Table 3). Moreover, aligning paragraphs first provides more training instances and reduces the label space for our neural CRF model.
We propose Algorithm 1 and 2 for paragraph alignment. Given a simple article  $S$  with  $k$  paragraphs  $S = (S_{1}, S_{2}, \ldots, S_{k})$  and a complex article  $C$  with  $l$  paragraphs  $C = (C_{1}, C_{2}, \ldots, C_{l})$ , we first apply Algorithm 1 to calculate the semantic similarity matrix simP between paragraphs by averaging or maximizing over the sentence-level similarities (§2.2). Then, we use Algorithm 2 to generate the paragraph alignment matrix alignP. We align paragraph pairs if they satisfy one of the two conditions: (a) having high semantic similarity and appearing in similar positions in the article pair (e.g., both at the beginning), or (b) two continuous paragraphs in the complex article having relatively high semantic similarity with one paragraph in the simple side, (e.g., paragraph splitting or fusion). The difference of relative position in documents
7945
is defined as  $d(i,j) = \left|\frac{i}{k} -\frac{j}{l}\right|$ , and the thresholds  $\tau_{1} - \tau_{5}$  in Algorithm 2 are selected using the dev set. Finally, we merge the neighbouring paragraphs which are aligned to the same paragraph in the simple article before feeding them into our neural CRF aligner. We provide more details in Appendix A.1.
3 Constructing Alignment Datasets
To address the lack of reliable sentence alignment for Newsela (Xu et al., 2015) and Wikipedia (Zhu et al., 2010; Woodsend and Lapata, 2011), we designed an efficient annotation methodology to first manually align sentences between a few complex and simple article pairs. Then, we automatically aligned the rest using our alignment model trained on the human annotated data. We created two sentence-aligned parallel corpora (details in §5), which are the largest to date for text simplification.
3.1 Sentence Aligned Newsela Corpus
Newsela corpus (Xu et al., 2015) consists of 1,932 English news articles where each article (level 0) is
Table 1: Statistics of our manually and automatically created sentence alignment annotations on Newsela.  $\dagger$  This number includes all complex-simple sentence pairs (including aligned, partially-aligned, or not-aligned) across all 10 combinations of 5 readability levels (level 0-4), of which 20,343 sentence pairs between adjacent readability levels were manually annotated and the rest of labels were derived.
re-written by professional editors into four simpler versions at different readability levels (level 1-4). We annotate sentence alignments for article pairs at adjacent readability levels (e.g., 0-1, 1-2) as the alignments between non-adjacent levels (e.g., 0-2) can be then derived automatically. To ensure efficiency and quality, we designed the following three-step annotation procedure:
1. Align paragraphs using CATS toolkit (Štajner et al., 2018), and then correct the automatic paragraph alignment errors by two in-house annotators. $^3$  Performing paragraph alignment as the first step significantly reduces the number of sentence pairs to be annotated from every possible sentence pair to the ones within the aligned paragraphs. We design an efficient visualization toolkit for this step, for which a screenshot can be found in Appendix E.2.
2. For each sentence pair within the aligned paragraphs, we ask five annotators on the Figure
<sup>3</sup>We consider any sentence pair not in the aligned paragraph pairs as not-aligned. This assumption leads to a small number of missing sentence alignments, which are manually corrected in Step 3.
7946
Figure 2: Manual inspection of 100 random sentence pairs from our corpora (NEWSELA-AUTO and WIKI-AUTO) and the existing Newsela (Xu et al., 2015) and Wikipedia (Zhang and Lapata, 2017) corpora. Our corpora contain at least  $44\%$  more complex rewrites (Deletion + Paraphrase or Splitting + Paraphrase) and  $27\%$  less defective pairs (Not Aligned or Not Simpler).
Eight<sup>4</sup> crowdsourcing platform to classify into one of the three categories: aligned, partially-aligned, or not-aligned. We provide the annotation instructions and interface in Appendix E.1. We require annotators to spend at least ten seconds per question and embed one test question in every five questions. Any worker whose accuracy drops below  $85\%$  on test questions is removed. The inter-annotator agreement is 0.807 measured by Cohen's kappa (Artstein and Poesio, 2008).
3. We have four in-house annotators (not authors) verify the crowdsourced labels.
We manually aligned 50 article groups to create the NEWSELA-MANUAL dataset with a 35/5/10 split for train/dev/test, respectively. We trained our aligner on this dataset (details in §4), then automatically aligned sentences in the remaining 1,882 article groups in Newsela (Table 1) to create a new sentence-aligned dataset, NEWSELA-AUTO, which consists of 666k sentence pairs predicted as aligned and partially-aligned. NEWSELA-AUTO is considerably larger than the previous NEWSELA (Xu et al., 2015) dataset of 141,582 pairs, and contains  $44\%$  more interesting rewrites (i.e., rephrasing and splitting cases) as shown in Figure 2.
3.2 Sentence Aligned Wikipedia Corpus
We also create a new version of Wikipedia corpus by aligning sentences between English Wikipedia and Simple English Wikipedia. Previous work (Xu et al., 2015) has shown that Wikipedia is much noisier than the Newsela corpus. We provide this dataset in addition to facilitate future research.
We first extract article pairs from English and Simple English Wikipedia by leveraging Wikidata, a well-maintained database that indexes named entities (and events etc.) and their Wikipedia pages in different languages. We found this method to be more reliable than using page titles (Coster and Kauchak, 2011) or cross-lingual links (Zhu et al., 2010; Woodsend and Lapata, 2011), as titles can be ambiguous and cross-lingual links may direct to a disambiguation or mismatched page (more details in Appendix B). In total, we extracted 138,095 article pairs from the 2019/09 Wikipedia dump, which is two times larger than the previous datasets (Coster and Kauchak, 2011; Zhu et al., 2010) of only  $60\sim 65\mathrm{k}$  article pairs, using an improved version of the WikiExtractor library.
Then, we crowdsourced the sentence alignment annotations for 500 randomly sampled document pairs (10,123 sentence pairs total). As document length in English and Simple English Wikipedia articles vary greatly,[6] we designed the following annotation strategy that is slightly different from Newsela. For each sentence in the simple article, we select the sentences with the highest similarity scores from the complex article for manual annotation, based on four similarity measures: lexical similarity from CATS (Štajner et al., 2018), cosine similarity using TF-IDF (Paetzold et al., 2017), cosine similarity between BERT sentence embeddings, and alignment probability by a BERT model fine-tuned on our NEWSELA-MANUAL data (§3.1). As these four metrics may rank the same sentence at the top, on an average, we collected 2.13 complex sentences for every simple sentence and annotated the alignment label for each sentence pair. Our pilot study showed that this method captured  $93.6\%$  of the aligned sentence pairs. We named this manually labeled dataset WIKI-MANUAL with a train/dev/test split of 350/50/100 article pairs.
Finally, we trained our alignment model on this
$^{5}$ https://github.com/attardi/wikiextractor
The average number of sentences in an article is  $9.2 \pm 16.5$  for Simple English Wikipedia and  $74.8 \pm 94.4$  for English Wikipedia.
<sup>4</sup>https://www_figure-eight.com/
7947
Task 1 (aligned&partial vs. others)	Task 2 (aligned vs. others)
Precision	Recall	F1	Precision	Recall	F1
Similarity-based models
Jaccard (Xu et al., 2015)	94.93	76.69	84.84	73.43	75.61	74.51
TF-IDF (Paetzold et al., 2017)	96.24	83.05	89.16	66.78	69.69	68.20
LR (Štajner et al., 2018)	93.11	84.96	88.85	73.21	74.74	73.97
Similarity-based models w/ alignment strategy (previous SOTA)
JaccardAlign (Xu et al., 2015)	98.66	67.58	80.22†	51.34	86.76	64.51†
MASSAlign (Paetzold et al., 2017)	95.49	82.27	88.39†	40.98	87.11	55.74†
CATS (Štajner et al., 2018)	88.56	91.31	89.92†	38.29	97.39	54.97†
Our CRF Aligner	97.86	93.43	95.59	87.56	89.55	88.54
Table 2: Performance of different sentence alignment methods on the NEWSELA-MANUAL test set.  $\dagger$  Previous work was designed only for Task 1 and used alignment strategy (greedy algorithm or dynamic programming) to improve either precision or recall.
Task 1	Task 2
P	R	F1	P	R	F1
Neural sentence pair models
Inferent	92.8	69.7	79.6	87.8	74.0	80.3
ESIM	91.5	71.2	80.0	82.5	73.7	77.8
BERTScore	90.6	76.5	83.0	83.2	74.3	78.5
BERTembedding	84.7	53.0	65.2	77.0	74.7	75.8
BERTfinetune	93.3	84.3	88.6	90.2	80.0	84.8
+ ParaAlign	98.4	84.2	90.7	91.9	79.0	85.0
Neural CRF aligner
Our CRF Aligner	96.5	90.1	93.2	88.6	87.7	88.1
+ gold ParaAlign	97.3	91.1	94.1	88.9	88.0	88.4
Table 3: Ablation study of our aligner on dev set.
annotated dataset to automatically align sentences for all the 138,095 document pairs (details in Appendix B). In total, we yielded 604k non-identical aligned and partially-aligned sentence pairs to create the WIKI-AUTO dataset. Figure 2 illustrates that WIKI-AUTO contains  $75\%$  less defective sentence pairs than the old WIKILARGE (Zhang and Lapata, 2017) dataset.
4 Evaluation of Sentence Alignment
In this section, we present experiments that compare our neural sentence alignment against the state-of-the-art approaches on NEWSELA-MANUAL (§3.1) and WIKI-MANUAL (§3.2) datasets.
4.1 Existing Methods
We compare our neural CRF aligner with the following baselines and state-of-the-art approaches:
1. Three similarity-based methods: Jaccard similarity (Xu et al., 2015), TF-IDF cosine similarity (Paetzold et al., 2017) and a logistic regression classifier trained on our data with lexical features from Štajner et al. (2018).
2. JaccardAlign (Xu et al., 2015), which uses Jaccard coefficient for sentence similarity and a greedy approach for alignment.
3. MASSAlign (Paetzold et al., 2017), which
combines TF-IDF cosine similarity with a vicinity-driven dynamic programming algorithm for alignment.
4. CATS toolkit (Stajner et al., 2018), which uses character n-gram features for sentence similarity and a greedy alignment algorithm.
We report Precision, Recall and F1 on two binary classification tasks: aligned + partially-aligned vs. not-aligned (Task 1) and aligned vs. partially-aligned + not-aligned (Task 2). It should be noted that we excluded identical sentence pairs in the evaluation as they are trivial to classify.
Table 2 shows the results on NEWSELA-MANUAL test set. For similarity-based methods, we choose a threshold based on the maximum F1 on the dev set. Our neural CRF aligner outperforms the state-of-the-art approaches by more than 5 points in F1. In particular, our method performs better than the previous work on partial alignments, which contain many interesting simplification operations, such as sentence splitting and paraphrasing with deletion.
Similarly, our CRF alignment model achieves 85.1 F1 for Task 1 (aligned + partially-aligned vs. not-aligned) on the WIKI-MANUAL test set. It outperforms one of the previous SOTA approaches CATS (Stajner et al., 2018) by 15.1 points in F1. We provide more details in Appendix C.
We analyze the design choices crucial for the good performance of our alignment model, namely CRF component, the paragraph alignment and the BERT-based semantic similarity measure. Table 3 shows the importance of each component with a series of ablation experiments on the dev set.
7948
Newsela	Wikipedia
Auto	Old	Auto	Old
# of article pairs	13k	7.9k	138k	65k
# of sent. pairs (train)	394k	94k	488k	298k
# of sent. pairs (dev)	43k	1.1k	2k	2k
# of sent. pairs (test)	44k	1k	359	359
avg. sent. len (complex)	25.4	25.8	26.6	25.2
avg. sent. len (simple)	13.8	15.7	18.7	18.5
Table 4: Statistics of our newly constructed parallel corpora for sentence simplification compared to the old datasets (Xu et al., 2015; Zhang and Lapata, 2017).
CRF Model Our aligner achieves 93.2 F1 and 88.1 F1 on Task 1 and 2, respectively, which is around 3 points higher than its variant without the CRF component (BERT finetune + ParaAlign). Modeling alignment label transitions and sequential predictions helps our neural CRF aligner to handle sentence splitting cases better, especially when sentences undergo dramatic rewriting.
Paragraph Alignment Adding paragraph alignment  $(\mathrm{BERT}_{\mathrm{finetune}} + \mathrm{ParaAlign})$  improves the precision on Task 1 from 93.3 to 98.4 with a negligible decrease in recall when compared to not aligning paragraphs  $(\mathrm{BERT}_{\mathrm{finetune}})$ . Moreover, paragraph alignments generated by our algorithm (Our Aligner) perform close to the gold alignments (Our Aligner + gold ParaAlign) with only 0.9 and 0.3 difference in F1 on Task 1 and 2, respectively.