SlowGuess/NAACL_2021_OCR huggingface dataset:
Mounica Maddela<sup>1</sup>, Fernando Alva-Manchego<sup>2</sup>, Wei Xu<sup>1</sup>
<sup>1</sup>School of Interactive Computing, Georgia Institute of Technology
$^{2}$ Department of Computer Science, University of Sheffield
{mounica.maddela, wei.xu}@cc.gatech.edu f.alva@sheffield.ac.uk
Text Simplification improves the readability of sentences through several rewriting transformations, such as lexical paraphrasing, deletion, and splitting. Current simplification systems are predominantly sequence-to-sequence models that are trained end-to-end to perform all these operations simultaneously. However, such systems limit themselves to mostly deleting words and cannot easily adapt to the requirements of different target audiences. In this paper, we propose a novel hybrid approach that leverages linguistically-motivated rules for splitting and deletion, and couples them with a neural paraphrasing model to produce varied rewriting styles. We introduce a new data augmentation method to improve the paraphrasing capability of our model. Through automatic and manual evaluations, we show that our proposed model establishes a new state-of-the-art for the task, paraphrasing more often than the existing systems, and can control the degree of each simplification operation applied to the input texts.<sup>1</sup>
Text Simplification aims to improve the readability of texts with simpler grammar and word choices while preserving meaning (Saggion, 2017). It provides reading assistance to children (Kajiwara et al., 2013), non-native speakers (Petersen and Ostendorf, 2007; Pellow and Eskenazi, 2014; Paetzold, 2016), and people with reading disabilities (Rello et al., 2013). It also helps with downstream natural language processing tasks, such as parsing (Chandrasekar et al., 1996), semantic role labelling (Vickrey and Koller, 2008), information extraction (Miwa et al., 2010), and machine translation (MT, Chen et al., 2012; Štajner and Popovic, 2016).
Since 2016, nearly all text simplification systems have been sequence-to-sequence (seq2seq)
OLen	%new	%eq	%split
Complex (input)	20.7	0.0	100.0	0.0
Narayan and Gardent (2014)†	10.4	0.7	0.8	0.4
Zhang and Lapata (2017)†	13.8	8.1	16.8	0.0
Dong et al. (2019)†	10.9	8.4	4.6	0.0
Kriz et al. (2019)†	10.8	11.2	1.2	0.0
LSTM	17.0	6.1	28.4	1.2
Our Model	17.1	17.0	3.0	31.8
Simple (reference)	17.9	29.0	0.0	30.0
Table 1: Output statistics of 500 random sentences from the Newsela test set. Existing systems rely on deletion and do not paraphrase well. OLen, %new, %eq and %split denote the average output length, percentage of new words added, percentage of system outputs that are identical to the inputs, and percentage of sentence splits, respectively. †We used the system outputs shared by their authors.
models trained end-to-end, which have greatly increased the fluency of the outputs (Zhang and Lapata, 2017; Nisioi et al., 2017; Zhao et al., 2018; Kriz et al., 2019; Dong et al., 2019; Jiang et al., 2020). However, these systems mostly rely on deletion and tend to generate very short outputs at the cost of meaning preservation (Alva-Manchego et al., 2017). Table 1 shows that they neither split sentences nor paraphrase well as reflected by the low percentage of splits  $(< 1\%)$  and new words introduced  $(< 11.2\%)$ . While deleting words is a viable (and the simplest) way to reduce the complexity of sentences, it is suboptimal and unsatisfying. Professional editors are known to use a sophisticated combination of deletion, paraphrasing, and sentence splitting to simplify texts (Xu et al., 2015).
Another drawback of these end-to-end neural systems is the lack of controllability. Simplification is highly audience dependant, and what constitutes simplified text for one group of users may not be acceptable for other groups (Xu et al., 2015; Lee and Yeung, 2018). An ideal simplification system should be able to generate text with varied characteristics, such as different lengths, readability levels, and number of split sentences, which can be difficult to control in end-to-end systems.
1Our code and data are available at https://github. com/mounicam/controllable_simplification.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3536-3553 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Overview of our proposed model for text simplification, which can perform a controlled combination of sentence splitting, deletion, and paraphrasing.
To address these issues, we propose a novel hybrid approach that combines linguistically-motivated syntactic rules with data-driven neural models to improve the diversity and controllability of the simplifications. We hypothesize that the seq2seq generation model will learn lexical and structural paraphrases more efficiently from the parallel corpus, when we offload some of the burden of sentence splitting (e.g., split at comma) and deletion (e.g., remove trailing preposition phrases) decisions to a separate component. Previous hybrid approaches for simplification (Narayan and Gardent, 2014; Siddharthan and Mandya, 2014; Sulem et al., 2018c) used splitting and deletion rules in a deterministic step before applying an MT-based paraphrasing model. In contrast, our approach provides a more flexible and dynamic integration of linguistic rules with the neural models through ranking and data augmentation (Figure 1).
We compare our method to several state-of-the-art systems in both automatic and human evaluations. Our model achieves overall better performance measured by SARI (Xu et al., 2016) and other metrics, showing that the generated outputs are more similar to those written by human editors. We also demonstrate that our model can control the extent of each simplification operation by: (1) imposing a soft constraint on the percentage of words to be copied from the input in the seq2seq model, thus limiting lexical paraphrasing; and (2) selecting candidates that underwent a desired amount of splitting and/or deletion. Finally, we create a new test dataset with multiple human references for Newsela (Xu et al., 2015), the widely used text simplification corpus, to specifically evaluate lexical paraphrasing.
Figure 1 shows an overview of our hybrid approach. We combine linguistic rules with data-driven neural models to improve the controllability and diversity of the outputs. Given an input complex sentence  $\mathbf{x}$ , we first generate a set of intermediate simplifications  $V = \{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n\}$  that have undergone splitting and deletion (§2.1). These intermediate sentences are then used for two purposes: (1) Selected by a pairwise neural ranking model (§2.2) based on the simplification quality and then rewritten by the paraphrasing component; (2) Used for data augmentation to improve the diversity of the paraphrasing model (§2.3).
2.1 Splitting and Deletion
We leverage the state-of-the-art system for structural simplification, called DisSim (Niklaus et al., 2019), to generate candidate simplifications that focus on splitting and deletion. The English version of DisSim applies 35 hand-crafted grammar rules to break down a complex sentence into a set of hierarchically organized sub-sentences (see Figure 1 for an example). We choose a rule-based approach for sentence splitting because it works really well. In our pilot experiments, DisSim successfully split  $92\%$  of 100 complex sentences from the training data with more than 20 words, and introduced errors for only  $6.8\%$  of these splits. We consider these sub-sentences as candidate simplifications for the later steps, except those that are extremely short or long (compression ratio  $\notin [0.5, 1.5]$ ). The compression ratio is calculated as the number of
$^{2}$ https://github.com/Lambda-3/DiscourseSimplification
words in a candidate simplification  $\mathbf{v}^i$  (which may contain one or more sub-sentences) divided by that of the original sentence  $\mathbf{x}$ .
To further increase the variety of generated candidates, we supplement DisSim with a Neural Deletion and Split module trained on the text simplification corpus (§3.1). We use a Transformer seq2seq model with the same configuration as the base model for paraphrasing (§2.3). Given the input sentence  $\mathbf{x}$ , we constrain the beam search to generate 10 outputs with splitting and another 10 outputs without splitting. Then, we select the outputs that do not deviate substantially from  $\mathbf{x}$  (i.e., Jaccard similarity  $> 0.5$ ). We add outputs from the two systems to the candidate pool  $V$ .
2.2 Candidate Ranking
We design a neural ranking model to score all the candidates that underwent splitting and deletion,  $V = \{\mathbf{v}_1,\mathbf{v}_2,\dots ,\mathbf{v}_n\}$ , then feed the top-ranked one to the lexical paraphrasing model for the final output. We train the model on a standard text simplification corpus consisting of pairs of complex sentence  $\mathbf{x}$  and manually simplified reference  $\mathbf{y}$ .
Scoring Function. To assess the "goodness" of each candidate  $\mathbf{v}_i$  during training, we define the gold scoring function  $g^*$  as a length-penalized BERTscore:
$$
\begin{array}{l} g ^ {*} (\mathbf {v} _ {i}, \mathbf {y}) = e ^ {- \lambda | | \phi_ {\mathbf {v} _ {i}} - \phi_ {\mathbf {y}} | |} \times \\ B E R T S c o r e (\mathbf {v} _ {i}, \mathbf {y}) \quad (1) \\ \end{array}
$$
BERTScore (Zhang et al., 2020b) is a text similarity metric that uses BERT (Devlin et al., 2019) embeddings to find soft matches between word pieces (Wu et al., 2016) instead of exact string matching. We introduce a length penalty to favor the candidates that are of similar length to the human reference  $\mathbf{y}$  and penalize those that deviate from the target compression ratio  $\phi_{\mathbf{y}}$ .  $\lambda$  defines the extent of penalization and is set to 1 in our experiments.  $\phi_{\mathbf{v}_i}$  represents the compression ratios of  $\mathbf{v}_i$  compared to the input  $\mathbf{x}$ . In principle, other similarity metrics can also be used for scoring.
Pairwise Ranking Model. We train the ranking model in a pairwise setup since BERTScore is sensitive to the relative rather than absolute similarity, when comparing multiple candidates with the same reference. We transform the gold ranking of  $V$ $(|V| = n)$  into  $n^2$  pairwise comparisons for every
candidate pair, and learn to minimize the pairwise ranking violations using hinge loss:
$$
L _ {M R} = \frac {1}{m} \sum_ {k = 1} ^ {m} \frac {1}{n _ {k} ^ {2}} \sum_ {i = 1} ^ {n _ {k}} \sum_ {j = 1, i \neq j} ^ {n _ {k}} \max (0, 1 - l _ {i j} ^ {k} d _ {i j} ^ {k})
$$
$$
d _ {i j} ^ {k} = g (\mathbf {v} _ {i} ^ {k}) - g (\mathbf {v} _ {j} ^ {k})
$$
$$
l _ {i j} ^ {k} = \operatorname {s i g n} \left(g ^ {*} \left(\mathbf {v} _ {i} ^ {k}, \mathbf {y} ^ {k}\right) - g ^ {*} \left(\mathbf {v} _ {j} ^ {k}, \mathbf {y} ^ {k}\right)\right) \tag {2}
$$
where  $g(.)$  is a feedforward neural network,  $m$  is the number of training complex-simple sentence pairs,  $k$  is the index of training examples, and  $n_k$  represents the number of generated candidates (§2.1). On average,  $n_k$  is about 14.5 for a sentence of 30 words, and can be larger for longer sentences. We consider 10 randomly sampled candidates for each complex sentence during training.
Features. For the feedforward network  $g(\cdot)$ , we use the following features: number of words in  $\mathbf{v}_i$  and  $\mathbf{x}$ , compression ratio of  $\mathbf{v}_i$  with respect to  $\mathbf{x}$ , Jaccard similarity between  $\mathbf{v}_i$  and  $\mathbf{x}$ , the rules applied on  $\mathbf{x}$  to obtain  $\mathbf{v}_i$ , and the number of rule applications. We vectorize all the real-valued features using Gaussian binning (Maddela and Xu, 2018), which has shown to help neural models trained on numerical features (Liu et al., 2016; Sil et al., 2017; Zhong et al., 2020). We concatenate these vectors before feeding them to the ranking model. We score each candidate  $\mathbf{v}_i$  separately and rank them in the decreasing order of  $g(\mathbf{v}_i)$ . We provide implementation details in Appendix A.
2.3 Paraphrase Generation
We then paraphrase the top-ranked candidate  $\hat{\mathbf{v}}\in V$  to generate the final simplification output  $\hat{\mathbf{y}}$ . Our paraphrase generation model can explicitly control the extent of lexical paraphrasing by specifying the percentage of words to be copied from the input sentence as a soft constraint. We also introduce a data augmentation method to encourage our model to generate more diverse outputs.
Base Model. Our base generation model is a Transformer encoder-decoder initialized by the BERT checkpoint (?), which achieved the best reported performance on text simplification in the recent work (Jiang et al., 2020). We enhance this model with an attention-based copy mechanism to encourage lexical paraphrasing, while remaining faithful to the input.
Copy Control. Given the input candidate  $\hat{\mathbf{v}} = (\hat{v}_1, \hat{v}_2, \dots, \hat{v}_l)$  of  $l$  words and the percentage of copying  $cp \in (0, 1]$ , our goal is to paraphrase the rest of  $(1 - cp) \times l$  words in  $\hat{\mathbf{v}}$  to a simpler version. To achieve this, we convert  $cp$  into a vector of the same dimension as BERT embeddings using Gaussian binning (Maddela and Xu, 2018) and add it to the beginning of the input sequence  $\hat{\mathbf{v}}$ . The Transformer encoder then produces a sequence of context-aware hidden states  $\mathbf{H} = (\mathbf{h}_1, \mathbf{h}_2 \dots \mathbf{h}_l)$ , where  $\mathbf{h}_i$  corresponds to the hidden state of  $\hat{v}_i$ . Each  $\mathbf{h}_i$  is fed into the copy network which predicts the probability  $p_i$  that word  $\hat{v}_i$  should be copied to output. We create a new hidden state  $\bar{\mathbf{h}}_i$  by adding  $\mathbf{h}_i$  to a vector  $\mathbf{u}$  scaled according to  $p_i$ . In other words, the scaled version of  $\mathbf{u}$  informs the decoder whether the word should be copied. A single vector  $\mathbf{u}$  is used across all sentences and hidden states, and is randomly initialized then updated during training. More formally, the encoding process can be described as follows:
$$
\left(\mathbf {h} _ {1}, \mathbf {h} _ {2}, \dots , \mathbf {h} _ {l}\right) = e n c o d e r ([ c p; \hat {v} _ {1}, \hat {v} _ {2}, \dots , \hat {v} _ {l} ])
$$
$$
\overline {{\mathbf {h}}} _ {i} = \mathbf {h} _ {\mathrm {i}} + p _ {i} \cdot \mathbf {u},
$$
$$
\bar {\mathbf {H}} = \left(\bar {\mathbf {h}} _ {1}, \bar {\mathbf {h}} _ {2}, \dots , \bar {\mathbf {h}} _ {l}\right) \tag {3}
$$
The Transformer decoder generates the output sequence from  $\bar{\mathbf{H}}$ . Our copy mechanism is incorporated into the encoder rather than copying the input words during the decoding steps (Gu et al., 2016; See et al., 2017). Unless otherwise specified, we use the average copy ratio of the training dataset, 0.7, for our experiments.
Multi-task Training. We train the paraphrasing model and the copy network in a multi-task learning setup, where predicting whether a word should be copied serves as an auxiliary task. The gold labels for this task are obtained by checking if each word in the input sentence also appears in the human reference. When a word occurs multiple times in the input, we rely on the monolingual word alignment results from JacanaAlign (Yao et al., 2013) to determine which occurrence is the one that gets copied. We train the Transformer model and the copy network jointly by minimizing the cross-entropy loss for both decoder generation and binary word classification. We provide implementation and training details in Appendix A.
Data Augmentation. The sentence pairs in the training corpus often exhibit a variable mix of splitting and deletion operations along with paraphras
ing (see Figure 1 for an example), which makes it difficult for the encoder-decoder models to learn paraphrases. Utilizing DisSim, we create additional training data that focuses on lexical paraphrasing
For each sentence pair  $\langle \mathbf{x},\mathbf{y}\rangle$ , we first generate a set of candidates  $V = \{\mathbf{v}_1,\mathbf{v}_2,\dots ,\mathbf{v}_n\}$  by applying DisSim to  $\mathbf{x}$ , as described in §2.1. Then, we select a subset of  $V$ , called  $V^{\prime} = \{\mathbf{v}_{1}^{\prime},\mathbf{v}_{2}^{\prime},\ldots ,\mathbf{v}_{n^{\prime}}^{\prime}\}$ $(V^{\prime}\in V)$  that are fairly close to the reference  $\mathbf{y}$  but have only undergone splitting and deletion. We score each candidate  $\mathbf{v}_i$  using the length-penalized BERTScore  $g^{*}(\mathbf{v}_{i},\mathbf{y})$  in Eq. (1), and discard those with scores lower than 0.5. While calculating  $g^{*}$  we set  $\phi_{\mathbf{y}}$  and  $\lambda$  to 1 and 2 respectively to favor candidates of similar length to the reference  $\mathbf{y}$ . We also discard the candidates that have different number of split sentences with respect to the reference. Finally, we train our model on the filtered candidate-reference sentence pairs  $\langle \mathbf{v}_1',\mathbf{y}\rangle$ ,  $\langle \mathbf{v}_2',\mathbf{y}\rangle$ ,  $\dots$ ,  $\langle \mathbf{v}_{n'}',\mathbf{y}\rangle$ , which focus on lexical paraphrasing, in addition to  $\langle \mathbf{x},\mathbf{y}\rangle$ .
2.4 Controllable Generation
We can control our model to concentrate on specific operations. For split- or delete-focused simplification, we select candidates with desirable length or number of splits during the candidate generation step. We perform only the paraphrase generation step for paraphrase-focused simplification. The paraphrasing model is designed specifically to paraphrase with minimal deletion and without splitting. It retains the length and the number of split sentences in the output, thus preserving the extent of deletion and splitting controlled in the previous steps. We control the degree of paraphrasing by changing the copy ratio.
In this section, we compare our approach to various sentence simplification models using both automatic and manual evaluations. We show that our model achieves a new state-of-the-art and can adapt easily to different simplification styles, such as paraphrasing and splitting without deletion.
3.1 Data and Experiment Setup
We train and evaluate our models on Newsela (Xu et al., 2015) $^3$  and Wikipedia copora (Zhu et al., 2010; Woodsend and Lapata, 2011; Coster and Kauchak, 2011). Newsela consists of 1,882 news
<sup>3</sup>https://newsela.com/data/
Models	SARI	add	keep	del	FK	SLen	OLen	CR	%split	s-BL	%new	%eq
Complex (input)	15.9	0.0	47.6	0.0	12.0	23.7	23.8	1.0	0.0	100.0	0.0	100.0
Simple (reference)	90.5	86.8	86.6	98.2	7.4	14.4	19.0	0.83	28.0	35.5	33.0	0.0
LSTM	35.0	1.6	45.5	57.8	8.9	17.6	17.9	0.8	1.9	66.5	5.0	20.2
Hybrid-NG	35.8	1.9	41.8	63.7	9.9	21.2	23.7	1.0	11.6	59.7	8.8	5.1
TransformerBERT	37.0	3.1	43.6	64.4	8.1	15.6	20.2	0.87	24.1	58.8	12.8	10.2
EditNTS	38.1	1.6	45.8	66.5	8.5	16.0	21.4	0.92	32.0	71.4	8.3	0.2
Our Model	38.7	3.3	42.9	70.0	7.9	15.8	20.1	0.86	23.9	48.7	16.2	0.4
Table 2: Automatic evaluation results on NEWSELA-AUTO test set. We report SARI, the main automatic metric for simplification, and its three edit scores namely precision for delete (del) and F1 scores for add and keep operations. We also report FKGL (FK), average sentence length (SLen), output length (OLen), compression ratio (CR), self-BLEU (s-BL), percentage of sentence splits (%split), average percentage of new words added to the output (%new), and percentage of sentences identical to the input (%eq). Bold typeface denotes the best performances (i.e., closest to the reference).
articles with each article rewritten by professional editors for students in different grades. We used the complex-simple sentence pairs automatically aligned by Jiang et al. (2020), called the NEWSELA-AUTO dataset. To capture sentence splitting, we joined the adjacent sentences in the simple article that are aligned to the same sentence in the complex article. Following Šajner et al. (2015), we removed the sentence pairs with high (>0.9) and low (<0.1) BLEU (Papineni et al., 2002) scores, which mostly correspond to the near identical and semantically divergent sentence pairs respectively. The final dataset consists of 259,778 train, 32,689 validation and 33,391 test complex-simple sentence pairs, where ~30% of pairs involve sentence splitting. Besides Newsela, we also provide the details of experiments on Wikipedia corpus in Appendix F, which show similar trends.
To demonstrate that our model can be controlled to generate diverse simplifications, we evaluate under the following settings: (i) Standard evaluation on the NEWSELA-AUTO test set similar to the methodology in the recent literature (Jiang et al., 2020; Dong et al., 2019; Zhang and Lapata, 2017), and (ii) Evaluation on different subsets of the NEWSELA-AUTO test set that concentrate on a specific operation. We selected 9,356 sentence pairs with sentence splits for split-focused evaluation. Similarly, we chose 9,511 sentence pairs with compression ratio  $< 0.7$  and without sentences splits to evaluate delete-focused simplification. We created a new dataset, called NEWSELATURK, to evaluate lexical paraphrasing. Similar to the WIKIPEDIA-TURK benchmark corpus (Xu et al., 2016), NEWSELA-TURK consists of human-written references focused on lexical para
phrasing. We first selected sentence pairs from the NEWSELA-AUTO test set of roughly similar length (compression ratio between 0.8 and 1.2) and no sentence splits because they more likely involve paraphrasing. Then, we asked Amazon Mechanical Turk workers to simplify the complex sentence without any loss in meaning.<sup>5</sup> To ensure the quality of simplifications, we manually selected the workers using the qualification test proposed in Alva-Manchego et al. (2020), during which the workers were asked to simplify three sentences. We selected top  $35\%$  of the 300 workers that participated in the test. We periodically checked the submissions and removed the bad workers. In the end, we collected 500 sentences with 4 references for each sentence.
3.2 Existing Methods
We use the following simplification approaches as baselines: (i) BERT- Initialized Transformer (?), where the encoder is initialized with  $\mathrm{BERT}_{\mathrm{base}}$  checkpoint and the decoder is randomly initialized. It is the current state-of-the-art for text simplification (Jiang et al., 2020). (ii) EditNTS (Dong et al., 2019), another state-of-the-art model that uses a neural programmer- interpreter (Reed and de Freitas, 2016) to predict the edit operation on each word, and then generates the simplified sentence. (iii) LSTM baseline, a vanilla encoder-decoder model used in Zhang and Lapata (2017). (iv) Hybrid-NG (Narayan and Gardent, 2014), one of the best existing hybrid systems that performs splitting and deletion using a probabilistic model and lexical substitution with a phrase-based machine translation system. We retrained all the models on the NEWSLA-AUTO dataset.
4We also provide results on 8,371 sentence pairs of NEWSELA-AUTO test set with compression ratio  $>0.9$  and no splits in Appendix D, which show similar trends.
We provide instructions in Appendix B https://github.com/yuedongP/EditNTS https://github.com/shashionggithub/ Sentence-Simplification-ACL14
Models	SARI	add	keep	del	FK	SLen	OLen	CR	%split	s-BL	%new	%eq
Complex (input)	22.3	0.0	67.0	0.0	12.8	23.3	23.5	1.0	0.0	100.0	0.0	100.0
Simple (reference)	62.3	44.8	68.3	73.9	11.1	23.8	23.5	1.01	0.0	48.5	24.1	0.0
Hybrid-NG	38.2	2.8	57.0	54.8	10.7	21.6	23.1	0.98	7.0	57.2	9.1	1.4
\( \text{Transformer}_{bert} \)	36.0	3.3	54.9	49.8	8.9	16.1	20.2	0.87	23.0	58.7	13.3	7.6
EditNTS	37.4	1.6	61.0	49.6	9.5	16.9	21.9	0.94	0.0	73.1	5.8	0.0
Our Model	38.1	3.9	55.1	55.5	8.8	16.6	20.2	0.86	19.6	50.4	15.7	0.0
Our Model (no split; cp = 0.6)	39.0	3.8	57.7	55.6	11.2	22.1	22.9	0.98	0.2	55.9	18.0	1.0
Our Model (no split; cp = 0.7)	41.0	3.4	63.1	56.6	11.5	22.2	22.9	0.98	0.0	69.4	10.4	4.2
Our Model (no split; cp = 0.8)	40.6	2.9	65.0	54.0	11.8	22.4	23.2	0.99	0.0	77.7	6.6	10.8