Sentence Simplification with Memory-Augmented Neural Networks

Abstract: Sentence simplification aims to simplify the content and structure of complex sentences, and thus make them easier to interpret for human readers, and easier to process for downstream NLP applications. Recent advances in neural machine translation have paved the way for novel approaches to the task. In this paper, we adapt an architecture with augmented memory capacities called Neural Semantic Encoders (Munkhdalai and Yu, 2017) for sentence simplification. Our experiments demonstrate the effectiveness of our approach on different simplification datasets, both in terms of automatic evaluation measures and human judgments.

The goal of sentence simplification is to compose complex sentences into simpler ones so that they are more comprehensible and accessible, while still retaining the original information content and meaning. Sentence simplification has a number of practical applications. On one hand, it provides reading aids for people with limited language proficiency BIBREF1 , BIBREF2 , or for patients with linguistic and cognitive disabilities BIBREF3 . On the other hand, it can improve the performance of other NLP tasks BIBREF4 , BIBREF5 , BIBREF6 . Prior work has explored monolingual machine translation (MT) approaches, utilizing corpora of simplified texts, e.g., Simple English Wikipedia (SEW), and making use of statistical MT models, such as phrase-based MT (PBMT) BIBREF7 , BIBREF8 , BIBREF9 , tree-based MT (TBMT) BIBREF10 , BIBREF11 , or syntax-based MT (SBMT) BIBREF12 .
Inspired by the success of neural MT BIBREF13 , BIBREF14 , recent work has started exploring neural simplification with sequence to sequence (Seq2seq) models, also referred to as encoder-decoder models. Nisioi et al. Nisioi:17 implemented a standard LSTM-based Seq2seq model and found that they outperform PBMT, SBMT, and unsupervised lexical simplification approaches. Zhang and Lapata BIBREF15 viewed the encoder-decoder model as an agent and employed a deep reinforcement learning framework in which the reward has three components capturing key aspects of the target output: simplicity, relevance, and fluency. The common practice for Seq2seq models is to use recurrent neural networks (RNNs) with Long Short-Term Memory BIBREF16 or Gated Recurrent Unit BIBREF17 for the encoder and decoder BIBREF18 , BIBREF15 . These architectures were designed to be capable of memorizing long-term dependencies across sequences. Nevertheless, their memory is typically small and might not be enough for the simplification task, where one is confronted with long and complicated sentences. In this study, we go beyond the conventional LSTM/GRU-based Seq2seq models and propose to use a memory-augmented RNN architecture called Neural Semantic Encoders (NSE). This architecture has been shown to be effective in a wide range of NLP tasks BIBREF0 . The contribution of this paper is twofold:
(1) First, we present a novel simplification model which is, to the best of our knowledge, the first model that use memory-augmented RNN for the task. We investigate the effectiveness of neural Seq2seq models when different neural architectures for the encoder are considered. Our experiments reveal that the NseLstm model that uses an NSE as the encoder and an LSTM as the decoder performed the best among these models, improving over strong simplification systems. (2) Second, we perform an extensive evaluation of various approaches proposed in the literature on different datasets. Results of both automatic and human evaluation show that our approach is remarkably effective for the task, significantly reducing the reading difficulty of the input, while preserving grammaticality and the original meaning. We further discuss some advantages and disadvantages of these approaches.
Our approach is based on an attention-based Seq2seq model BIBREF19 (Figure FIGREF1 ). Given a complex source sentence INLINEFORM0 , the model learns to generate its simplified version INLINEFORM1 . The encoder reads through INLINEFORM2 and computes a sequence of hidden states INLINEFORM3 :
 INLINEFORM0 ,
where INLINEFORM0 is a non-linear activation function (e.g., LSTM), INLINEFORM1 is the hidden state at time INLINEFORM2 . Each time the model generates a target word INLINEFORM3 , the decoder looks at a set of positions in the source sentence where the most relevant information is located. Specifically, another non-linear activation function INLINEFORM4 is used for the decoder where the hidden state INLINEFORM5 at time INLINEFORM6 is computed by:
 INLINEFORM0 .
Here, the context vector INLINEFORM0 is computed as a weighted sum of the hidden vectors INLINEFORM1 :
 INLINEFORM0 , INLINEFORM1 ,
where INLINEFORM0 is the dot product of two vectors. Generation is conditioned on INLINEFORM1 and all the previously generated target words INLINEFORM2 :
 INLINEFORM0 ,
 INLINEFORM0 ,
where INLINEFORM0 is some non-linear function. The training objective is to minimize the cross-entropy loss of the training source-target pairs.
An RNN allows us to compute a hidden state INLINEFORM0 of each word summarizing the preceding words INLINEFORM1 , but not considering the following words INLINEFORM2 that might also be useful for simplification. An alternative approach is to use a bidirectional-RNN BIBREF20 . Here, we propose to use Neural Semantic Encoders BIBREF21 . During each encoding time step INLINEFORM3 , we compute a memory matrix INLINEFORM4 where INLINEFORM5 is the dimensionality of the word vectors. This matrix is initialized with the word vectors and is refined over time through NSE's functions to gain a better understanding of the input sequence. Concretely, NSE sequentially reads the tokens INLINEFORM6 with its read function:
 INLINEFORM0 ,
where INLINEFORM0 is an LSTM, INLINEFORM1 is the hidden state at time INLINEFORM2 . Then, a compose function is used to compose INLINEFORM3 with relevant information retrieved from the memory at the previous time step, INLINEFORM4 :
 INLINEFORM0 ,
where INLINEFORM0 is a multi-layer perceptron with one hidden layer, INLINEFORM1 is the output vector, and INLINEFORM2 is a linear combination of the memory slots of INLINEFORM3 , weighted by INLINEFORM4 :
 INLINEFORM0 , INLINEFORM1 
Here, INLINEFORM0 is the INLINEFORM1 row of the memory matrix at time INLINEFORM2 , INLINEFORM3 . Next, a write function is used to map INLINEFORM4 to the encoder output space:
 INLINEFORM0 ,
where INLINEFORM0 is an LSTM, INLINEFORM1 is the hidden state at time INLINEFORM2 . Finally, the memory is updated accordingly. The retrieved memory content pointed by INLINEFORM3 is erased and the new content is added:
 INLINEFORM0 
NSE gives us unrestricted access to the entire source sequence stored in the memory. As such, the encoder may attend to relevant words when encoding each word. The sequence INLINEFORM0 is then used as the sequence INLINEFORM1 in Section SECREF2 .
We differ from the approach of Zhang et al. Zhang:17 in the sense that we implement both a greedy strategy and a beam-search strategy to generate the target sentence. Whereas the greedy decoder always chooses the simplification candidate with the highest log-probability, the beam-search decoder keeps a fixed number (beam) of the highest scoring candidates at each time step. We report the best simplification among the outputs based on automatic evaluation measures.
Following BIBREF15 , we experiment on three simplification datasets, namely: (1) Newsela BIBREF22 , a high-quality simplification corpus of news articles composed by Newsela professional editors for children at multiple grade levels. We used the split of the data in BIBREF15 , i.e., 94,208/1,129/1,077 pairs for train/dev/test. (2) WikiSmall BIBREF10 , which contains aligned complex-simple sentence pairs from English Wikipedia (EW) and SEW. The dataset has 88,837/205/100 pairs for train/dev/test. (3) WikiLarge BIBREF15 , a larger corpus in which the training set is a mixture of three Wikipedia datasets in BIBREF10 , BIBREF11 , BIBREF23 , and the development and test sests are complex sentences taken from WikiSmall, each has 8 simplifications written by Amazon Mechanical Turk workers BIBREF12 . The dataset has 296,402/2,000/359 pairs for train/dev/test. Table TABREF7 provides statistics on the training sets.
We implemented two attention-based Seq2seq models, namely: (1) LstmLstm: the encoder is implemented by two LSTM layers; (2) NseLstm: the encoder is implemented by NSE. The decoder in both cases is implemented by two LSTM layers. The computations for a single model are run on an NVIDIA Titan-X GPU. For all experiments, our models have 300-dimensional hidden states and 300-dimensional word embeddings. Parameters were initialized from a uniform distribution [-0.1, 0.1). We used the same hyperparameters across all datasets. Word embeddings were initialized either randomly or with Glove vectors BIBREF24 pre-trained on Common Crawl data (840B tokens), and fine-tuned during training. We used a vocabulary size of 20K for Newsela, and 30K for WikiSmall and WikiLarge. Our models were trained with a maximum number of 40 epochs using Adam optimizer BIBREF25 with step size INLINEFORM0 for LstmLstm, and INLINEFORM1 for NseLstm, the exponential decay rates INLINEFORM2 . The batch size is set to 32. We used dropout BIBREF26 for regularization with a dropout rate of 0.3. For beam search, we experimented with beam sizes of 5 and 10. Following BIBREF27 , we replaced each out-of-vocabulary token INLINEFORM3 with the source word INLINEFORM4 with the highest alignment score INLINEFORM5 , i.e., INLINEFORM6 .
Our models were tuned on the development sets, either with BLEU BIBREF28 that scores the output by counting INLINEFORM0 -gram matches with the reference, or SARI BIBREF12 that compares the output against both the reference and the input sentence. Both measures are commonly used to automatically evaluate the quality of simplification output. We noticed that SARI should be used with caution when tuning neural Seq2seq simplification models. Since SARI depends on the differences between a system's output and the input sentence, large differences may yield very good SARI even though the output is ungrammatical. Thus, when tuning with SARI, we ignored epochs in which the BLEU score of the output is too low, using a threshold INLINEFORM1 . We set INLINEFORM2 to 22 on Newsela, 33 on WikiSmall, and 77 on WikiLarge.
We compared our models, either tuned with BLEU (-B) or SARI (-S), against systems reported in BIBREF15 , namely Dress, a deep reinforcement learning model, Dress-Ls, a combination of Dress and a lexical simplification model BIBREF15 , Pbmt-R, a PBMT model with dissimilarity-based re-ranking BIBREF9 , Hybrid, a hybrid semantic-based model that combines a simplification model and a monolingual MT model BIBREF29 , and Sbmt-Sari, a SBMT model with simplification-specific components. BIBREF12 .
We measured BLEU, and SARI at corpus-level following BIBREF15 . In addition, we also evaluated system output by eliciting human judgments. Specifically, we randomly selected 40 sentences from each test set, and included human reference simplifications and corresponding simplifications from the systems above. We then asked three volunteers to rate simplifications with respect to Fluency (the extent to which the output is grammatical English), Adequacy (the extent to which the output has the same meaning as the input sentence), and Simplicity (the extent to which the output is simpler than the input sentence) using a five point Likert scale.
The results of the automatic evaluation are displayed in Table TABREF15 . We first discuss the results on Newsela that contains high-quality simplifications composed by professional editors. In terms of BLEU, all neural models achieved much higher scores than Pbmt-R and Hybrid. NseLstm-B scored highest with a BLEU score of 26.31. With regard to SARI, NseLstm-S scored best among neural models (29.58) and came close to the performance of Hybrid (30.00). This indicates that NSE offers an effective means to better encode complex sentences for sentence simplification.
On WikiSmall, Hybrid – the current state-of-the-art – achieved best BLEU (53.94) and SARI (30.46) scores. Among neural models, NseLstm-B yielded the highest BLEU score (53.42), while NseLstm-S performed best on SARI (29.75). On WikiLarge, again, NseLstm-B had the highest BLEU score of 92.02. Sbmt-Sari – that was trained on a huge corpus of 106M sentence pairs and 2B words – scored highest on SARI with 39.96, followed by Dress-Ls (37.27), Dress (37.08), and NseLstm-S (36.88).
The results of human judgments are displayed in Table TABREF16 . On Newsela, NseLstm-B scored highest on Fluency. Pbmt-R was significantly better than all other systems on Adequacy while LstmLstm-S performed best on Simplicity. NseLstm-B did very well on both Adequacy and Simplicity, and was best in terms of Average. Example model outputs on Newsela are provided in Table TABREF18 .
On WikiSmall, NseLstm-B performed best on both Fluency and Adequacy. On WikiLarge, LstmLstm-B achieved the highest Fluency score while NseLstm-B received the highest Adequacy score. In terms of Simplicity and Average, NseLstm-S outperformed all other systems on both WikiSmall and WikiLarge.
As shown in Table TABREF16 , neural models often outperformed traditional systems (Pbmt-R, Hybrid, Sbmt-Sari) on Fluency. This is not surprising given the recent success of neural Seq2seq models in language modeling and neural machine translation BIBREF30 , BIBREF27 . On the downside, our manual inspection reveals that neural models learn to perform copying very well in terms of rewrite operations (e.g., copying, deletion, reordering, substitution), often outputting the same or parts of the input sentence.
Finally, as can be seen in Table TABREF16 , Reference scored lower on Adequacy compared to Fluency and Simplicity on Newsela. On Wikipedia-based datasets, Reference obtained high Adequacy scores but much lower Simplicity scores compared to Newsela. This supports the assertion by previous work BIBREF22 that SEW has a large proportion of inadequate simplifications.
Table TABREF20 shows the correlations between the scores assigned by humans and the automatic evaluation measures. There is a positive significant correlation between Fluency and Adequacy (0.69), but a negative significant correlation between Adequacy and Simplicity (-0.64). BLEU correlates well with Fluency (0.63) and Adequacy (0.90) while SARI correlates well with Simplicity (0.73). BLEU and SARI show a negative significant correlation (-0.54). The results reflect the challenge of managing the trade-off between Fluency, Adequacy and Simplicity in sentence simplification.
In this paper, we explore neural Seq2seq models for sentence simplification. We propose to use an architecture with augmented memory capacities which we believe is suitable for the task, where one is confronted with long and complex sentences. Results of both automatic and human evaluation on different datasets show that our model is capable of significantly reducing the reading difficulty of the input, while performing well in terms of grammaticality and meaning preservation.
We would like to thank Emily Druhl, Jesse Lingeman, and the UMass BioNLP team for their help with this work. We also thank Xingxing Zhang, Sergiu Nisioi for valuable discussions. The authors would like to acknowledge the reviewers for their thoughtful comments and suggestions. 

Figure and table captions:
Figure 1: Attention-based encoder-decoder model. The model may attend to relevant positions in the source sentence while decoding the simplification, e.g., to generate the target word won the model may attend to the words received, nominated and Prize in the source sentence.
Table 1: Statistics for the training sets: the vocabulary size (vocab size), and the average number of tokens per sentence (#tokens/sent) of the source (src) and target (tgt) language.
Table 2: Model performance using automatic evaluation measures (BLEU and SARI).
Table 3: Average human ratings (Fluency (F), Adequacy (A), Simplicity (S), and Average (Avg.)).
Table 4: Example model outputs on Newsela. Substitutions are shown in bold.
Table 5: Pearson correlation between the scores assigned by humans and the automatic evaluation measures. Scores marked ∗∗ are significant at p < 0.01.

Questions about this paper:
how do humans judge the simplified sentences?

Answers:
By fluency, adequacy, and simplicity using a five point Likert scale.