Question: What models' effectiveness do they investigate when different neural architectures for the encoder are considered?
Evidence: The goal of sentence simplification is to compose complex sentences into simpler ones so that they are more comprehensible and accessible, while still retaining the original information content and meaning. Sentence simplification has a number of practical applications. On one hand, it provides reading aids for people with limited language proficiency BIBREF1 , BIBREF2 , or for patients with linguistic and cognitive disabilities BIBREF3 . On the other hand, it can improve the performance of other NLP tasks BIBREF4 , BIBREF5 , BIBREF6 . Prior work has explored monolingual machine translation (MT) approaches, utilizing corpora of simplified texts, e.g., Simple English Wikipedia (SEW), and making use of statistical MT models, such as phrase-based MT (PBMT) BIBREF7 , BIBREF8 , BIBREF9 , tree-based MT (TBMT) BIBREF10 , BIBREF11 , or syntax-based MT (SBMT) BIBREF12 .
Inspired by the success of neural MT BIBREF13 , BIBREF14 , recent work has started exploring neural simplification with sequence to sequence (Seq2seq) models, also referred to as encoder-decoder models. Nisioi et al. Nisioi:17 implemented a standard LSTM-based Seq2seq model and found that they outperform PBMT, SBMT, and unsupervised lexical simplification approaches. Zhang and Lapata BIBREF15 viewed the encoder-decoder model as an agent and employed a deep reinforcement learning framework in which the reward has three components capturing key aspects of the target output: simplicity, relevance, and fluency. The common practice for Seq2seq models is to use recurrent neural networks (RNNs) with Long Short-Term Memory BIBREF16 or Gated Recurrent Unit BIBREF17 for the encoder and decoder BIBREF18 , BIBREF15 . These architectures were designed to be capable of memorizing long-term dependencies across sequences. Nevertheless, their memory is typically small and might not be enough for the simplification task, where one is confronted with long and complicated sentences. In this study, we go beyond the conventional LSTM/GRU-based Seq2seq models and propose to use a memory-augmented RNN architecture called Neural Semantic Encoders (NSE). This architecture has been shown to be effective in a wide range of NLP tasks BIBREF0 . The contribution of this paper is twofold:
(1) First, we present a novel simplification model which is, to the best of our knowledge, the first model that use memory-augmented RNN for the task. We investigate the effectiveness of neural Seq2seq models when different neural architectures for the encoder are considered. Our experiments reveal that the NseLstm model that uses an NSE as the encoder and an LSTM as the decoder performed the best among these models, improving over strong simplification systems. (2) Second, we perform an extensive evaluation of various approaches proposed in the literature on different datasets. Results of both automatic and human evaluation show that our approach is remarkably effective for the task, significantly reducing the reading difficulty of the input, while preserving grammaticality and the original meaning. We further discuss some advantages and disadvantages of these approaches.
We investigate the effectiveness of neural Seq2seq models when different neural architectures for the encoder are considered.
Answer: Neural Seq2seq models.