prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How are models evaluated in this human-machine communication game?
Context 1: Introduction
Suppose a user wants to write a sentence “I will be 10 minutes late ” Ideally, she would type just a few keywords such as “10 minutes late” and an autocomplete system would be able to infer the intended sentence (Figure FIGREF1)  Existing left-to-right autocomplete systems BIBREF0, BIBREF1 can often be inefficient, as the prefix of a sentence (e g  “I will be”) fails to capture the core meaning of the sentence  Besides the practical goal of building a better autocomplete system, we are interested in exploring the tradeoffs inherent to such communication schemes between the efficiency of typing keywords, accuracy of reconstruction, and interpretability of keywords 
One approach to learn such schemes is to collect a supervised dataset of keywords-sentence pairs as a training set, but (i) it would be expensive to collect such data from users, and (ii) a static dataset would not capture a real user's natural predilection to adapt to the system BIBREF2  Another approach is to avoid supervision and jointly learn a user-system communication scheme to directly optimize the combination of efficiency and accuracy  However, learning in this way can lead to communication schemes that are uninterpretable to humans BIBREF3, BIBREF4 (see Appendix for additional related work) 
In this work, we propose a simple, unsupervised approach to an autocomplete system that is efficient, accurate, and interpretable  For interpretability, we restrict keywords to be subsequences of their source sentences based on the intuition that humans can infer most of the original meaning from a few keywords  We then apply multi-objective optimization approaches to directly control and achieve desirable tradeoffs between efficiency and accuracy 
We observe that naively optimizing a linear combination of efficiency and accuracy terms is unstable and leads to suboptimal schemes  Thus, we propose a new objective which optimizes for communication efficiency under an accuracy constraint  We show this new objective is more stable and efficient than the linear objective at all accuracy levels 
As a proof-of-concept, we build an autocomplete system within this framework which allows a user to write sentences by specifying keywords  We empirically show that our framework produces communication schemes that are 52 16% more accurate than rule-based baselines when specifying 77 37% of sentences, and 11 73% more accurate than a naive, weighted optimization approach when specifying 53 38% of sentences  Finally, we demonstrate that humans can easily adapt to the keyword-based autocomplete system and save nearly 50% of time compared to typing a full sentence in our user study 
Approach
Consider a communication game in which the goal is for a user to communicate a target sequence $x= (x_1,    , x_m)$ to a system by passing a sequence of keywords $z= (z_1,    , z_n)$  The user generates keywords $z$ using an encoding strategy $q_{\alpha }(z\mid x)$, and the system attempts to guess the target sequence $x$ via a decoding strategy $p_{\beta }(x\mid z)$ 
A good communication scheme $(q_{\alpha }, p_{\beta })$ should be both efficient and accurate  Specifically, we prefer schemes that use fewer keywords (cost), and the target sentence $x$ to be reconstructed with high probability (loss) where
Based on our assumption that humans have an intuitive sense of retaining important keywords, we restrict the set of schemes to be a (potentially noncontiguous) subsequence of the target sentence  Our hypothesis is that such subsequence schemes naturally ensure interpretability, as efficient human and machine communication schemes are both likely to involve keeping important content words 
Approach ::: Modeling with autoencoders 
To learn communication schemes without supervision, we model the cooperative communication between a user and system through an encoder-decoder framework  Concretely, we model the user's encoding strategy $q_{\alpha }(z\mid x)$ with an encoder which encodes the target sentence $x$ into the keywords $z$ by keeping a subset of the tokens  This stochastic encoder $q_{\alpha }(z\mid x)$ is defined by a model which returns the probability of each token retained in the final subsequence $z$  Then, we sample from Bernoulli distributions according to these probabilities to either keep or drop the tokens independently (see Appendix for an example) 
We model the autocomplete system's decoding strategy $p_{\beta }(x\mid z)$ as a probabilistic model which conditions on the keywords $z$ and returns a distribution over predictions $x$  We use a standard sequence-to-sequence model with attention and copying for the decoder, but any model architecture can be used (see Appendix for details) 
Approach ::: Multi-objective optimization 
Our goal now is to learn encoder-decoder pairs which optimally balance the communication cost and reconstruction loss  The simplest approach to balancing efficiency and accuracy is to weight $\mathrm {cost}(x, \alpha )$ and $\mathrm {loss}(x, \alpha , \beta )$ linearly using a weight $\lambda $ as follows,
where the expectation is taken over the population distribution of source sentences $x$, which is omitted to simplify notation  However, we observe that naively weighting and searching over $\lambda $ is suboptimal and highly unstable—even slight changes to the weighting results in degenerate schemes which keep all or none of its tokens  This instability motivates us to develop a new stable objective 
Our main technical contribution is to draw inspiration from the multi-objective optimization literature and view the tradeoff as a sequence of constrained optimization problems, where we minimize the expected cost subject to varying expected reconstruction error constraints $\epsilon $,
This greatly improves the stability of the training procedure  We empirically observe that the model initially keeps most of the tokens to meet the constraints, and slowly learns to drop uninformative words from the keywords to minimize the cost  Furthermore, $\epsilon $ in Eq (DISPLAY_FORM6) allows us to directly control the maximum reconstruction error of resulting schemes, whereas $\lambda $ in Eq (DISPLAY_FORM5) is not directly related to any of our desiderata 
To optimize the constrained objective, we consider the Lagrangian of Eq (DISPLAY_FORM6),
Much like the objective in Eq (DISPLAY_FORM5) we can compute unbiased gradients by replacing the expectations with their averages over random minibatches  Although gradient descent guarantees convergence on Eq (DISPLAY_FORM7) only when the objective is convex, we find that not only is the optimization stable, the resulting solution achieves better performance than the weighting approach in Eq (DISPLAY_FORM5) 
Approach ::: Optimization 
Optimization with respect to $q_{\alpha }(z\mid x)$ is challenging as $z$ is discrete, and thus, we cannot differentiate $\alpha $ through $z$ via the chain rule  Because of this, we use the stochastic REINFORCE estimate BIBREF5 as follows:
We perform joint updates on $(\alpha , \beta , \lambda )$, where $\beta $ and $\lambda $ are updated via standard gradient computations, while $\alpha $ uses an unbiased, stochastic gradient estimate where we approximate the expectation in Eq (DISPLAY_FORM9)  We use a single sample from $q_{\alpha }(z\mid x)$ and moving-average of rewards as a baseline to reduce variance 
Experiments
We evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details)  We quantify the efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords  The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence 
Experiments ::: Effectiveness of constrained objective 
We first show that the linear objective in Eq (DISPLAY_FORM5) is suboptimal compared to the constrained objective in Eq (DISPLAY_FORM6)  Figure FIGREF10 compares the achievable accuracy and efficiency tradeoffs for the two objectives, which shows that the constrained objective results in more efficient schemes than the linear objective at every accuracy level (e g  11 73% more accurate at a 53 38% retention rate) 
We also observe that the linear objective is highly unstable as a function of the tradeoff parameter $\lambda $ and requires careful tuning  Even slight changes to $\lambda $ results in degenerate schemes that keep all or none of the tokens (e g  $\lambda \le 4 2$ and $\lambda \ge 4 4$)  On the other hand, the constrained objective is substantially more stable as a function of $\epsilon $ (e g  points for $\epsilon $ are more evenly spaced than $\lambda $) 
Experiments ::: Efficiency-accuracy tradeoff 
We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword  The Unif encoder randomly keeps tokens to generate keywords with the probability $\delta $  The Stopword encoder keeps all tokens but drops stop words (e g  `the', `a', `or') all the time ($\delta =0$) or half of the time ($\delta =0 5$)  The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i e  $\mathrm {loss}(x, \alpha , \beta )$) 
Figure FIGREF10 shows that two baselines achieve similar tradeoff curves, while the constrained model achieves a substantial 52 16% improvement in accuracy at a 77 37% retention rate compared to Unif, thereby showing the benefits of jointly training the encoder and decoder 
Experiments ::: Robustness and analysis 
We provide additional experimental results on the robustness of learned communication schemes as well as in-depth analysis on the correlation between the retention rates of tokens and their properties, which we defer to Appendix and for space 
Experiments ::: User study 
We recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus  Each user was shown alternating autocomplete and writing tasks across 50 sentences (see Appendix for user interface)  For the autocomplete task, we gave users a target sentence and asked them to type a set of keywords into the system  The users were shown the top three suggestions from the autocomplete system, and were asked to mark whether each of these three suggestions was semantically equivalent to the target sentence  For the writing task, we gave users a target sentence and asked them to either type the sentence verbatim or a sentence that preserves the meaning of the target sentence 
Table TABREF13 shows two examples of the autocomplete task and actual user-provided keywords  Each column contains a set of keywords and its corresponding top three suggestions generated by the autocomplete system with beam search  We observe that the system is likely to propose generic sentences for under-specified keywords (left column) and almost the same sentences for over-specified keywords (right column)  For properly specified keywords (middle column), the system completes sentences accordingly by adding a verb, adverb, adjective, preposition, capitalization, and punctuation 
Overall, the autocomplete system achieved high accuracy in reconstructing the keywords  Users marked the top suggestion from the autocomplete system to be semantically equivalent to the target $80 6$% of the time, and one of the top 3 was semantically equivalent $90 11$% of the time  The model also achieved a high exact match accuracy of 18 39%  Furthermore, the system was efficient, as users spent $3 86$ seconds typing keywords compared to $5 76$ seconds for full sentences on average  The variance of the typing time was $0 08$ second for keywords and $0 12$ second for full sentences, indicating that choosing and typing keywords for the system did not incur much overhead 
Experiments ::: Acknowledgments
We thank the reviewers and Yunseok Jang for their insightful comments  This work was supported by NSF CAREER Award IIS-1552635 and an Intuit Research Award 
Experiments ::: Reproducibility
All code, data and experiments are available on CodaLab at https://bit ly/353fbyn 
Context 2: Introduction
Text classification has become an indispensable task due to the rapid growth in the number of texts in digital form available online  It aims to classify different texts, also called documents, into a fixed number of predefined categories, helping to organize data, and making easier for users to find the desired information  Over the past three decades, many methods based on machine learning and statistical models have been applied to perform this task, such as latent semantic analysis (LSA), support vector machines (SVM), and multinomial naive Bayes (MNB) 
The first step in utilizing such methods to categorize textual data is to convert the texts into a vector representation  One of the most popular text representation models is the bag-of-words model BIBREF0 , which represents each document in a collection as a vector in a vector space  Each dimension of the vectors represents a term (e g , a word, a sequence of words), and its value encodes a weight, which can be how many times the term occurs in the document 
Despite showing positive results in tasks such as language modeling and classification BIBREF1 , BIBREF2 , BIBREF3 , the BOW representation has limitations: first, feature vectors are commonly very high-dimensional, resulting in sparse document representations, which are hard to model due to space and time complexity  Second, BOW does not consider the proximity of words and their position in the text and consequently cannot encode the words semantic meanings 
To solve these problems, neural networks have been employed to learn vector representations of words BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7   In particular, the word2vec representation BIBREF8 has gained attention  Given a training corpus, word2vec can generate a vector for each word in the corpus that encodes its semantic information  These word vectors are distributed in such a way that words from similar contexts are represented by word vectors with high correlation, while words from different contexts are represented by word vectors with low correlation 
One crucial aspect of the word2vec representation is that arithmetic and distance calculation between two word vectors can be performed, giving information about their semantic relationship  However, rather than looking at pairs of word vectors, we are interested in studying the relationship between sets of vectors as a whole and, therefore, it is desirable to have a text representation based on a set of these word vectors 
To tackle this problem, we introduce the novel concept of word subspace  It is mathematically defined as a low dimensional linear subspace in a word vector space with high dimensionality  Given that words from texts of the same class belong to the same context, it is possible to model word vectors of each class as word subspaces and efficiently compare them in terms of similarity by using canonical angles between the word subspaces  Through this representation, most of the variability of the class is retained  Consequently, a word subspace can effectively and compactly represent the context of the corresponding text  We achieve this framework through the mutual subspace method (MSM) BIBREF9  
The word subspace of each text class is modeled by applying PCA without data centering to the set of word vectors of the class  When modeling the word subspaces, we assume only one occurrence of each word inside the class 
However, as seen in the BOW approach, the frequency of words inside a text is an informative feature that should be considered  In order to introduce this feature in the word subspace modeling and enhance its performance, we further extend the concept of word subspace to the term-frequency (TF) weighted word subspace 
In this extension, we consider a set of weights, which encodes the words frequencies, when performing the PCA  Text classification with TF weighted word subspace can also be performed under the framework of MSM  We show the validity of our modeling through experiments on the Reuters database, an established database for natural language processing tasks  We demonstrate the effectiveness of the word subspace formulation and its extension, comparing our methods' performance to various state-of-art methods 
The main contributions of our work are:
The remainder of this paper is organized as follows  In Section "Related Work" , we describe the main works related to text classification  In Section "Word subspace" , we present the formulation of our proposed word subspace  In Section "Conventional text classification methods" , we explain how text classification with word subspaces is performed under the MSM framework  Then, we present the TF weighted word subspace extension in Section "TF weighted word subspace"   Evaluation experiments and their results are described in Section "Experimental Evaluation"   Further discussion is then presented in Section "Discussion" , and our conclusions are described in Section "Conclusions and Future Work"  
Related Work
In this section, we outline relevant work towards text classification  We start by describing how text data is conventionally represented using the bag-of-words model and then follow to describe the conventional methods utilized in text classification 
Text Representation with bag-of-words
The bag-of-words representation comes from the hypothesis that frequencies of words in a document can indicate the relevance of the document to a query BIBREF0 , that is, if documents and a query have similar frequencies for the same words, they might have a similar meaning  This representation is based on the vector space model (VSM), that was developed for the SMART information retrieval system BIBREF10   In the VSM, the main idea is that documents in a collection can be represented as a vector in a vector space, where vectors close to each other represent semantically similar documents 
More formally, a document $d$ can be represented by a vector in $\mathbb {R}^{n}$ , where each dimension represents a different term  A term can be a single word, constituting the conventional bag-of-words, or combinations of $N$ words, constituting the bag-of-N-grams  If a term occurs in the document, its position in the vector will have a non-zero value, also known as term weight  Two documents in the VSM can be compared to each other by taking the cosine distance between them BIBREF1  
There are several ways to compute the term weights  Among them, we can highlight some: Binary weights, term-frequency (TF) weights, and term-frequency inverse document-frequency (TF-IDF) weights 
Consider a corpus with documents $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ and a vocabulary with all terms in the corpus $V = \lbrace w_i\rbrace _{i=1}^{|V|}$   The term weights can be defined as:
Binary weight: If a term occurs in the document, its weight is 1  Otherwise, it is zero 
Term-frequency weight (TF): The weight of a term $w$ is defined by the number of times it occurs in the document $d$  
$$TF(w,d) = n_d^w$$   (Eq  8)
Inverse document-frequency: The weight of a term $w$ , given the corpus $D$ , is defined as the total number of documents $|D|$ divided by the number of documents that have the term $w$ , $|D^w|$  
$$IDF(w | D) = \frac{|D|}{|D^w|}$$   (Eq  10)
Term-frequency inverse document-frequency (TF-IDF): The weight of a term $w$ is defined by the multiplication of its term-frequency and its inverse document-frequency  When considering only the TF weights, all terms have the same importance among the corpus  By using the IDF weight, words that are more common across all documents in $D$ receive a smaller weight, giving more importance to rare terms in the corpus 
$$TFIDF(w,d | D)=TF \times IDF$$   (Eq  12)
In very large corpus, it is common to consider the logarithm of the IDF in order to dampen its effect 
$$TFIDF(w,d | D)=TF \times log_{10}(IDF)$$   (Eq  13)
Conventional text classification methods
Multi-variate Bernoulli (MVB) and multinomial naive Bayes (MNB) are two generative models based on the naive Bayes assumption  In other words, they assume that all attributes (e g , the frequency of each word, the presence or absence of a word) of each text are independent of each other given the context of the class BIBREF11  
In the MVB model, a document is represented by a vector generated by a bag-of-words with binary weights  In this case, a document can be considered an event, and the presence or the absence of the words to be the attributes of the event  On the other hand, the MNB model represents each document as a vector generated by a bag-of-words with TF weights  Here, the individual word occurrences are considered as events and the document is a collection of word events 
Both these models use the Bayes rule to classify a document  Consider that each document should be classified into one of the classes in $C=\lbrace c_j\rbrace _{j=1}^{|C|}$   The probability of each class given the document is defined as:
$$P(c_j|d_i) = \frac{P(d_i|c_j)P(c_j)}{P(d_i)} $$   (Eq  16)
The prior $P(d_i)$ is the same for all classes, so to determine the class to which $d_i$ belongs to, the following equation can be used:
$$prediction(d_i) = argmax_{c_j}P(d_i|c_j)P(c_j)$$   (Eq  17)
The prior $P(c_j)$ can be obtained by the following equation:
$$P(c_j) = \frac{1+|D_j|}{|C|+|D|},$$   (Eq  18)
where $|D_j|$ is the number of documents in class $c_j$  
As for the posterior $P(d_i|c_j)$ , different calculations are performed for each model  For MVB, it is defined as:
$$P(d_i|c_j) = \prod _{k=1}^{|V|}P(w_k|c_j)^{t_i^k}(1-P(w_k|c_j))^{1-t_i^k},$$   (Eq  19)
where $w_k$ is the k-th word in the vocabulary $V$ , and $t_i^k$ is the value (0 or 1) of the k-th element of the vector of document $d_i$  
For the MNB, it is defined as:
$$P(d_i|c_j) = P(|d_i|)|d_i|!\prod _{k=1}^{|V|}\frac{P(w_k|c_j)^{n_i^k}}{n_i^k!},$$   (Eq  20)
where $|d_i|$ is the number of words in document $d_i$ and $n_i^k$ is the k-th element of the vector of document $d_i$ and it represents how many times word $w_k$ occurs in $d_i$  
Finally, the posterior $P(w_k|c_j)$ can be obtained by the following equation:
$$P(w_k|c_j) = \frac{1+|D_j^k|}{|C|+|D|},$$   (Eq  21)
where $|D_j^k|$ is the number of documents in class $c_j$ that contain the word $w_k$  
In general, MVB tends to perform better than MNB at small vocabulary sizes whereas MNB is more efficient on large vocabularies 
Despite being robust tools for text classification, both these models depend directly on the bag-of-words features and do not naturally work with representations such as word2vec 
Latent semantic analysis (LSA), or latent semantic indexing (LSI), was proposed in BIBREF12 , and it extends the vector space model by using singular value decomposition (SVD) to find a set of underlying latent variables which spans the meaning of texts 
It is built from a term-document matrix, in which each row represents a term, and each column represents a document  This matrix can be built by concatenating the vectors of all documents in a corpus, obtained using the bag-of-words model, that is, $ {X} = [ {v}_1, {v}_2,    , {v}_{|D|}]$ , where ${v}_i$ is the vector representation obtained using the bag-of-words model 
In this method, the term-document matrix is decomposed using the singular value decomposition,
$${X} = {U\Sigma V}^\top ,$$   (Eq  23)
where $U$ and $V$ are orthogonal matrices and correspond to the left singular vectors and right singular vectors of $X$ , respectively  $\Sigma $ is a diagonal matrix, and it contains the square roots of the eigenvalues of $X^TX$ and $XX^T$   LSA finds a low-rank approximation of $X$ by selecting only the $k$ largest singular values and its respective singular vectors,
$${X}_k = {U}_k{\Sigma }_k {V}_k^{\top } $$   (Eq  24)
To compare two documents, we project both of them into this lower dimension space and calculate the cosine distance between them  The projection ${\hat{d}}$ of document ${d}$ is obtained by the following equation:
$${\hat{d}} = {\Sigma }_k^{-1} {U}_k^\top {d} $$   (Eq  25)
Despite its extensive application on text classification BIBREF13 , BIBREF14 , BIBREF15 , this method was initially proposed for document indexing and, therefore, does not encode any class information when modeling the low-rank approximation  To perform classification, 1-nearest neighbor is usually performed, placing a query document into the class of the nearest training document 
The support vector machine (SVM) was first presented in BIBREF16 and performs the separation between samples of two different classes by projecting them onto a higher dimensionality space  It was first applied in text classification by BIBREF17 and have since been successfully applied in many tasks related to natural language processing BIBREF18 , BIBREF19  
Consider a training data set $D$ , with $n$ samples
$$D = \lbrace ({x}_i,c_i)|{x}_i\in \mathbb {R}^p, c_i \in \lbrace -1,1\rbrace  \rbrace _{i=1}^{n},$$   (Eq  27)
where $c_i$ represents the class to which ${x}_i$ belongs to  Each ${x}_i$ is a $p$ -dimensional vector  The goal is to find the hyperplane that divides the points from $c_i = 1$ from the points from $c_i = -1$   This hyperplane can be written as a set of points $x$ satisfying:
$${w} \cdot {x} - b = 0,$$   (Eq  28)
where $\cdot $ denotes the dot product  The vector ${w}$ is perpendicular to the hyperplane  The parameter $\frac{b}{\Vert {w}\Vert }$ determines the offset of the hyperplane from the origin along the normal vector ${w}$  
We wish to choose ${w}$ and $b$ , so they maximize the distance between the parallel hyperplanes that are as far apart as possible, while still separating the data 
If the training data is linearly separable, we can select two hyperplanes in a way that there are no points between them and then try to maximize the distance  In other words, minimize $\Vert {w}\Vert $ subject to $c_i({w}\cdot {x}_u-b) \ge 1, i=\lbrace 1,2,   ,n\rbrace $   If the training data is not linearly separable, the kernel trick can be applied, where every dot product is replaced by a non-linear kernel function 
Word subspace
All methods mentioned above utilize the BOW features to represent a document  Although this representation is simple and powerful, its main problem lies on disregarding the word semantics within a document, where the context and meaning could offer many benefits to the model such as identification of synonyms 
In our formulation, words are represented as vectors in a real-valued feature vector space $\mathbb {R}^{p}$ , by using word2vec BIBREF8   Through this representation, it is possible to calculate the distance between two words, where words from similar contexts are represented by vectors close to each other, while words from different contexts are represented as far apart vectors  Also, this representation brings the new concept of arithmetic operations between words, where operations such as addition and subtraction carry meaning (eg , “king”-“man”+“woman”=“queen”) BIBREF20  
Consider a set of documents which belong to the same context $D_c = \lbrace d_i\rbrace _{i=1}^{|D_c|}$   Each document $d_i$ is represented by a set of $N_i$ words, $d_i = \lbrace w_k\rbrace _{k=1}^{N_i}$   By considering that all words from documents of the same context belong to the same distribution, a set of words $W_c = \lbrace w_k\rbrace _{k=1}^{N_c}$ with the words in the context $c$ is obtained
Context 3: Introduction
Text simplification aims to reduce the lexical and structural complexity of a text, while still retaining the semantic meaning, which can help children, non-native speakers, and people with cognitive disabilities, to understand text better  One of the methods of automatic text simplification can be generally divided into three categories: lexical simplification (LS) BIBREF0 , BIBREF1 , rule-based BIBREF2 , and machine translation (MT) BIBREF3 , BIBREF4   LS is mainly used to simplify text by substituting infrequent and difficult words with frequent and easier words  However, there are several challenges for the LS approach: a great number of transformation rules are required for reasonable coverage and should be applied based on the specific context; third, the syntax and semantic meaning of the sentence is hard to retain  Rule-based approaches use hand-crafted rules for lexical and syntactic simplification, for example, substituting difficult words in a predefined vocabulary  However, such approaches need a lot of human-involvement to manually define these rules, and it is impossible to give all possible simplification rules  MT-based approach has attracted great attention in the last several years, which addresses text simplification as a monolingual machine translation problem translating from 'ordinary' and 'simplified' sentences 
In recent years, neural Machine Translation (NMT) is a newly-proposed deep learning approach and achieves very impressive results BIBREF5 , BIBREF6 , BIBREF7   Unlike the traditional phrased-based machine translation system which operates on small components separately, NMT system is being trained end-to-end, without the need to have external decoders, language models or phrase tables  Therefore, the existing architectures in NMT are used for text simplification BIBREF8 , BIBREF4   However, most recent work using NMT is limited to the training data that are scarce and expensive to build  Language models trained on simplified corpora have played a central role in statistical text simplification BIBREF9 , BIBREF10   One main reason is the amount of available simplified corpora typically far exceeds the amount of parallel data  The performance of models can be typically improved when trained on more data  Therefore, we expect simplified corpora to be especially helpful for NMT models 
In contrast to previous work, which uses the existing NMT models, we explore strategy to include simplified training corpora in the training process without changing the neural network architecture  We first propose to pair simplified training sentences with synthetic ordinary sentences during training, and treat this synthetic data as additional training data  We obtain synthetic ordinary sentences through back-translation, i e  an automatic translation of the simplified sentence into the ordinary sentence BIBREF11   Then, we mix the synthetic data into the original (simplified-ordinary) data to train NMT model  Experimental results on two publicly available datasets show that we can improve the text simplification quality of NMT models by mixing simplified sentences into the training set over NMT model only using the original training data 
Related Work
Automatic TS is a complicated natural language processing (NLP) task, which consists of lexical and syntactic simplification levels BIBREF12   It has attracted much attention recently as it could make texts more accessible to wider audiences, and used as a pre-processing step, improve performances of various NLP tasks and systems BIBREF13 , BIBREF14 , BIBREF15   Usually, hand-crafted, supervised, and unsupervised methods based on resources like English Wikipedia and Simple English Wikipedia (EW-SEW) BIBREF10 are utilized for extracting simplification rules  It is very easy to mix up the automatic TS task and the automatic summarization task BIBREF3 , BIBREF16 , BIBREF6   TS is different from text summarization as the focus of text summarization is to reduce the length and redundant content 
At the lexical level, lexical simplification systems often substitute difficult words using more common words, which only require a large corpus of regular text to obtain word embeddings to get words similar to the complex word BIBREF1 , BIBREF9   Biran et al  BIBREF0 adopted an unsupervised method for learning pairs of complex and simpler synonyms from a corpus consisting of Wikipedia and Simple Wikipedia  At the sentence level, a sentence simplification model was proposed by tree transformation based on statistical machine translation (SMT) BIBREF3   Woodsend and Lapata BIBREF17 presented a data-driven model based on a quasi-synchronous grammar, a formalism that can naturally capture structural mismatches and complex rewrite operations  Wubben et al  BIBREF18 proposed a phrase-based machine translation (PBMT) model that is trained on ordinary-simplified sentence pairs  Xu et al  BIBREF19 proposed a syntax-based machine translation model using simplification-specific objective functions and features to encourage simpler output 
Compared with SMT, neural machine translation (NMT) has shown to produce state-of-the-art results BIBREF5 , BIBREF7   The central approach of NMT is an encoder-decoder architecture implemented by recurrent neural networks, which can represent the input sequence as a vector, and then decode that vector into an output sequence  Therefore, NMT models were used for text simplification task, and achieved good results BIBREF8 , BIBREF4 , BIBREF20   The main limitation of the aforementioned NMT models for text simplification depended on the parallel ordinary-simplified sentence pairs  Because ordinary-simplified sentence pairs are expensive and time-consuming to build, the available largest data is EW-SEW that only have 296,402 sentence pairs  The dataset is insufficiency for NMT model if we want to NMT model can obtain the best parameters  Considering simplified data plays an important role in boosting fluency for phrase-based text simplification, and we investigate the use of simplified data for text simplification  We are the first to show that we can effectively adapt neural translation models for text simplifiation with simplified corpora 
Simplified Corpora
We collected a simplified dataset from Simple English Wikipedia that are freely available, which has been previously used for many text simplification methods BIBREF0 , BIBREF10 , BIBREF3   The simple English Wikipedia is pretty easy to understand than normal English Wikipedia  We downloaded all articles from Simple English Wikipedia  For these articles, we removed stubs, navigation pages and any article that consisted of a single sentence  We then split them into sentences with the Stanford CorNLP BIBREF21 , and deleted these sentences whose number of words are smaller than 10 or large than 40  After removing repeated sentences, we chose 600K sentences as the simplified data with 11 6M words, and the size of vocabulary is 82K 
Text Simplification using Neural Machine Translation
Our work is built on attention-based NMT BIBREF5 as an encoder-decoder network with recurrent neural networks (RNN), which simultaneously conducts dynamic alignment and generation of the target simplified sentence 
The encoder uses a bidirectional RNN that consists of forward and backward RNN  Given a source sentence INLINEFORM0 , the forward RNN and backward RNN calculate forward hidden states INLINEFORM1 and backward hidden states INLINEFORM2 , respectively  The annotation vector INLINEFORM3 is obtained by concatenating INLINEFORM4 and INLINEFORM5  
The decoder is a RNN that predicts a target simplificated sentence with Gated Recurrent Unit (GRU) BIBREF22   Given the previously generated target (simplified) sentence INLINEFORM0 , the probability of next target word INLINEFORM1 is DISPLAYFORM0
where INLINEFORM0 is a non-linear function, INLINEFORM1 is the embedding of INLINEFORM2 , and INLINEFORM3 is a decoding state for time step INLINEFORM4  
State INLINEFORM0 is calculated by DISPLAYFORM0
where INLINEFORM0 is the activation function GRU 
The INLINEFORM0 is the context vector computed as a weighted annotation INLINEFORM1 , computed by DISPLAYFORM0
where the weight INLINEFORM0 is computed by DISPLAYFORM0 DISPLAYFORM1
where INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are weight matrices  The training objective is to maximize the likelihood of the training data  Beam search is employed for decoding 
Synthetic Simplified Sentences
We train an auxiliary system using NMT model from the simplified sentence to the ordinary sentence, which is first trained on the available parallel data  For leveraging simplified sentences to improve the quality of NMT model for text simplification, we propose to adapt the back-translation approach proposed by Sennrich et al  BIBREF11 to our scenario  More concretely, Given one sentence in simplified sentences, we use the simplified-ordinary system in translate mode with greedy decoding to translate it to the ordinary sentences, which is denoted as back-translation

ground_truth: by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews

efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords, accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What can word subspace represent?
Context 2: 
We then translate these words into word vectors using word2vec, resulting in a set of word vectors $X_c = \lbrace {x}^k_c\rbrace _{k=1}^{N_c} \in \mathbb {R}^p$   This set of word vectors is modeled into a word subspace, which is a compact, scalable and meaningful representation of the whole set  Such a word subspace is generated by applying PCA to the set of word vectors 
First, we compute an autocorrelation matrix, ${R}_c$ :
$${R}_c = \frac{1}{N_c}\sum _{i=1}^{N_c}{x}^{i}_c{x}_c^{i^{\top }} $$   (Eq  29)
The orthonormal basis vectors of $m_c$ -dimensional subspace ${Y}_c$ are obtained as the eigenvectors with the $m_c$ largest eigenvalues of the matrix ${R}_c$   We represent a subspace ${Y}_c$ by the matrix ${Y}_c \in \mathbb {R}^{p \times m_c}$ , which has the corresponding orthonormal basis vectors as its column vectors 
Text classification based on word subspace
We formulate our problem as a single label classification problem  Given a set of training documents, which we will refer as corpus, $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ , with known classes $C = \lbrace c_j\rbrace _{j=1}^{|C|}$ , we wish to classify a query document $d_q$ into one of the classes in $C$  
Text classification based on word subspace can be performed under the framework of mutual subspace method (MSM)  This task involves two different stages: A learning stage, where the word subspace for each class is modeled, and a classification stage, where the word subspace for a query is modeled and compared to the word subspaces of the classes 
In the learning stage, it is assumed that all documents of the same class belong to the same context, resulting in a set of words $W_c = \lbrace w_c^k\rbrace _{k=1}^{N_c}$   This set assumes that each word appears only once in each class  Each set $\lbrace W_c\rbrace _{c=1}^{|C|}$ is then modeled into a word subspace ${Y}_c$ , as explained in Section "Word subspace"   As the number of words in each class may vary largely, the dimension $m_c$ of each class word subspace is not set to the same value 
In the classification stage, for a query document $d_q$ , it is also assumed that each word occurs only once, generating a subspace ${Y}_q$  
To measure the similarity between a class word subspace ${Y}_c$ and a query word subspace ${Y}_q$ , the canonical angles between the two word subspaces are used BIBREF21   There are several methods for calculating canonical angles BIBREF22 , BIBREF23 , and BIBREF24 , among which the simplest and most practical is the singular value decomposition (SVD)  Consider, for example, two subspaces, one from the training data and another from the query, represented as matrices of bases, ${Y}_{c} = [{\Phi }_{1} \ldots {\Phi }_{m_c}] \in \mathbb {R}^{p \times m_c}$ and ${Y}_{q} = [{\Psi }_{1} \ldots {\Psi }_{m_q}] \in \mathbb {R}^{p \times m_q}$ , where ${\Phi }_{i}$ are the bases for ${Y}_c$ and ${\Psi }_{i}$ are the bases for ${Y}_q$   Let the SVD of ${Y}_c^{\top }{Y}_q \in \mathbb {R}^{m_c \times m_q}$ be ${Y}_c^{\top }{Y}_q = {U \Sigma V}^{\top }$ , where ${Y}_q$0 , ${Y}_q$1 represents the set of singular values  The canonical angles ${Y}_q$2 can be obtained as ${Y}_q$3 ${Y}_q$4   The similarity between the two subspaces is measured by ${Y}_q$5 angles as follows:
$$S_{({Y}_c,{Y}_q)}[t] = \frac{1}{t}\sum _{i = 1}^{t} \cos ^{2} \theta _{i},\; 1 \le t \le m_q, \; m_q \le m_c $$   (Eq  30)
Fig  1 shows the modeling and comparison of sets of words by MSM  This method can compare sets of different sizes, and naturally encodes proximity between sets with related words 
Finally, the class with the highest similarity with $d_q$ is assigned as the class of $d_q$ :
$$prediction(d_q) = argmax_c(S_{({Y}_c,{Y}_q)}) $$   (Eq  32)
TF weighted word subspace
The word subspace formulation presented in Section "Word subspace" is a practical and compact way to represent sets of word vectors, retaining most of the variability of features  However, as seen in the BOW features, the frequency of words is relevant information that can improve the characterization of a text  To incorporate this information into the word subspace modeling, we propose an extension of the word subspace, called the term-frequency (TF) weighted word subspace 
Like the word subspace, the TF weighted word subspace is mathematically defined as a low-dimensional linear subspace in a word vector space with high dimensionality  However, a weighted version of the PCA BIBREF25 , BIBREF26 is utilized to incorporate the information given by the frequencies of words (term-frequencies)  This TF weighted word subspace is equivalent to the word subspace if we consider all occurrences of the words 
Consider the set of word vectors $\lbrace {x}_c^k\rbrace _{k=1}^{N_c} \in \mathbb {R}^{p}$ , which represents each word in the context $c$ , and the set of weights $\lbrace \omega _i\rbrace _{i=1}^{N_c}$ , which represent the frequencies of the words in the context $c$  
We incorporate these frequencies into the subspace calculation by weighting the data matrix ${X}$ as follows:
$${\widetilde{X}}={X}{\Omega }^{1/2},$$   (Eq  33)
where ${X} \in \mathbb {R}^{p \times N_c}$ is a matrix containing the word vectors $\lbrace {x}_c^k\rbrace _{k=1}^{N_c}$ and ${\Omega }$ is a diagonal matrix containing the weights $\lbrace \omega _i\rbrace _{i=1}^{N_c}$  
We then perform PCA by solving the SVD of the matrix ${\widetilde{X}}$ :
$${\widetilde{X}}={AMB}^{\top },$$   (Eq  34)
where the columns of the orthogonal matrices ${A}$ and ${B}$ are, respectively, the left-singular vectors and right-singular vectors of the matrix ${\widetilde{X}}$ , and the diagonal matrix ${M}$ contains singular values of ${\widetilde{X}}$  
Finally, the orthonormal basis vectors of the $m_c$ -dimensional TF weighted subspace ${W}$ are the column vectors in ${A}$ corresponding to the $m_c$ largest singular values in ${M}$  
Text classification with TF weighted word subspace can also be performed under the framework of MSM  In this paper, we will refer to MSM with TF weighted word subspace as TF-MSM 
Experimental Evaluation
In this section we describe the experiments performed to demonstrate the validity of our proposed method and its extension  We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578  Words in the texts were considered as they appeared, without performing stemming or typo correction  This database has eight different classes with the number of samples varying from 51 to over 3000 documents, as can be seen in Table 1  
To obtain the vector representation of words, we used a freely available word2vec model, trained by BIBREF8 , on approximately 100 billion words, which encodes the vector representation in $\mathbb {R}^{300}$ of over 3 million words from several different languages  Since we decided to focus on English words only, we filtered these vectors to about 800 thousand words, excluding all words with non-roman characters 
To show the validity of our word subspace representation for text classification and the proposed extension, we divided our experiment section into two parts: The first one aims to verify if sets of word vectors are suitable for subspace representation, and the second one puts our methods in practice in a text classification test, comparing our results with the conventional methods described in Section "Related Work"  
Evaluation of the word subspace representation
In this experiment, we modeled the word vectors from each class in the Reuters-8 database into a word subspace  The primary goal is to visualize how much of the text data can be represented by a lower dimensional subspace 
Subspace representations are very efficient in compactly represent data that is close to a normal distribution  This characteristic is due to the application of the PCA, that is optimal to find the direction with the highest variation within the data 
In PCA, the principal components give the directions of maximum variance, while their corresponding eigenvalues give the variance of the data in each of them  Therefore, by observing the distribution of the eigenvalues computed when performing PCA in the modeling of the subspaces, we can suggest if the data is suitable or not for subspace representation 
For each class, we normalized the eigenvalues by the largest one of the class  Fig  2 shows the mean of the eigenvalues and the standard deviation among classes  It is possible to see that the first largest eigenvalues retain larger variance than the smallest ones  In fact, looking at the first 150 largest eigenvalues, we can see that they retain, on average, 86 37% of the data variance  Also, by observing the standard deviation, we can understand that the eigenvalues distribution among classes follows the same pattern, that is, most of the variance is in the first dimensions  This plot indicates that text data represented by vectors generated with word2vec is suitable for subspace representation 
Text classification experiment
In this experiment, we performed text classification among the classes in the Reuters-8 database  We compared the classification using the word subspace, and its weighted extension, based on MSM (to which we will refer as MSM and TF-MSM, respectively) with the baselines presented in Section "Related Work" : MVB, MNB, LSA, and SVM  Since none of the baseline methods work with vector set classification, we also compared to a simple baseline for comparing sets of vectors, defined as the average of similarities between all vector pair combinations of two given sets  For two matrices ${A}$ and ${B}$ , containing the sets of vectors $\lbrace  {x}^{i}_a \rbrace _{i = 1}^{N_A}$ and $\lbrace  {x}^{i}_b \rbrace _{i = 1}^{N_B}$ , respectively, where $N_A$ and $N_B$ are the number of main words in each set, the similarity is defined as:
$$Sim_{(A,B)} = \frac{1}{N_A N_B}\sum _{i}^{N_A}\sum _{j}^{N_B}{{x}_a^i}^{\top }{x}_b^j $$   (Eq  41)
We refer to this baseline as similarity average (SA)  For this method, we only considered one occurrence of each word in each set 
Different features were used, depending on the method  Classification with SA, MSM, and TF-MSM was performed using word2vec features, to which we refer as w2v  For MVB, due to its nature, only bag-of-words features with binary weights were used (binBOW)  For the same reason, we only used bag-of-words features with term-frequency weights (tfBOW) with MNB  Classification with LSA and SVM is usually performed using bag-of-words features and, therefore, we tested with binBOW, tfBOW, and with the term-frequency inverse document-frequency weight, tfidfBOW  We also tested them using word2vec vectors  In this case, we considered each word vector from all documents in each class to be a single sample 
To determine the dimensions of the class subspaces and query subspace of MSM and TF-MSM, and the dimension of the approximation performed by LSA, we performed a 10-fold cross validation, wherein each fold, the data were randomly divided into train (60%), validation (20%) and test set (20%) 
The results can be seen in Table 2   The simplest baseline, SA with w2v, achieved an accuracy rate of 78 73%  This result is important because it shows the validity of the word2vec representation, performing better than more elaborate methods based on BOW, such as MVB with binBOW 
LSA with BOW features was almost 10% more accurate than SA, where the best results with binary weights were achieved with an approximation with 130 dimensions, with TF weights were achieved with 50 dimensions, and with TF-IDF weights were achieved with 30 dimensions  SVM with BOW features was about 3% more accurate than LSA, with binary weights leading to a higher accuracy rate 
It is interesting to note that despite the reasonably high accuracy rates achieved using LSA and SVM with BOW features, they poorly performed when using w2v features 
Among the baselines, the best method was MNB with tfBOW features, with an accuracy of 91 47%, being the only conventional method to outperform MSM  MSM with w2v had an accuracy rate of 90 62%, with the best results achieved with word subspace dimensions for the training classes ranging from 150 to 181, and for the query ranging from 3 to 217  Incorporating the frequency information in the subspace modeling resulted in higher accuracy, with TF-MSM achieving 92 01%, with dimensions of word subspaces for training classes ranging from 150 to 172, and for the query, ranging from 2 to 109  To confirm that TF-MSM is significantly more accurate than MNB, we performed a t-test to compare their results  It resulted in a p-value of 0 031, which shows that at a 95% significance level, TF-MSM has produced better results 
Discussion
Given the observation of the eigenvalues distribution of word vectors, we could see that word vectors that belong to the same context, i e , same class, are suitable for subspace representation  Our analysis showed that half of the word vector space dimensions suffice to represent most of the variability of the data in each class of the Reuters-8 database 
The results from the text classification experiment showed that subspace-based methods performed better than the text classification methods discussed in this work  Ultimately, our proposed TF weighted word subspace with MSM surpassed all the other methods  word2vec features are reliable tools to represent the semantic meaning of the words and when treated as sets of word vectors, they are capable of representing the content of texts  However, despite the fact that word vectors can be treated separately, conventional methods such as SVM and LSA may not be suitable for text classification using word vectors 
Among the conventional methods, LSA and SVM achieved about 86% and 89%, respectively, when using bag-of-words features  Interestingly, both methods had better performance when using binary weights  For LSA, we can see that despite the slight differences in the performance, tfidfBOW required approximations with smaller dimensions  SVM had the lowest accuracy rate when using the tfidfBOW features  One possible explanation for this is that TF-IDF weights are useful when rare words and very frequent words exist in the corpus, giving higher weights for rare words and lower weights for common words  Since we removed the stop words, the most frequent words among the training documents were not considered and, therefore, using TF-IDF weights did not improve the results 
Only MNB with tfBOW performed better than MSM  This result may be because tfBOW features encode the word frequencies, while MSM only considers a single occurrence of words  When incorporating the word frequencies with our TF weighted word subspace, we achieved a higher accuracy of 92 01%, performing better than MNB at a significance level of 95% 
Conclusions and Future Work
In this paper, we proposed a new method for text classification, based on the novel concept of word subspace under the MSM framework  We also proposed the term-frequency weighted word subspace which can incorporate the frequency of words directly in the modeling of the subspace by using a weighted version of PCA 
Most of the conventional text classification methods are based on the bag-of-words features, which are very simple to compute and had been proved to produce positive results  However, bag-of-words are commonly high dimensional models, with a sparse representation, which is computationally heavy to model  Also, bag-of-words fail to convey the semantic meaning of words inside a text  Due to these problems, neural networks started to be applied to generate a vector representation of words  Despite the fact that these representations can encode the semantic meaning of words, conventional methods do not work well when considering word vectors separately 
In our work, we focused on the word2vec representation, which can embed the semantic structure of words, rendering vector angles as a useful metric to show meaningful similarities between words  Our experiments showed that our word subspace modeling along with the MSM outperforms most of the conventional methods  Ultimately, our TF weighted subspace formulation resulted in significantly higher accuracy when compared to all conventional text classification methods discussed in this work  It is important to note that our method does not consider the order of the words in a text, resulting in a loss of context information  As a future work, we wish to extend our word subspace concept further in mainly two directions  First, we seek to encode word order, which may enrich the representation of context information  Second, we wish to model dynamic context change, enabling analysis of large documents, by having a long-short memory to interpret information using cues from different parts of a text 
Acknowledgment
This work is supported by JSPS KAKENHI Grant Number JP16H02842 and the Japanese Ministry of Education, Culture, Sports, Science, and Technology (MEXT) scholarship 

ground_truth: Word vectors, usually in the context of others within the same class

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. by how much did their model improve?
Context 3:  This way, we obtain a synthetic parallel simplified-ordinary sentences  Both the synthetic sentences and the available parallel data are used as training data for the original NMT system 
Evaluation
We evaluate the performance of text simplification using neural machine translation on available parallel sentences and additional simplified sentences 
Dataset  We use two simplification datasets (WikiSmall and WikiLarge)  WikiSmall consists of ordinary and simplified sentences from the ordinary and simple English Wikipedias, which has been used as benchmark for evaluating text simplification BIBREF17 , BIBREF18 , BIBREF8   The training set has 89,042 sentence pairs, and the test set has 100 pairs  WikiLarge is also from Wikipedia corpus whose training set contains 296,402 sentence pairs BIBREF19 , BIBREF20   WikiLarge includes 8 (reference) simplifications for 2,359 sentences split into 2,000 for development and 359 for testing 
Metrics  Three metrics in text simplification are chosen in this paper  BLEU BIBREF5 is one traditional machine translation metric to assess the degree to which translated simplifications differed from reference simplifications  FKGL measures the readability of the output BIBREF23   A small FKGL represents simpler output  SARI is a recent text-simplification metric by comparing the output against the source and reference simplifications BIBREF20  
We evaluate the output of all systems using human evaluation  The metric is denoted as Simplicity BIBREF8   The three non-native fluent English speakers are shown reference sentences and output sentences  They are asked whether the output sentence is much simpler (+2), somewhat simpler (+1), equally (0), somewhat more difficult (-1), and much more difficult (-2) than the reference sentence 
Methods  We use OpenNMT BIBREF24 as the implementation of the NMT system for all experiments BIBREF5   We generally follow the default settings and training procedure described by Klein et al (2017)  We replace out-of-vocabulary words with a special UNK symbol  At prediction time, we replace UNK words with the highest probability score from the attention layer  OpenNMT system used on parallel data is the baseline system  To obtain a synthetic parallel training set, we back-translate a random sample of 100K sentences from the collected simplified corpora  OpenNMT used on parallel data and synthetic data is our model  The benchmarks are run on a Intel(R) Core(TM) i7-5930K CPU@3 50GHz, 32GB Mem, trained on 1 GPU GeForce GTX 1080 (Pascal) with CUDA v  8 0 
We choose three statistical text simplification systems  PBMT-R is a phrase-based method with a reranking post-processing step BIBREF18   Hybrid performs sentence splitting and deletion operations based on discourse representation structures, and then simplifies sentences with PBMT-R BIBREF25   SBMT-SARI BIBREF19 is syntax-based translation model using PPDB paraphrase database BIBREF26 and modifies tuning function (using SARI)  We choose two neural text simplification systems  NMT is a basic attention-based encoder-decoder model which uses OpenNMT framework to train with two LSTM layers, hidden states of size 500 and 500 hidden units, SGD optimizer, and a dropout rate of 0 3 BIBREF8   Dress is an encoder-decoder model coupled with a deep reinforcement learning framework, and the parameters are chosen according to the original paper BIBREF20   For the experiments with synthetic parallel data, we back-translate a random sample of 60 000 sentences from the collected simplified sentences into ordinary sentences  Our model is trained on synthetic data and the available parallel data, denoted as NMT+synthetic 
Results  Table 1 shows the results of all models on WikiLarge dataset  We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI  It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2 11 BLEU, 1 7 FKGL and 1 07 SARI  We also substantially outperform Dress, who previously reported SOTA result  The results of our human evaluation using Simplicity are also presented in Table 1  NMT on synthetic data is significantly better than PBMT-R, Dress, and SBMT-SARI on Simplicity  It indicates that our method with simplified data is effective at creating simpler output 
Results on WikiSmall dataset are shown in Table 2  We see substantial improvements (6 37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences  Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI  Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models  In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data 
Conclusion
In this paper, we propose one simple method to use simplified corpora during training of NMT systems, with no changes to the network architecture  In the experiments on two datasets, we achieve substantial gains in all tasks, and new SOTA results, via back-translation of simplified sentences into the ordinary sentences, and treating this synthetic data as additional training data  Because we do not change the neural network architecture to integrate simplified corpora, our method can be easily applied to other Neural Text Simplification (NTS) systems  We expect that the effectiveness of our method not only varies with the quality of the NTS system used for back-translation, but also depends on the amount of available parallel and simplified corpora  In the paper, we have only utilized data from Wikipedia for simplified sentences  In the future, many other text sources are available and the impact of not only size, but also of domain should be investigated 

ground_truth: For the WikiLarge dataset, the improvement over baseline NMT is 2.11 BLEU, 1.7 FKGL and 1.07 SARI.
For the WikiSmall dataset, the improvement over baseline NMT is  8.37 BLEU.

6.37 BLEU