user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Write by Learning the Objective

Paper content:
Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models for learning prevalent patterns in natural language.
Yet language generated by RNNs often shows several degenerate characteristics that are uncommon in human language; while fluent, RNN language production can be overly generic, repetitive, and even self-contradictory.
We postulate that the objective function optimized by RNN language models, which amounts to the overall perplexity of a text, is not expressive enough to capture the abstract qualities of good generation such as Grice’s Maxims.
In this paper, we introduce a general learning framework that can construct a decoding objective better suited for generation.
Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address the limitations of RNN generation.
Human evaluation demonstrates that text generated by the resulting generator is preferred over  that  of  baselines  by  a  large  margin  and  significantly  enhances  the  overall coherence, style, and information content of the generated text.
Recurrent Neural Network (RNN) based language models such as Long Short-Term Memory Networks (LSTMs) BID6 and Gated Recurrent Units (GRUs) BID2 have achieved enormous success across a variety of language tasks due to their ability to learn fluency patterns in natural language BID8 BID10 BID17 .
When used as a generator, however, the quality of language generated from RNNs deviates drastically from that of human language.
While fluent, RNN-produced language displays several degenerate characteristics, favoring generic and contentless output that tends to be repetitive and self-contradictory.
These issues are especially prominent when RNNs are used for open-ended, long-form text generation, as illustrated in Figure 1 .RNNs model the conditional probability P (x t |x 1 , ..., x t−1 ) of generating the next word x t given all previous words observed or generated.
In theory, this conditional model should be able to learn all crucial aspects of human language production, for example, that we don't normally repeat the same content over and over.
In practice, however, the learned conditional probability model often assigns higher probability to a repetitive, overly generic sentence than to higher quality sentences, as shown in Figure 1 .
We postulate that this is in part because the network architectures of RNN variants do not provide a strong enough inductive bias for the model to learn the complex communication goals pursued in human writing.
In addition, long-term context easily gets lost as it is explained away in the presence of more immediately relevant short-term context BID34 , and as gradients diminish over a long sequence BID21 .
Consequently, RNNs acquire relatively shallow and myopic patterns, which tend to only take advantage of a small fraction of the training set vocabulary BID9 .
RNNs are thus unable to generate language that matches the complexity and coherence of human generated text.
Several methods in the literature attempt to mitigate these issues.
Overly simple and generic generation can be improved by using a diversity-boosting objective function BID24 BID28 .
Repetitive generation can be reduced by prohibiting recurrence of the same trigrams as a hard rule BID22 .
Although such constraints form a partial solution, they All in all, I would highly recommend this hotel to anyone who wants to be in the heart of the action, and want to be in the heart of the action.
If you want to be in the heart of the action, this is not the place for you.
However, If you want to be in the middle of the action, this is the place to be.
Figure 1: A Trip Advisor review generated by an RNN based LM trained on over a million reviews.are generally too coarse and both penalize good behavior (e.g. reuse of an idiom) and fail to capture more complex bad behavior (e.g. paraphrasing of the same content again and again).Hand tailoring rules is both time consuming and unstable across different generative scenarios, so we instead propose a general learning framework to construct a better decoding objective.
Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address limitations of the base RNN generator.
Our learning framework therefore generalizes over various existing modifications to the decoding objective.
Our approach learns to overcome the particular limitations of the RNN generator directly by incorporating language generated from RNNs as negative samples to discriminatively train several companion models, each specializing in a different aspect of Grice's Maxims of communication BID4 ).Empirical results demonstrate that our learning framework is highly effective in converting a generic RNN language model into a substantially stronger generator.
Human evaluation confirms that language generated by our model is preferred over that of competitive baselines by a large margin and significantly enhances the overall coherence, style, and information content of the generated text.
We motivate our learning framework using Grice's Maxims of communication BID4 ):1.
Quantity: Make your contribution as informative as required, and no more than necessary.
RNN generations tend to violate this maxim as they are often overly short and generic, and therefore less informative than desired.
When encouraged to generate longer text, RNNs easily repeat themselves, which also works against conveying the right amount of information.
This observation motivates the length and repetition models in §3.2.1 and §3.2.2.
Do not say what you believe to be false.
When used for generation with long term context, RNNs often generate text that is self-contradictory.
We propose an entailment model to address this problem in §3.2.3.
RNNs used for long generation can start digressing as the long-term context gets easily washed out.
We address this problem by proposing two different relevance models in §3.2.4 and §3.2.5.
Avoid obscurity of expression.
Avoid ambiguity.
Be brief.
Be orderly.
As RNN generation favors highly probable but generic words and phrases, it lacks adequate style and specificity.
We address this issue with the lexical style model in §3.2.6.
We propose a general learning framework for conditional language generation of sequence y given context x. The decoding objective for generation takes the general form: DISPLAYFORM0 This objective combines the RNN language model probability P lm ( §3.1) with a set of additional scores s k (x, y) produced by discriminatively trained communication models ( §3.2) and weighted with learned mixture coefficients λ k ( §3.3).
This corresponds to a Product of Experts (PoE) model BID5 ) when the scores s k are log probabilities.
Further model and hyperparameter details are given in appendix B.Generation is performed using beam search ( §3.4), scoring partially generated candidate generations y 1:i at each time step i. The RNN language model decomposes into per-word probabilities using the chain rule.
However, in order to allow for more expressivity over long range context we do not require the discriminative model scores to factorize over the elements of y, thereby addressing a key limitation of RNNs.
More specifically, we use an estimated score s k (x, y 1:i ) that can be computed for any prefix of y = y 1:n to approximate the objective during beam search, such that s k (x, y 1:n ) = s k (x, y).
The scoring functions are trained on prefixes of y (except where stated otherwise) to simulate their application to rank partial continuations during beam search.
We train an RNN language model to estimate DISPLAYFORM0 The language model treats the context x and the continuation y as a single sequence s, in contrast to sequence to sequence models which distinguish between them.
Next we introduce a set of models motivated by Grice's Maxims of communication.
Each model is trained to discriminate between good and bad generation using a ranking objective, so that its model scores have the effect of re-ranking in the decoder.
We vary the model parameterization and training examples to guide each model to focus on different aspects of Grice's Maxims.
The classifier scores are interpreted as classification probabilities and added to the objective function as log probabilities.
Let D = {(x 1 , y 1 ), . . . (x n , y n )} be the set of training examples for conditional generation.
D x denote all contexts and D y all continuations.
In all models the first layer embeds words into 300-dimensional vectors initialized with GloVe (Pennington et al., 2014) embeddings, which are fine-tuned during training unless stated otherwise.
The dimensions of hidden layers are also 300.
Let e(w) be the word embedding of word w.
RNNs tend to bias toward shorter generation even with a highly expressive network structure with attention, thus length normalization is still a common practice .
We use a geometrically decaying length reward.
We tune the initial value and decaying rate on the final systems, and use the same value for all systems with the initial value and decay rate being 1 and 0.9 respectively.
The score per a time step (for a partially generated completion) is thus DISPLAYFORM0
The goal of this model is to learn to distinguish between RNN-generated and gold continuations by exploiting our empirical observation that repetitions are more common in completions generated by the RNN language model.
We quantify the claim that samples from the base RNN is an appropriate source of negative examples for repetition detection on our datasets in §4.1.
However, we do not want to completely discourage repetition, as words sometimes do recur in English.
Thus we model natural levels of repetition as follows.
First a score d i is computed for each token in the continuation y based on pairwise cosine similarity of word embeddings within a fixed window of the previous k words: DISPLAYFORM0 This score can be interpreted as the repetition "strength" of the sequence at position i, where d i = 1 for all positions at which a word is repeated within k + 1 words of its previous use.
The score of the entire continuation is then defined as DISPLAYFORM1 where RNN(d) is the final state of a unidirectional RNN ran over the similarity scores and w r is a learned vector.
The model is trained to maximize binary cross-entropy, with gold continuations as positive examples and samples from the base RNN as negative examples: DISPLAYFORM2 Word embeddings are kept fixed during training for this model.
The goal of the entailment model is to guide the generator to neither contradict its own past generation (the maxim of Quality) nor state something that readily follows from the context (the maxim of Quantity).
The latter case is driven by the RNNs habit of paraphrasing itself (alongside its tendency for more direct repetitions, which is captured by the repetition model).
We train a classifier using the MultiSNLI dataset BID30 ) that takes two sentences a and b as input and predicts the relation between them as either contradiction, entailment or neutral.
We use the score of the neutral class probability of the sentence pair in order to discourage both contradiction and entailment.
We use a continuous bag-of-words model similar to previously reported NLI baselines BID19 .
Sentence representations are obtained by summing word embeddings (which are fine-tuned during training), such that r(a) = i e(a i ) and r(b) = i e(b i ).
An MLP classifier with a single tanh hidden layer takes as input concatenations of the two sentence embeddings, along with their element-wise difference and multiplication; the output is a three-way softmax.
The log probability of the neural class is t(a, b).
The classifier obtains 63% validation accuracy.
In contrast to our other communication models, this classifier cannot be applied directly to the full context and continuation sequences it is scoring.
Instead every completed sentence in the continuation should be scored against all preceding sentences in both the context and continuation.
Let S(y 1:i ) be the set of complete sentences in y 1:i , and S −1 (y 1:i ) the last complete sentence.
We compute the entailment score of S −1 (y 1:i ) against all preceding sentences in x and y, and use the score of the sentence-pair for which we have the least confidence that entailment is neutral: DISPLAYFORM0 In contrast to our other models, the score this model returns for any given continuation only corresponds to a subsequence of the continuation, but as we will see below ( §3.4) due to the way generation is performed with beam search this score will not be accumulated across sentences.
The purpose of the relevance model is to predict whether the content of a candidate continuation is relevant to the given context.
We train the model to distinguish between true continuations and random continuations sampled from other (human-written) endings in the corpus, conditioned on the given context.
A single convolutional layer is applied to both the context and candidate embedding sequences.
The scoring function is defined as DISPLAYFORM0 where 1D maxpooling is performed over each sequence to obtain a vector representing its most important semantic dimensions.
Element-wise multiplication of the context and continuation vectors will amplify similarities between the two.
We optimize the following ranking log likelihood: DISPLAYFORM1 i.e., the probability of the true ending y t receiving a higher score than the random ending y r .
For each context and true continuation we extract k = 5 randomly selected endings as negative training examples.
The model ranks true endings higher than randomly selected ones with 85% accuracy on the validation set.
The trained scores do not correspond to probabilities but we scale with the logistic (sigmoid) function for compatibility with other scoring modules.
Given even a fragment of context (e.g. "We were excited about our beach getaway...") certain topics become more relevant (e.g. "swimming", "relax", "romantic").
To capture this intuition, we predict a centroid in the embedding space from the context, which describes a neighborhood of the embedding space we expect the continuation to intersect.
The score is computed as DISPLAYFORM0 where RNN(x) is the final state of an RNN trained end-to-end with s voc using a discriminative loss: DISPLAYFORM1 During decoding, this score is combined with the language model before sampling next word candidates, while the other models are used to rescore candidates (see §3.4).
The reason for this is that this model improves diversity in the next word distribution, while encouraging continuation of topics in the context.
The goal of the lexical style model is to learn stylistic aspects of desirable writing based on observed lexical distributions.
The scoring function is defined as DISPLAYFORM0 The model is trained with a similar ranking criteria as the relevance model, except that the negative examples are sampled from the language model: DISPLAYFORM1
Once all the communication models have been trained, we learn the combined objective.
In particular we learn the weight coefficients λ k in equation 1 to linearly combine the scoring functions, using a discriminative objective DISPLAYFORM0 where A is the inference algorithm for beam search decoding.
The objective learns to rank the gold continuation above the continuation predicted by the current model.
The full model is therefore trained discriminatively to classify sequences, and the same model is used to generate the sequences.
For each training iteration inference is performed based on the current values of λ.
This has the effect that the objective function changes dynamically during training: As the current samples from the model are used to update the mixture weights, it creates its own learning signal by applying the generative model discriminatively.
Due to the limitations of greedy decoding and the fact that our scoring functions do not decompose across time steps, we perform generation with a beam search procedure, shown in Algorithm 1.
The naive approach would be to perform beam search based only on the language model, and then rescore the k best candidate completions with our full model.
We found that this approach leads to limited diversity in the beam and therefore cannot exploit the strengths of the full model.
Therefore we rescore the current hypotheses in the beam with the full (partial) objective function and keep the k best candidate sequences after expanding to the next word.
To further increase diversity we sample k candidate next words from the distribution when expanding a hypothesis, instead of obtaining the k highest scoring next words.
In our experiments we generate text using a beam size of 8.Two modules are handled differently during beam search: First, the working vocabulary score is integrated before sampling next word candidates, i.e. inside the next k call.
Second, the entailment score of a candidate is recomputed only when sentence-terminating punctuation is generated.
Otherwise the current entailment score is re-used.
Due to the nature of beam-search the entailment score is not accumulated across sentences; rather we assume that the effect of the last sentence score will already be reflected in the content of the beam when the next sentence is scored.
While the first sentence is generated, s entail takes an initial constant value to avoid bias towards incomplete sentences.
Corpora We use two corpora for evaluation.
The first is TripAdvisor reviews.1 We use only reviews that have at least 5 sentences, using the first 4 as the context and the remainder as the ending to be generated.
There are on average 11 sentences per review.
We use the first 1M reviews for training.
The second is the ROCStory corpus BID18 , which is a collection of crowdsourced commonsense short stories, consisting of five sentences each (98k stories in the training set).
We use this corpus to train our model to predict a coherent final sentence given the first four.
For the (larger) TripAdvisor corpus, we train the language model on that corpus alone.
For the ROCStory corpus, we pretrain the model for 420M tokens on the Toronto Books corpus 2 before training on the ROCStory text.
To analyze the suitability of the reference endings and base language model samples to train the repetition model, we define a repetition ratio as the ratio of total words to unique words.
We found that ROCStory endings generated by the language model have a higher repetition ratio than the reference endings (1.12 vs. 1.02).
On TripAdvisor the difference is more pronounced, with a ratio of 1.48 for the language model against 1.17 for the references.
Previous work has reported that automatic measures such as BLEU, ROUGE, and Meteor, do not lead to meaningful evaluation when used for long or creative text generation where there can be high variance among correct generation output BID32 BID27 .
For open-ended generation tasks such as our own, human evaluation is the only reliable measure BID15 BID32 .
However, we also report the automatic measures to echo the previous observation regarding the mismatch between automatic and human evaluation.
Additionally we provide multiple generation samples that give insights into the characteristics of different models.
We pose the evaluation of our model as the task of generating an appropriate ending given an initial context of n sentences.
For the purposes of automatic evaluation, the ending is compared against a gold reference ending that was written by a human (drawn from the original corpus).
For human evaluation, the ending is presented to a human, who assesses the text according to several criteria, which are closely inspired by Grice's Maxims:1.
Quantity: Does the generation avoid repetition?
2. Quality: Does the generation contradict itself (or the initial context)?
3. Relation: Does the generation appropriately continue the initial context?
4.
Manner: Does the generation use the same style as the initial context?Finally, a Turing test question is presented to each judge: was the ending written by a human?For both automatic and human evaluation, we select 1000 passages from the test set of each corpus and generate endings using all models, using the initial n = 4 sentences as context.
Automatic Table 2 : Scores of baselines and our model on both datasets.
Top: TripAdvisor, bottom: ROCStory.
Automated metrics are from 0-100.
Human metrics are from 1-5, except for the Turing test, which is from 0-100.
Each cell is the micro-averaged across 1000 datums from a held-out test set.evaluation scores each generated ending against its reference ending.
To conduct human evaluation, we present the initial context with the generation from each of the models individually to workers on Mechanical Turk, who score them using the above criteria.
As a control, we also ask workers to score the original (human-written) reference endings for each selected passage.
Results for both automatic and human evaluation metrics are presented for both corpora in Table 2 .For the TripAdvisor corpus, the language model baseline does best on all of the automated metrics.
However on all human metrics, the L2W (NO WK.VOCAB) model scores higher, and achieves nearly twice the percentage of passed Turing tests (75% compared to 39%).
It scores even better than the L2W (FULL) model.
Manual inspection reveals that for the TripAdvisor task, the topical focus and lexical specificity given by vocabulary scoring actually lead to slightly worse model behavior.
In TripAdvisor reviews, many possible sentences could logically follow a reviews context.
A well-behaving but generic language generator is better able to leave judge's unsurprised, if also uninformed.
The full model brings additional focus to the generations, but at the cost of a reduction in overall coherence.
In the ROCStory generation task the language model is more competitive with L2W (FULL).
The raw language model is able to take advantage of the simpler data inherent in this task: ROCStories are shorter (the endings are always a single sentence) and have a smaller vocabulary.
However, the L2W full model still performs the best on all human metrics.
The ROCStory corpus presents specific contexts that require a narrower set of responses to maintain coherency.
The L2W (FULL) takes advantage of reranking by matching the topic of the context, yielding a higher 'Relation' score and more Turing test passes overall.
The very low BLEU scores observed in our results in the TripAdvisor domain are an artifact of the BLEU metrics length penalty.
The average length of reference completions is 12 sentences, which is much longer than the average length of endings generated by our Learning to Write models.
This forces the BLEU score's length penalty to drive down the scores, despite our observation that there is still a significant amount of word and phrase overlap.
The completions generated by the base language model are longer on average (as it tends to repeat itself over and over) and therefore to not suffer from this problem.
Learning to Write (L2W) generations are more topical and coherently mold with the context.
TAB2 shows that both the L2W system as well as the classic RNN start similarly, commenting on the hotel staff and the room.
L2W is able to condense the same content into fewer words, following the context's style.
Beginning with the third sentence, L2W and the LM diverge more significantly.
The LM makes a generic comment: "The location is perfect," while L2W goes on to list specific items it enjoyed at breakfast, which was mentioned in the context.
Furthermore, the LM becomes "stuck" repeating itself-a common problem for RNNs in general-whereas the L2W system finds a natural ending.
The L2W models do not fix every degenerate characteristic of RNNs.
The TripAdvisor L2W generation in TAB2 , while coherent and diverse, leaves room for improvement.
The need to relate to topical phrases can override fluency, as with the long list of "food, fresh fruit, juices, coffee, tea, hot chocolate."
Furthermore phrases such as "friendly and helpful" and "clean and comfortable" occur in the majority of generations.
These phrases, while common, tend to have semantics that are expressed in many different surface forms in real writing (e.g., "the staff were kind and quick to respond").
Appendix A presents sample generations from all models on both corpora for further inspection.
Generation with Long-term Context While relatively less studied, there have been several attempts at RNN-based paragraph generation for multiple domains including image captions BID13 , product reviews BID16 BID3 , sport reports BID31 , and recipes BID9 .
Most of these approaches are based on sequence-tosequence (seq-to-seq) type architectures.
One effective way to avoid the degenerative characteristics of RNNsunder seq-to-seq is to augment the input with sufficient details and structure that can guide the output generation through attention and a copying mechanism.
However, for many NLP generation tasks, it is not easy to obtain such the large-scale training data required to support a robust sequence-to-sequence model.
For example, a recent dataset for image-to-paragraph generation contains 14,575 training pairs BID13 , and state-of-the-art hierarchical models, while improving over strong baselines, still lead to generation that repeats, contradicts, and is generic.
Alternative Decoding Objectives The tendency of RNN generation to be short, blend, repetitive and contradictory has been noted multiple times in prior literature.
A number of papers propose approaches that can be categorized as alternative decoding objectives BID24 .
For example, BID14 propose a diversity-promoting objective that interpolates the conditional probability score with negative marginal or reverse conditional probabilities.
Unlike the negative marginal term that blindly penalizes generic generation, all our communication models are contextual in that they compare the context with continuation candidates.
Incorporating the reverse conditional probability has been also proposed through the noisy channel models of BID34 .
The clear benefit of the reverse conditional probability model is that it can prevent the explaining away problem of the long-term context.
However, decoding with the reverse conditional probability is significantly more expensive, making it impractical for paragraph generation.
In addition, both above approaches do not allow for learning more expressive models than our work presents.
The communication models presented in our work can be easily integrated into the existing beam search procedure, and each model is lightweight to compute.
The modified decoding objective has long been a common practice in statistical machine translation literature BID12 BID20 BID29 BID1 .
Notably, it still remains a common practice with neural machine translation, even with a highly expressive network structure trained on an extremely large amount of data .
Inspired by all above approaches, our work presents a general learning framework together with a more comprehensive set of composite communication models.
Unlike previous approaches, our composite models are trained to directly address the particular limitations of the base RNN models, by integrating RNN generation into the discriminative learning.
This allows for customization of the decoding objective to better cope with the undesired behavior of the base language models.
Meta Learning There has been a broad range of literature on meta learning, where the learning goal broadens the search space of learning by targeting to learn model architectures, hyperparameters, and learning algorithms that are typically hand-designed or selected.
BID0 , for example, learn the learning rate, while BID35 train a neural network to gen-erate neural network architectures.
while BID25 proposed Bayesian optimization to learn hyperparameters that cannot be optimized with gradient descent.
The learning framework we presented here can be interpreted as a type of meta learning in a broad sense in that there are multiple layers of learning: learning the base RNN language model, then learning a collection of composite communication models that are customized to the particular behavior of the base model, and finally the generator that learns to combine all sub-components.
Our work presents a unified learning framework that can learn to generate long, coherent text overcoming the limitations of RNNs as text generation models.
Our framework learns a decoding objective suitable for generation through a combination of sub-models that capture linguisticallymotivated qualities of good writing.
Our work makes a unique contribution that complements existing literature on long text generation that is predominantly based on seq-to-seq models with a large amount of in-domain training data; we demonstrate that the fluency of general RNN language models can be successfully guided to generate more lengthy and sensical text, which can be useful for domains where in-domain data is not sufficient to support seq-to-seq type training.
We propose a general framework for learning a decoding objective in two parts: learning component models to rank candidate generations, which are motivated by different aspects of Grice's Maxims, and learning a weighing scheme that balances the influence of each of these scoring functions.
This framework is amenable to any number and kind of sub-objectives, allowing for multiple qualities of good writing to be modeled across diverse domains.
Human evaluation shows that the quality of the text produced by our model exceeds that of RNN baselines by a large margin and the generations score significantly higher on a Turing test evaluation.
Context The location is amazingly convenient.
Just 3min walk from the bus station or 1min walk from the train station and less than a 100m from the canal grande.
I liked the fact that it's at the start of the city so that you begin your venice experience slowly.
I recommend walking to the san marco piazza on your first time ( make sure you wear comfortable shoes ).
Language Model The hotel is in a great location.
The room was clean and comfortable.
The staff was very friendly and helpful.
The breakfast was very good.
The staff was very friendly and helpful.
I would recommend this hotel to anyone who is looking for a nice place to stay in venice.
I would recommend this hotel to anyone who is looking for a place to stay in venice.
I would recommend this hotel to anyone who is looking for a place to stay while in No Meta-Learning The staff were very friendly and helpful and the breakfast buffet Learn2Write The rooms are spacious and clean.
Breakfast is very good.
Human Then you exit the alley and face the magnificent san marco piazza and trust me, you will be pleasantly shocked.
What an experience!!! : ) ) the abbazia rooms are decent and very clean.
Breakfast is poor but adequate and wi-fi is free.
The garden is very peaceful and offers some very relaxing moments.
I was worried about noises from the train station next door but you can't hear a thing so no problem there.
The guys at the reception are amazing.
Very friendly and very helpful : ) ) what you want from a hotel in venice is a decent place to sleep, have a relaxing bath and some breakfast in the morning.
From then on you will be spending all your time in town anyway so fo me the abbazia hotel was an excellent choice and i will go back for sure.
Price is not cheap, but nothing is cheap in venice anyway.
Base language model We use a 2-layer RNN language model with 1024 GRU cells per layer.
Embeddings vectors are of length 1024 .
Following Inan et al. (2017 we tie the input and output embedding layers' parameters.
To regularize we dropout BID26 cells in the output layer of the first layer with probability 0.2.
We use mini-batch stochastic gradient descent (SGD) and anneal the learning rate regularly when the validation set performance fails to improve.
Learning rate, annealing rate, and batch size were tuned on the validation set for each dataset, with details in section ??.
Gradients are backpropagated 35 time steps and clipped to a maximum value of 0.25.Entailment model Dropout is performed on both the input word embeddings and the MLP hidden layer with rate 0.5.
Training is performed with Adam (Kingma & Ba, 2015) with learning rate 0.0005, batch size 128.Relevance model The convolutional layer is a 1D convolution with filter size 3 and stride 1; the sequences are padded so that the sequence output is the same length as the input.
The model is trained with Adam, learning rate 0.001 and dropout is applied before the final linear layer with rate 0.5.
Training is performed with SGD with a learning rate of 1.

assistant: text: This paper combines RNN language model with several discriminatively trained models to improve the language generation.