A BIRD’S EYE VIEW ON COHERENCE, AND A WORM’S EYE VIEW ON COHESION
Year: 2018

Abstract:
Generating coherent and cohesive long-form texts is a challenging problem in natural language generation. Previous works relied on a large amount of humangenerated texts to train neural language models, however, few attempted to explicitly model the desired linguistic properties of natural language text, such as coherence and cohesion using neural networks. In this work, we train two expert discriminators for coherence and cohesion to provide hierarchical feedback for text generation. We also propose a simple variant of policy gradient, called negative-critical sequence training in which the reward baseline is constructed from randomly generated negative samples. We demonstrate the effectiveness of our approach through empirical studies, showing improvements over the strong baseline – attention-based bidirectional MLE-trained neural language model – in a number of automated metrics. The proposed model can serve as baseline architectures to promote further research in modeling additional linguistic properties for downstream NLP tasks.

heading: 1 INTRODUCTION
text: The terms coherence and cohesion in linguistics are commonly defined as follows (Williams & Colomb, 1995).
Cohesion: sentence pairs fitting together the way two pieces of a jigsaw puzzle do. Coherence: what all the sentences in a piece of writing add up to, the way all the pieces in a puzzle add up to the picture on the box.
In layman’s terms, cohesion indicates that two consecutive sentences are locally well-connected, and coherence indicates that multiple sentences globally hold together.
Generating cohesive and coherent natural language texts that span multiple sentences is a challenging task mainly due to two reasons. First, there is no formal specification of cross-sentence linguistic properties, such as coherence and cohesion of a text. Second, there is no widely accepted surrogate neural model to evaluate the two properties.
Most state-of-the-art neural approaches to natural language generation (NLG) relied on a large amount of human-generated texts to train language models (Cho et al., 2014; Graves, 2013; Sutskever et al., 2014). Although these models can generate sentences that, if judged individually, are similar to human-generated ones, they often fail to capture the local and global dependencies among sentences, resulting in a neither coherent nor cohesive text. For example, neural language models based on Recurrent Neural Networks (RNNs) are widely applied to response generation for dialogue (Vinyals & Le, 2015; Shang et al., 2015; Sordoni et al., 2015; Li et al., 2015). Although the responses by themselves look reasonable, they are either bland such as “I don’t know”, or incoherent with the whole dialogue session. See Gao et al. (2018) for a comprehensive survey.
In this paper, we strive to address the challenge in a principled manner. We propose a pair of discriminators to score whether and to what extent a text is coherent or cohesive. The coherence discriminator measures the compatibility among all sentences in a text using sentence-level features, thus providing a macro-level view on the multi-sentence text. The cohesion discriminator, on the other hand, measures the compatibility of each pair of consecutive sentences using only word-level features, thus providing a micro-level view on neighboring sentences. These models, given a conditional input text and multiple candidate output texts, are learned to score the candidates with respect
to the criterion by optimizing pairwise ranking losses. These scores are then used as reward signals to train an RNN-based language model to generate (more) coherent and cohesive texts.
Our main contributions are three-fold: (1) we propose two linguistic discriminators for modeling coherence and cohesion of a text; (2) we present a simple yet effective training mechanism to encode these linguistic properties; and (3) we propose negative-critical sequence training, a variant of policy gradient method, which uses negative samples to construct its reward baseline.
This paper proposes a neural approach to explicitly model cross-sentence linguistic properties, i.e., coherence and cohesion, for long text generation. Despite the encouraging initial results, we only scratched the surface of the problem. The proposed method is yet to be significantly improved to meet the ultimate goal of generating meaningful and logical long-form texts. We cast the text generation as an RL problem and review recent work in Section 2, and detail our approach in Section 3.

heading: 2 RELATED WORK
text: Coherence and cohesion. Coherence and cohesion have been extensively studied in the computational linguistics community, particularly in the ‘pre-deep-learning’ era. Lack of formal specifications for coherence and cohesion (Mani et al., 1998), resulted in many different formalism, such as Rhetorical Structure Theory (Mann & Thompson, 1988), and other forms of coherence and cohesion relations and their quantification (Mani et al., 1998; Hobbs, 1985; Hovy, 1988; McKeown, 1985; Cohen & Levesque, 1985; Hovy, 1991; Cristea et al., 1998; Halliday & Hasan, 1996; Liddy, 1991; Van Dijk, 2013; Edmundson, 1969; Barzilay & Lapata, 2008). In fact, the list is not exhaustive. However, modeling coherence and cohesion of a text using models parametrized by neural networks have not been previously explored.
Word sequence generation in a reinforcement learning framework. A word sequence generation task can be framed as a reinforcement learning (RL) problem, in which the generatorG is acting as a policy π, with parameters θπ , and each generated word at time t, wt, can be viewed as an action to be chosen by the policy from a large discrete space, or vocabulary, conditioned on state st−1 = w≤t−1, which encodes the previously generated text sequence.
Let rt be the reward for a partially generated text sequence w≤t. We define the long-term expected reward J (π) = Es0∼q,π[ ∑∞ t=1 γ
t−1rt], where q is the initial distribution of conditional input texts. Following Sutton et al. (1999), the gradient of J with respect to θπ is
∇θπJ = Es∼ρπ,a∼π(·|s)[Qπ(s, a)∇θπ log πθπ (a|s)]
where ρπ is the stationary distribution and Qπ(s, a) is the expected return from state s and taking action a, both following policy π. For brevity, we omit the derivation. In our work, we formulate text generation as an episodic RL problem with episode length L, rewards rL being available only at the end of episode and γ = 1.
There are many works on training neural language models using reward signals, such as Ranzato et al. (2015) and Paulus et al. (2017). These works directly optimize for specific metrics, such as BLEU (Papineni et al., 2002) or ROUGE (Lin & Hovy, 2003), using REINFORCE (Williams, 1992; Sutton et al., 1999). However, it is well-known that these metrics do not give a complete picture on the quality of the generated text. Only recently have there been efforts to provide more relevant quality objectives for which to optimize (Li et al., 2015; 2016a; Holtzman et al., 2018) the quality of interest such as consistency, repetition of text. But these works use the objective function to re-rank candidate outputs, not to reward or penalize outputs when they are generated in the first place. Li et al. (2016b) constructed a set of reward models, such as information flow and semantic coherence, to tune the generator, yet they do not provide an ablation study to elaborate the relative contribution of these reward models individually.
GANs for text generation. Another line of research is to use Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) to incorporate feedback signals for text generation (Yu et al., 2017; Lin et al., 2017; Zhang et al., 2017c; Guo et al., 2017; Fedus et al., 2018; Zhang et al., 2018). However, the discriminator in these works are trained to distinguish real texts from the generated ones, operating as a black-box rather than providing fine-grained feedback on particular linguistic aspects of the texts. In fact, Yang et al. (2018) has partially addressed this issue by using a trained language model as the discriminator. Although the discriminator provides a fine-grained feedback at
the word-level, it does not critique on many important linguistic properties of generated texts, such as cohesion and coherence.
These text generators, when facing a long-form text generation task that span multiple sentences, are by no means perfect and often exhibit some critical errors, such as a breakdown of local connections between consecutive sentences (cohesion), let alone globally solid intention (coherence). As a result, readers can easily take these cues and discriminate such generated texts against real texts. In this paper, we argue that the primary reason is the lack of an effective mechanism of measuring and controlling the text quality in the generation process. The method we propose in the next section is intended to address the problem.

heading: 3 MODEL
text: We assume that global coherence of a text depends to a large degree upon how its individual sentences with different meanings are organized. So we focus our evaluation of coherence solely on the sentence-level. If the sentences are not organized properly, the intention of the paragraph as a whole is obscure, regardless of seamless local connectivity between consecutive sentences.
This is not to say that local connections between any two neighboring sentences should be overlooked. One can easily distinguish a model-generated sentence from a real one, simply by looking at whether the sentence followed by another sentence relates, besides grammar.
We instill these two different yet important concepts in two discriminators, each operating on the sentence level and word level. Our models closely resemble successful models for computer vision, such as StackGAN (Zhang et al., 2017a;b) and PatchGAN (Isola et al., 2017) in that they all provide hierarchical signals to their corresponding generators, where the signals are derived from raw lowlevel data. We call the sentence-level discriminator the coherence discriminator Dcoherence, and the word-level discriminator the cohesion discriminator Dcohesion.
3.1 COHERENCE DISCRIMINATOR: DCOHERENCE
This discriminator measures how likely two text chunks form a coherent paragraph. Let S := [s1, s2, ..., sn] be the source text chunk that consists of n sentences, T := [t1, t2, ..., tm] be the real target text chunk that consists of m sentences, and T̃ := [ t̃1, t̃2, ..., t̃m̃ ] be the model-generated target text chunk that consists of m̃ sentences. Dcoherence is designed to distinguish a real pair (S, T ) from a synthetic pair (S, T̃ ) by assigning them with different scores, i.e., Scorecoherence(S, T ) > Scorecoherence(S, T̃ ).
Design. Our design ofDcoherence is inspired by the Deep Structured Semantic Model (DSSM) (Huang et al., 2013; Gao et al., 2014; Xu et al., 2017). Given source text chunk S and target text chunk T , Dcoherence computes their coherence score in two steps, as illustrated in Figure 1. First, a pair of convolutional networks (CNNs)1 or RNNs are applied to encode both S and T into two lowdimensional continuous vectors. Second, the coherence score is computed as the cosine similarity of the two vectors. The score is a real value between −1 and 1, where 1 indicates the maximal coherence score, and −1 the minimal coherence score. Dcoherence measures how likely S and T add up to a single coherent passage. The score depends on the parameters of the two CNNs/RNNs, or in other words, how S and T are encoded. Since we focus solely on the sentence-level features, we view a text chunk as a sequence of sentences, and view each sentence as a bag-of-words. Therefore, we represent each word using its pre-trained word embedding vector (Pennington et al., 2014) and represent each sentence using a vector which takes the average of its word embedding vectors. A text chunk is then represented as a sequence of sentence vectors which are fed to the CNN/RNN (either f(·) or g(·) in Figure 1). Notice that f(·) and g(·) do not share parameters. The parameters of f(·) and g(·) are optimized in such a way that a real pair scores higher than a synthetic pair:
∆(θf , θg) := Dcoherence(fθf (S), gθg (T ))−Dcoherence(fθf (S), gθg (T̃ )) > 0
Formally, the task of optimizingDcoherence can be stated as follows. Given a set of training samples of the form ((S, T ), (S, T̃ ))(i), i = 1...M , we optimize parameters (θf , θg) by minimizing the pairwise rank loss on training data defined as
1
M M∑ i=1 L(∆(θf , θg) (i))
where L(·) is a loss function, differentiable w.r.t. (θf , θg). In the following subsection, we will describe in turn how we construct the pairwise training samples and the form of the loss function. Since Dcoherence is used as a pairwise ranker, we employ the metrics commonly used in information retrieval for evaluation, such as recall at K (R@K), which is defined as the fraction of correctly identifying an item in the TOP-K retrieved list (Baeza-Yates & Ribeiro-Neto, 1999). We present the retrieval results on test data in Table 2.
Training mechanism. How do we construct these list of candidate target sentences T , given the source sentences S? We assume the T that follows an S in the data is a positive target sample, or the correct item to retrieve. Negative samples are constructed using three different methods within a batch while iterating through the training data, motivated by Wieting et al. (2016):
• Rotate T with S fixed (mismatch S and T ) across a batch. For a single S, this method yields B − 1 negative samples, where B is the batch size. • Shuffle the sentence order once, different from its original order, known as a derangement, in each T to break coherence, and this yields one negative sample. • Combine the previous two methods: rotate T across a batch and shuffle sentences within T , yielding B − 1 negative samples.
These 2B − 1 negative samples and a single positive sample, in sum, pose a significant challenge in learning. To fit this training task into a ranking framework, we optimize over
Dcoherence(fθf (S), gθg (T ))− AMλi∈{1,...,2B−1} [ Dcoherence(fθf (S), gθg (T̃i)) ] where the arithmetic mean parametrized by λ: AMλ(x) =
N∑ i wixi and wi = eλxi/ ∑ j e λxj .
In our experiments, we fix λ = 2 and this assigns more weight to a more challenging negative sample2. Notice that AMλ is the mean if λ = 0, and approaches the max as λ → ∞. Empirically, training the models using the weighted mean resulted in faster convergence, as opposed to using the single most challenging negative sample score (max) or the mean of all negative sample batch.
1We explored with deeper networks. However, the performance difference was marginal. For simplicity, we decided to use a 1-layer convolutional network architecture (Kim, 2014; Collobert et al., 2011).
2We performed a coarse grid search over the values of λ and setting λ = 2 resulted in fast convergence to high recall scores on the dev dataset.
3.2 COHESION DISCRIMINATOR: DCOHESION
Our second discriminator pays attention only to low-level features to capture a local connection between arbitrary two consecutive sentences because the pairwise connectivity influence readability.
For simplicity,Dcohesion is similar toDcoherence, except that its architecture, input, and negative sample construction are modified to encode cohesion between any pair of sentences on the word-level. A single input sample to Dcohesion is a pair of two consecutive sentences: [si,1, si,2, ..., si,n] and [si+1,1, si+1,2, ..., si+1,m], where si,k denotes the k-th word in sentence i. We construct the negative samples using the three methods for training Dcoherence, where shuffling occurs on the word level within each sentence, rather than shuffling multiple sentences on the sentence level.

heading: 3.3 GENERATOR: G
text: The two pre-trained discriminators, Dcoherence and Dcohesion, are used to modify the text generation behavior of G. G is an attention-based bidirectional sequence-to-sequence model. It is initially pretrained via maximizing the word-level likelihood given the training data, and we denote as GMLE.
However, the model-generated texts from GMLE often do not hold to the standards of two discriminators. Therefore, we need to change the text generation behavior of G with respect to the criteria. To this end, the scores from the discriminators that model the criteria are used as direct reward or penalty signals to modify the parameters of G. Given these signals, we use our proposed variant of the policy gradient, negative-critical sequence training, to apply parameter updates inG. We discuss the details in the next section.

heading: 4 NEGATIVE-CRITICAL SEQUENCE TRAINING
text: Actor-critic methods (Barto et al., 1983; Witten, 1977) parameterized by neural networks typically require learning a separate critic network to estimate the expected future reward as a baseline, which in many cases is a difficult task by itself. In NLP, we have observed similar practices and challenges by Ranzato et al. (2015), Bahdanau et al. (2016), and Nguyen et al. (2017). However, recently Rennie et al. (2017) proposed an effective self-critical sequence training that avoids learning a separate critic network. Similarly, our method does not require learning a separate critic network, instead we directly use the scores of negative samples assigned by the discriminators as the baseline.
For an arbitrary pair of S and Tgen, which is the generator’s output conditioned on S, we compute the coherence and cohesion scores by calling Dcoherence and Dcohesion. Since each review consists of multiple sentences, the overall cohesion score is computed as the average of scores of all consecutive sentence pairs. These scalar scores, however, have no interpretation since the discriminators are trained by optimizing a margin ranking loss. Instead, the differences between positive sample scores and the maximal or average negative sample scores provide insight of how well the models can distinguish between the positives and the negatives. Therefore, these margins can be considered as rewards with baselines, and thus we define the reward functions as:
Rcoherence(S, T ) := Dcoherence(fθf (S), gθg (T ))− ET̃ [Dcoherence(fθf (S), gθg (T̃i))] Rcohesion(si, si+1) := Dcohesion(hθh(si), jθj (si+1))− Es̃i+1 [Dcohesion(hθh(si), jθj (s̃i+1))]
where ·̃ denotes a negative sample for a given source condition, and ET̃ ( and Es̃i+1 ) are computed by averaging over an ensemble of negative samples. Notice that this reward resembles the ranking loss we use to train our discriminators, except that our baseline is an average score (instead of the weighted arithmetic mean) over negative samples. The rational for this difference is that: the maximal or the weighted arithmetic mean score baseline is in fact noisy to be used as rewards, because the best randomly constructed negative samples may be a formidably good sample. To alleviate such noise, we use the average discriminator scores of negative samples as the baseline, and this turns out to be an empirically better alternative. Finally, we use policy gradient (Williams, 1992; Sutton et al., 1999) to maximize a weighted combination of the coherence and cohesion rewards. For illustrative purposes, we equally weigh them for updating our policy, i.e., the generator G.
source sentences the hotel inglaterra delivered as promised . the staff was welcoming and spoke good english . the cleaning staff did a very good job every day . the rooms were spotless and very modern . the bathroom was large and had a very nice shower , and there were two generously sized bath towels that were twice the size of normal towels .
GMLE
the breakfast in the morning was delicious and very good . it was the only hotel where i slept very well . the staff was very helpful in late afternoon or late times . the breakfast was adequate , with a decent range of cereals , fruit , and fruits . there is also free use of the coffee in the reception area .
GMLE+RL(coherence, cohesion) the breakfast was plentiful including fresh breads and cooked to order . the location was fantastic . it is in the north of the marina and in a very short distance . the marina has a small swimming pool with sitting area and a small gym . they are very popular and guests have an evening reception which is very nice .
Table 1: Sample generations from our MLE-trained baseline model, GMLE, and our discriminatorguided model GMLE+RL(coherence, cohesion). The red texts highlight a common problem in GMLE - it exhibits a repetition, and an inconsistent opinion as a review. In contrast, our discriminator-guided model is able to generate a more interesting, and sentiment-consistent continuation.

heading: 5 EXPERIMENTS
text: In this section, we show results of training both Dcoherence and Dcohesion, and compare our RL-tuned generatorsGMLE+RL(cohesion),GMLE+RL(coherence), andGMLE+RL(coherence, cohesion) with the baseline model GMLE. We argue that through the use of feedback from our simple discriminators to GMLE, we improve the quality of generated texts. See Table 1 for a sample comparison.

heading: 5.1 DATASET AND EVALUATION METRICS
text: We use the publicly available TripAdvisor’s hotel reviews dataset collected by Wang et al. (2010) and the Yelp review dataset3. We consider only subsets of the two review datasets satisfying the following two conditions: a review must have (1) at least 10 sentences, and (2) each sentence should have more than 5 and less than 30 words. This yields roughly 60,000 TripAdvisor reviews and 220,000 Yelp reviews, split into [0.8, 0.1, 0.1] ratio for train/dev/test. We merge the source and target vocabularies, and limit it to the top 50,000 frequent words, excluding special tokens. For each of these reviews, as in Holtzman et al. (2018), we consider the first five sentences as the source input S to G, and the following five sentences as the target output T from G.
It is widely known that there is no accurate metric to evaluate the generator. Nevertheless, we report scores of standard metrics, such as negative log-likelihood (NLL), perplexity (PPL), BLEU and proportion of unique n-grams within a single generation (intra-unique-n), and across generations (inter-unique-n), as in Gu et al. (2018).

heading: 5.2 IMPLEMENTATION DETAILS
text: G takes individual words as inputs and embeds into a pre-trained 300-dimensional word vectors from GloVe (Pennington et al., 2014). This embedding layer is fixed throughout training. G uses a gated recurrent unit with two layers and a hidden size of 1024 for both bidirectional encoder and attention-based decoder. During optimization using Adam (Kingma & Ba, 2014), we set the learning rate to 2e-4 and clip the gradient’s L2-norm to 1.0. We initially trainGMLE by maximizing the wordlevel likelihood estimation (MLE) from data that consist of positive samples for 60 epochs on the TripAdvisor data and 30 epochs on the Yelp dataset, separately. These are our baseline models against which to empirically prove value of our hierarchical discriminators.
Dcoherence also uses the pre-trained GloVe word vectors4, which are fixed. The source processing network and the target processing network have the same structure, but different parameters. If the encoder type is a CNN, the convolutional layer has filters of sizes 2, 3, 4, and 5, each with 512 filters. Each convolution filter is followed by a tanh activation. Then we max-pool in time over the features and append a fully connected layer into a feature embedding of dimension 512, followed by a batch normalization layer and a tanh activation. If the encoder type is a RNN, we use a 1-layered bi-directional GRU, concatenate final hidden states from both ends, and append the same network layers as used in the CNN-encoded counterpart. We use an Adam optimizer with a learning rate of
3https://www.yelp.com/dataset 4The vector dimension can be different from that of G. The differences were marginal for sizes 50, 100,
and 300. For results shown in this paper, we used the same dimension of size 300.
1e-5. Dcohesion is the same as Dcoherence, except the CNN-encoded Dcohesion has convolutional filters of sizes 3, 4, 5, and 6. We train both discriminators for 50 epochs and choose models with the best R@1 validation scores. Retrieval results are shown in Table 2.
In the tuning stage, we use the negative-critical sequence training as explained in Section 4 up to 5 epochs, with a learning rate of 1e-5. We also continue with supervised learning to G to limit the policy search within a grammatically correct space, similar to Paulus et al. (2017); Wu et al. (2016); Lewis et al. (2017). In practice, sequence-level rewards are only available upon a completed generation, so they are sparse signals for the generator. Typically, sparse end-of-sequence rewards entail a noisy training, yet would want the learning generalize to the testing data. We observed that, for our particular task, most noises were caused by exploration, and the learning generalized to the testing data. Thus, reward shaping was unnecessary, unlike previous works (Li et al., 2017; Yang et al., 2018) that further provided signals for partially generated sequences. For all generations, we used greedy decoding since we did not see a significant difference using beam search. Results are shown in Table 3.

heading: 5.3 DISCUSSION
text: In Table 2, notice that for RNNs outperform CNNs for coherence models, and CNNs outperform RNNs for cohesion models. One explanation is that RNNs are effective in encoding a sequential input yet exhibit drawbacks when encoding into hidden states at both ends of a long input, otherwise well-known as a long-range dependency problem.
To aid understanding of the roles ofDcoherence andDcohesion, we show randomly selected positive and negative samples and corresponding rewards in Table 4.
While reinforcing coherence and cohesion properties in text generation through surrogate models is an important research direction, we consider our results to be preliminary, and our experiment results allude to room for improvement, such as recall scores. This is because our methods to
construct negative samples from an unlabelled dataset are not thorough. For example, a randomly mismatched sentence that follows a given sentence may actually be a valid continuation. In this work, we overlook this problem since our proposed schemes are shown to be effective in modeling coherence and cohesion.

heading: 6 CONCLUSION
text: In this paper, we propose to model coherence and cohesion through simple training mechanisms via models parametrized by neural networks, and quantify coherence and cohesion into negative-critical margin scores. The coherence discriminator Dcoherence provides a macro-level view on structuring a multi-sentence text. It assesses how likely two text chunks form a coherent paragraph, using sentence-level features. On the other hand, the cohesion discriminator Dcohesion provides a microlevel view on local connectivity between sentence pairs. It assesses how cohesive two consecutive sentences are, using word-level features.
The margin scores computed by these discriminators are used as reward signals for training neural language models via policy gradient. Empirical results on two long-form text generation tasks show that our surrogate coherence and cohesion models, trained through simple yet effective methods, help improve over the strong baseline, an attention-based bidirectional MLE-trained sequence-tosequence model in a number of automatic metrics.
Future work will focus on casting the long-form text generation task using the GANs framework. In this framework, the coherence and cohesion discriminators are modified against model-generated texts, and in turn, provide signals to learn neural language models.

Review summaries:
This paper addresses long-text generation, with a specific task of being given a prefix of a review and needing to add the next five sentences coherently.  The paper proposes adding two discriminators, one trained to maximize a cosine similarity between source sentences and target sentences (D_{coherence}) and one trained to maximize a cosine similarity between two consecutive sentences.  On some automatic metrics like BLEU and perplexity, an MLE model with these discriminators performs a little bit better than without.

The paper proposes a method for improving the quality of text generation by optimizing for coherence and cohesion. The authors develop two discriminators--a "coherence discriminator" which takes as input all of the sentence embeddings (i.e. averaged word embeddings) of the document and assigns a score, and a "cohesion discriminator" which takes as input the word embeddings of two consecutive sentences and assigns a score. In the former, the score is the cosine similarity between the encodings of the first and second half of the document. In the latter, the score is the cosine similarity between the encodings of the two sentences. Both discriminators use CNNs to encode the inputs. The discriminators are trained to rank true text over randomly drawn negative samples, which consist of randomly permuted sentence orderings and/or random combinations of first/second half of documents. This discriminators are then used to train a text generation model. The output of the text generation model is scored by various automatic metrics, including NLL, PPL, BLEU, and number of unique ngrams in the outputs. The improvements over a generically-trained generation model are very small.