Venue: NAACL 2016

# Lexical Coherence Graph Modeling Using Word Embeddings

 Mohsen Mesgar

 Michael Strube

Heidelberg Institute for Theoretical Studies gGmbH

Schloss-Wolfsbrunnenweg 35

69118 Heidelberg, Germany

(mohsen.mesgar|michael.strube)@h-its.org

###### Abstract

Coherence is established by semantic connections between sentences of a text which can be modeled by lexical relations. In this paper, we introduce the lexical coherence graph (LCG), a new graph-based model to represent lexical relations among sentences. The frequency of subgraphs (coherence patterns) of this graph captures the connectivity style of sentence nodes in this graph. The coherence of a text is encoded by a vector of these frequencies. We evaluate the LCG model on the readability ranking task. The results of the experiments show that the LCG model obtains higher accuracy than state-of-the-art coherence models. Using larger subgraphs yields higher accuracy, because they capture more structural information. However, larger subgraphs can be sparse. We adapt Kneser-Ney smoothing to smooth subgraphs' frequencies. Smoothing improves performance.

## 1 Introduction

The concept of coherence is based on cohesive semantic relations connecting elements of a text. Cohesive relations are expressed through grammar and the vocabulary of a language. The former is referred to as _grammatical coherence_, the latter as _lexical coherence_Halliday and Hasan (1976). Grammatical coherence encompasses coreference, substitution, ellipsis, etc. Lexical coherence comprises semantic connections among words of a text.

In this paper we measure text coherence by modeling _lexical coherence_. Lexical relations specify cohesive relations over the sentences of a text. These lexical relations can be any kind of semantic relation: repetition, synonymy, hyperonymy, meronymy, etc. These lexical items may or may not have the same reference Halliday and Hasan (1976).

_Why does the little boy wriggle all the time? Girls don't._

In this example the lexical items _boy_ and _girls_ are semantically related. Although they do not refer to the same entity, they still connect these two sentences.

There is coherence between any pair of lexical items that stand to each other in some lexicosemantic relation Halliday and Hasan (1976). For textual purposes it is not required to determine the type of the relation. It is only necessary to recognize semantically related lexical items, and these relations can be learned by cooccurring lexical items.

One can use world knowledge resources to determine semantic relations. This way is expensive in terms of determining the best resource, e.g. WordNet vs. Freebase. WordNet lacks broad coverage in particular with proper names, Freebase is restricted to nominal concepts and entities.

Recent improvements in embedding representations of words let us efficiently compute semantic relations among lexical items in the vector space. These models use a vector of numbers to encode the meaning of words. We use these vectors to check the existence of any kind of semantic relations between two words.

In the following example the sentences are connected because of the semantic relation between _king_ and _queen_ which can be induced by word em

bedding models (Mikolov et al., 2013; Pennington et al., 2014).

_...The king was in his counting-house, counting out his money, The queen was in the parlour, eating bread and honey._

We model lexical coherence between sentences by a lexical coherence graph (LCG). We consider subgraphs of this graph coherence patterns and use their frequency as features representing the connectivity of the graph and, hence, the coherence of a text (Mesgar and Strube, 2015).

An important task for evaluating a coherence model is readability assessment. The goal of this task is to rate texts based on their readability. The more coherent a text, the faster to read and easier to understand it is. Other coherence models (Barzilay and Lapata, 2008; Guinaudeau and Strube, 2013; Mesgar and Strube, 2014) are also evaluated on this task. Pitler and Nenkova (2008) use the entity grid (Barzilay and Lapata, 2008) to capture the coherence of a text for readability assessment. Mesgar and Strube (2015) extend the entity graph (Guinaudeau and Strube, 2013) as coherence model to measure the readability of texts. They encode coherence as a vector of frequencies of subgraphs of the graph representation of a text. We build upon their method and represent the connectivity of sentences in our LCG model by a vector of frequencies of subgraphs.

Although using the frequency of subgraphs of the lexical coherence graph encodes coherence features well, the subgraph frequency method, in general, is suffering from a sparsity problem when the subgraphs get larger. Large subgraphs capture more structural information, but they occur only rarely. We resolve this sparsity issue by adapting Kneser-Ney smoothing (Heafield et al., 2013) to smooth subgraph counts (Section 3). We estimate the probability of unseen subgraphs, i.e. coherence patterns. This prediction lets us measure the coherence of a text even when its corresponding graph representation contains a subgraph which does not occur in the training data. If the unseen coherence pattern is similar to seen ones, smoothing gives it closer probability to seen coherence patterns in comparison to dissimilar unseen ones. This is due to the base probability factor in Kneser-Ney smoothing.

We evaluate our LCG model on the two readability datasets provided by Pitler and Nenkova (2008) and De Clercq et al. (2014), respectively (Section 4). The results (Section 5) indicate that the LCG model outperforms state-of-the-art systems. By applying Kneser-Ney smoothing we solve the sparsity problem. Smoothing allows us to exploit the high informativity of large subgraphs which leads to new state-of-the-art results in readability assessment.

## 2 Related Work

The entity grid model (Barzilay and Lapata, 2008) is based on entity transitions over sentences. It uses a two dimensional matrix to represent transitions of entities among adjacent sentences. The entity grid is applied to readability assessment by Pitler and Nenkova (2008). The entity graph (Guinaudeau and Strube, 2013) is a graph-based, mainly unsupervised interpretation of the entity grid. This model represents the distribution of entities over sentences in a text with a bipartite graph. Connections between sentences are obtained by information on entites shared by sentences. Guinaudeau and Strube (2013) perform a one-mode projection on sentence nodes and use the average out-degree of the one-mode projection graph to quantify the coherence of the given text. Mesgar and Strube (2015) represent the connectivity of the one-mode projection graph by a vector whose elements are the frequencies of subgraphs in projection graphs. This encoding works much better than the entity graph for the readability task on the \(P\&N\) dataset and even outperforms Pitler and Nenkova (2008) by a large margin. Zhang et al. (2015) state that the entity graph model is limited, because it only captures mentions which refer to the same entity (the entity graph uses a very restricted version of coreference resolution to determine entities). Zhang et al. (2015) use world knowledge _YAGO_(Hoffart et al., 2013), _WikiPedia_(Denoyer and Gallinari, 2006) and _FreeBase_(Bolacker et al., 2008) to capture the semantic relatedness between entities even if they do not refer to the same entity. Main issues with using world knowledge are: the choice knowledge sources, selection of knowledge from the source, coverage, and language-dependence.

Word embedding approaches like _word2vec_ and 

_GloVe_(Mikolov et al., 2013; Pennington et al., 2014) show that the semantic connection between words can be captured by word vectors which are obtained by applying a neural network. The ability to train on very large data sets allows the model to learn complex relationships between words.

## 3 Method

We introduce a new graph representation of semantic connections over lexical items in texts. Afterwards we compute the frequency of all subgraphs, i.e. coherence patterns. The intuition is that subgraphs capture how sentence nodes are connected and, respectively, encode text coherence.

### Graph Model

We model semantic relations between sentences by a graph \(G=<\!V,E{>}\) where \(V\) is the set of sentence nodes and \(E\) is the set of edges between sentence nodes. Two nodes of \(G\) are adjacent if there is a semantic connection between the corresponding sentences. Two sentences are semantically connected if there is at least one strong semantic relation between the words of these sentences. We model semantic relations between words by their corresponding word embeddings [5]. Given word vectors \(v_{a}\) for word \(a\) of sentence \(A\) and \(v_{b}\) for word \(b\) of sentence \(B\), the cosine similarity value, \(cos(v_{a},v_{b})\), between the two word vectors is a measure of semantic connectivity of the two words. The range of \(cos(v_{a},v_{b})\) is between \([-1,+1]\). One interpretation of cosine is the normalized correlation coefficient, which states how well the two words are semantically correlated [7]. The absolute value of cosine, \(|cos(v_{a},v_{b})|\), encodes how strongly the two words are connected.

The connection between sentences is obtained from connections between their words (Figure 1). Assume sentence \(A\) precedes sentence \(B\), each word \(b\) of sentence \(B\) is connected with word \(a^{*}\) of \(A\), where

\[a^{*}=\operatornamewithlimits{argmax}_{a\in A}cos(b,a)\]

Then from all connections between the words of sentences \(A\) and \(B\), the connection with the maximum weight among the words of \(B\) is selected to connect these two sentences (Figure 2).

The output of this phase is a graph whose edge weights model the strength of connections between sentences. The edges in this graph are directed to model the order of sentences.

Word embeddings relate each word in sentence \(A\) with each word in sentence \(B\). Since the resulting graph is very dense, we filter out edges whose weights are below a threshold1.

Footnote 1: We set this threshold to \(0.9\) to connect only sentences with high confidence.

### Coherence Features

Mesgar and Strube (2015) propose that the connection style of an entity graph can be captured by the frequency of all \(k\)-\(node\) subgraphs in this graph. Larger2 subgraphs3 can capture more information about the structure of graphs and are more informative coherence patterns than smaller ones. We experiment with \(k\in\{3,4,5,6\}\). Text coherence is repre

Figure 1: Sentence A with three words \(\{w_{1},w_{2},w_{3}\}\) and sentence B with two words \(\{w_{4},w_{5}\}\). \(w_{4}\) is highly related to \(w_{2}\) and \(w_{5}\) is highly related to \(w_{3}\).

Figure 2: The word relation with the maximum weight (a) represents the connections between sentences (b).

 

sented by a vector whose elements are the frequency of subgraphs (coherence patterns) with \(k\)-\(node\).

### Smoothing

Although increasing the size \(k\) of subgraphs captures more structural information about the connections of sentence nodes, a main risk with large subgraphs is sparsity. Given a sentence graph, many large subgraph types do not occur in this graph. Small subgraph types occur frequently in most sentence graphs in the dataset, but these subgraphs do not capture enough information about the connectivity style of the graphs.

Inspired by Kneser-Ney smoothing in language models [1], each feature vector of a sentence graph can be smoothed. Smoothing deals with the problem of zero counts in the feature vector. It also lets the model having feature values for unseen subgraphs (like OOV in language modeling) which may be seen in the testing phase.

Kneser-Ney smoothing uses a discount factor to discount the raw count of each event (subgraph) and distributes the total discount to all event (subgraph) probabilities by means of a base probability.

The estimated frequency of subgraph \(sg\) in a given sentence graph is computed as follows:

\[KN(sg)=\frac{\max\{count(sg)-\alpha,0\}}{Z}+\frac{M\cdot\alpha}{Z}P_{b}(sg),\]

where \(\alpha\) is the discount factor and \(M\) is the number of times that discount factor is applied. \(Z\) is a normalization factor to ensure that the distribution sums to one and is obtained as follows:

\[Z=\sum_{sg\in A}count(sg),\]

where \(A\) is the set of all subgraphs with \(k\)-\(nodes\) and function \(count(\cdot)\) computes the number of instances of subgraph \(sg\) in the given sentence graph.

\(P_{b}(sg)\) in Kneser-Ney smoothing is the base probability of subgraph \(sg\) among all \(k\)-\(node\) subgraphs (\(A\)). The base probability can be computed based on hierarchical (parent-child) relations in subgraphs. \(k\)-\(node\) subgraph \(sg_{i}\) is a parent of \((k\)+1\()\)-\(node\) subgraph \(sg_{j}\), if \(sg_{i}\) is a subgraph of \(sg_{j}\). Figure 3 shows the parent-child relation between subgraphs via a weighted tree. The root of this tree is a null graph4. The weight of a parent-child relation connecting the parent subgraph \(sg_{i}\) and child subgraph \(sg_{j}\) is shown by \(w_{ij}\) and computed as follows:

Footnote 4: A null graph is a graph with no nodes.

\[w_{ij}=\frac{count(sg_{i},sg_{j})}{\sum_{sq_{l}\in A}count(sg_{i},sg_{l})},\]

where \(A\) is all subgraphs with \(k\)-\(node\) and \(k\) equals the number of nodes of \(sg_{j}\). Interpretation of weight \(w_{ij}\) is the normalized count of \(sg_{i}\) in \(sg_{j}\) with respect to all outgoing edges from \(sg_{i}\).

The base probability of each subgraph \(sg_{j}\) is the inner product of the Kneser-Ney probabilities of \(sg_{j}\)'s parents by the weights of the corresponding relations:

\[P_{b}(sg_{j})=P\cdot W, \tag{1}\]

where \(P\) is the vector of probabilities of all parents of \(sg_{j}\) and \(W\) is the vector of all corresponding edge weights connecting the parents of \(sg_{j}\) to \(sg_{j}\).

Since the root node of this tree is the null subgraph, and it is a subgraph of all possible sentence graphs, its base probability is one. Because the edge weights are in the range \([0,1]\) the sum of the probabilities of all subgraphs with \(k\)-\(node\) is always equal to one.

**Proof.** Assume \(I\) and \(J\) are the set of all \(k\)-\(node\) and \((k\)+1\()\)-\(node\) subgraphs. We also assume that \(I\) has \(n\) subgraphs and \(\sum_{i=1}^{n}p(sg_{i})=1\). Considering these assumptions we prove that

\[\sum_{j=1}^{m}p(sg_{j})=1,\]

where \(m\) is the number of subgraphs in \(J\).

We start from the left and compute the value of

\[\sum_{j=1}^{m}p(sg_{j}).\]

Based on the definition of base probability, the value of \(p(sg_{j})\) is computed based on its parents in \(I\),

\[p(sg_{j})=\sum_{i=1}^{n}w_{ij}p(sg_{i}),\]

where \(w_{ij}\) is the weight of the parent-child relation between \(sg_{i}\) and \(sg_{j}\). Now we have: 

\[\sum_{j=1}^{m}p(sg_{j})=\sum_{j=1}^{m}\sum_{i=1}^{n}w_{ij}p(sg_{i}).\]

If we exchange the place of the sums and re-write the equation, we have:

\[\sum_{j=1}^{m}p(sg_{j})=\sum_{i=1}^{n}\sum_{j=1}^{m}w_{ij}p(sg_{i}).\]

In this equation \(p(sg_{i})\) is independent of \(j\) (index of the inner sum), so it can be moved out of the inner sum:

\[\sum_{j=1}^{m}p(sg_{j})=\sum_{i=1}^{n}p(sg_{i})\sum_{j=1}^{m}w_{ij}\]

The inner sum equals \(1\).

\[\sum_{j=1}^{m}p(sg_{j})=\sum_{i=1}^{n}p(sg_{i}).\]

Based on our assumption the right side of the equation is \(1\) and

\[\sum_{j=1}^{m}p(sg_{j})=1.\]

So we proved that the sum of the base probability of all \(k\text{-}node\) subgraphs is \(1\). 

This way, Kneser-Ney smoothing distributes the total discount value by considering the weights of parent-child relations among the subgraphs. The result of applying smoothing is an estimation of the frequency of each subgraph in the sentence graph.

## 4 Experiments

### Evaluation Task

We evaluate our coherence model on the task of ranking texts by their readability. The intuition is that more coherent texts are easier to read.

Datasets.We run our experiments on two datasets annotated with readability information provided by human annotators: _P&N_(Pitler and Nenkova, 2008) and _De Clercq_(De Clercq et al., 2014).

The dataset _P&N_ contains 27 articles randomly selected from the Wall Street Journal corpus5. The average number of sentences is about \(10\) words. Every article is associated with a human score between \([0.0,5.0]\) indicating the readability score of that article. We create pairs of documents, if the difference between their readability scores is greater than \(0.5\). If the first document in a pair has the higher score, we label this pair with \(+1\), otherwise with \(-1\). The resulting number of text pairs in this dataset is \(209\).

Footnote 5: Pitler and Nenkova (2008)’s dataset contains 30 articles. They remove one. We assume this is wsj–0382 which does not exist in the Penn Treebank. We furthermore remove wsj–2090 which does not exist in the final release of the Penn Discourse Treebank. We also remove wsj-1398 which is a poem and, hence, not very informative for readability assessment.

The dataset _De Clercq_ consists of \(105\) articles from different genres: administrative (17 articles), journalistic (43 articles), manuals (14 articles) and miscellaneous (31 articles). The average number of sentences is about \(12\). This dataset was annotated by De Clercq et al. (2014) by asking human judges to compare two texts based on their readability. They use five labels:

Figure 3:parent child relation.

 



### Experimental Settings

Word Embeddings and Classification.In order to reduce the effect of very frequent words, stop words are filtered by using the SMART English stop word list Salton (1971). We use a pretrained model of GloVe for word embeddings. This model is trained on Common Crawl with 840B tokens, 2.2M vocabulary. We represent each word by a vector with length 300 Pennington et al. (2014). For handling out-of-vocabulary words, we assign a random vector to each word and memorize it for its next occurrence Kusner et al. (2015). The classification task is done by the SVM implementation in WEKA (SMO) with the linear kernel function. All settings are set to the default values. The evaluation is computed by 10-fold cross validation.

Graph Processing and Smoothing.In order to compare the performance of LCG with the entity graph model, we follow Mesgar and Strube (2015) and use the gSpan method Yan and Han (2002) to compute all common subgraphs on each dataset and their frequencies. Note that gSpan does not count all possible \(k\)-\(node\) subgraphs, whereas for applying Kneser-Ney smoothing it is necessary to count all possible \(k\)-\(node\) subgraphs, because the probability should be distributed among all possible subgraphs. This also helps to estimate the probability of unseen patterns. We use a random sampling method Shervashidze et al. (2009) to obtain the frequency of subgraphs in a sentence graph. In this regard, we take \(10,000\) samples of the given sentence graph by randomly selecting \(k\) nodes of the graph to count the occurrence of \(k\)-\(node\) subgraphs in this graph. We compute the base probability for at most \(k=6\). We find the best value for \(d\) in a greedy manner. First, we initialize \(d\) with \(0.001\). In each iteration we compute the performance. Then we multiply the discount factor by \(10\). We iterate as long as the discount factor is less than \(1000\). We report the best performance.

## 5 Results

In order to compare our method with related work, we run our model on the _P&N_ dataset. Table 2 reports the accuracy of _LCG_ with different values for \(k\) in \(k\)-\(node\) subgraphs. This corresponds to coherence patterns spanning different numbers of sentences.

We start in Table 2 with a majority class baseline (_ZeroR_). _EGFR_ is our reimplementation of Pitler and Nenkova (2008) which we use as non-trivial baseline. The column _EGraph_ is the entity graph model of Mesgar and Strube (2015). In _EGRraph+PRN_ we extend this model by a pronoun resolution system, so that entities mentioned by pronouns also enter the graph. We apply the Stanford coreference resolution system Lee et al. (2013). Using the full coreference resolution system, however, decreases performance, hence we only use resolved pronouns. The enriched model with resolved pronouns works slightly better for _3-node_ and _4-node_ subgraphs,

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt}} \hline Genre & No. of articles & No. of text pairs \\ \hline Administrative & 17 & 272 \\ Journalistic & 43 & 1806 \\ Manuals & 14 & 182 \\ Miscellaneous & 31 & 931 \\ \hline \end{tabular}
\end{table}
Table 1: Properties of the different genres in the _De Clercq_ dataset.



and slightly worse for _5-node_ subgraphs than the _EGraph_. The lexical coherence graph model, _LCG_, performs slightly worse than _EGraph_ on _3-node_ subgraphs. This could be because the graphs in _LCG_ have more edges than the graphs in _EGraph_. When graphs are denser _3-node_ subgraphs occur in every graph, hence their frequency is less discriminative. As shown in Table 2 larger subgraphs (_4-node_ and _5-node_) capture more information and improve upon _EGraph_ and for _5-node_ subgraphs even upon _EGraph+PRN_. _LCG_ significantly (\(p\_value=0.01\)) works better than _EGraph+PRN_ and _EGraph_ using 5-node subgraphs. The difference between _LCG_ and _EGraph_+PRN_ and _EGraph_ using 4-node subgraphs is not significant.

Table 3 shows the performance of different models on the _De Clercq_ dataset.

Again, we use a majority baseline (_ZeroR_) to put our results in context. While the performance of both methods almost does not beat the baseline for _3-node_ subgraphs, _4-node_-subgraphs work already better, and _5-node_ subgraphs yield reasonable performance on this dataset. Although _EGraph+PRN_ and _LCG_ reach almost the same performance for _4-node_, the difference between them is statistically significant (\(p\_value=0.01\)). With _5-node_ subgraphs, _LCG_ outperforms _EGraph+PRN_ subgraphs by a large margin and gets a very reasonable performance on this dataset.

The general performance on the _De Clercq_ dataset is lower than the performance on the the _P&N_ dataset. This can have two reasons: first, the ranking task on the _De Clercq_ dataset is three-label classification which is more difficult than the binary classification task on the _P&N_ dataset. Second, texts in the _De Clercq_ dataset are from different genres and coherence patterns may vary across genres. Hence, we take a closer look on the performance on the different genres.

Table 4 shows the performance for _EGraph+PRN_ and _LCG_ using _5-node_ subgraphs on the different genres in the _De Clercq_ dataset. The performance of _LCG_ is higher than _EGFRh+PRN_ on all genres. Unlike _EGraph+PRN_, _LCG_ gets the best performance on journalistic articles. The lowest performance of both models is obtained on manuals. On administrative articles, performance of _LCG_ is slightly better than _EGRraph+PRN_. On miscellaneous articles _LCG_ performs better than _EGraph+PRN_.

While large subgraphs are very informative for coherence modeling, extracting large subgraphs (\(k>4\)) in relatively small datasets leads to a data sparsity problem, as there are very many possible subgraphs to be represented in a high dimensional vector space. Hence, many possible subgraphs have low or even zero counts. The problem for such a vector is that each graph is only similar to itself and not to any other graph. Hence, we observe a drop in performance when the model deals with large subgraphs (_6-node_ subgraphs, _LCG1_ for _P&N_ in Table 5). We solve this problem by smoothing.

In order to apply Kneser-Ney smoothing we use a sampling method to create all possible (connected and disconnected) \(k\)-\(node\) subgraphs (for _LCG1_ and _LCG1*_ we use connected and disconnected subgraphs, for _LCG_ only connected ones).

Table 5 shows the performance of _LCG1_ when it is applied to ever larger subgraphs. As can be seen in Table 5, the performance on the _P&N_ dataset suddenly drops for _6-node_ subgraphs. This is could be caused by the sparsity problem.

When we apply Kneser-Ney smoothing as described in Section 3 the results for all tested values of \(k\) are superior for _LCG1*_ when compared to _LCG1_ (Table 5).

Kneser-Ney smoothing improves the performance of the system even with _3-node_ subgraphs by a large margin. Smoothing reduces the power of fre

\begin{table}
\begin{tabular}{l c c} \hline _5-node_ & EGraph+PRN & LCG \\ \hline Administrative & 69.49\% & 71.69\% \\ Journalistic & 65.01\% & 82.12\% \\ Manuals & 54.95\% & 61.54\% \\ Misc. & 70.68\% & 76.69\% \\ \hline \end{tabular}
\end{table}
Table 4: Accuracy of _EGraph+PRN_ and _LCG_ on different genres in the _De Clercq_ dataset.

\begin{table}
\begin{tabular}{c c c} \hline System & Accuracy \\ \hline ZeroR & \multicolumn{2}{c}{42.312\%} \\ \hline \(k\)-\(node\) & EGraph+PRN & LCG \\ \hline
3-node & 42.31\% & 42.31\% \\
4-node & 48.07\% & 49.12\%** \\
5-node & 65.77\% & 76.27\%** \\ \hline \end{tabular}
\end{table}
Table 3: De Clercq dataset.

 

quency and makes the frequency distribution of subgraphs more even. Smoothing reduces the values through all subgraphs by considering parent-child relations between subgraphs to relate similar subgraphs. That is the advantage of the Kneser-Ney method in comparison to the other smoothing methods like Laplace-Smoothing.

For the _P&N_ dataset we achieve the best results to date. Pitler and Nenkova (2008) reported 83.25% accuracy, Mesgar and Strube (2015) 89.95%. When smoothing _5-node_ subgraphs we are able to report 98.08%. This, however, indicates that this dataset may not be the best one to report performance on. Hence, we now check whether smoothing also improves the performance on the more difficult _De Clercq_ dataset.

On this dataset, we basically observe the same trends. Both settings result in better performance than _LCG_ (see Table 3).

Note that none of the parameters in this work is tuned on the datasets. One may get better performance by tuning the parameters. The results confirm the intuition that the lexical coherence graph _LCG_ captures coherence and models lexical coherence appropriately.

Applying smoothing on graphs of _EGraph+PRN_ model increases the performance of this model. But this improvement is not as high as the improvement on the _LCG_ graph.

Coherence Patterns.In this part we check the Pearson correlation coefficient between _LCG1_ and human judgements of a few frequent subgraphs on the _P&N_ dataset. In order to be consistent with Messgar and Strube (2015), we use the exhaustive value of subgraph frequencies, i.e. _LCG1_ for our work.

For the _3-node_ subgraphs only one subgraph (Figure 4) in the _LCG1_ representation is significantly (and positively) correlated (\(p\)-value\(<0.05\)) with human scores. For the _4-node_ subgraphs, we find six subgraphs which are significantly correlated with readability. Only one is positively correlated, while four are negatively correlated. Interestingly, both positively correlated _3-node_ and _4-node_ subgraphs have been determined as positively and significantly correlated by Mesgar and Strube (2015) as well. Both also capture a similar coherence pattern, indicating that our method is linguistically sound.

## 6 Conclusions and Future Work

In this paper we propose a new graph based coherence model, the lexical coherence graph, LCG. We view coherence as semantic connectedness between words which we model by word embeddings. We take only the strongest connection between sentences to create a graph with connected sentences. Then we extract large subgraphs capturing coherence patterns, which show similarity to patterns described in text linguistics (Danes, 1974).

Figure 4: Pearson correlation between _3-node_ and _4-node_ subgraphs and readability scores in the _P&N_ dataset.

\begin{table}
\begin{tabular}{l l c c c} \hline  & \multicolumn{2}{c}{_P\&N_} & \multicolumn{2}{c}{_De Clercq_} \\ \hline \(k\)-\(node\) & LCG1 & LCG1* & LCG1 & LCG1* \\ \hline
3-node & 84.52\% & 89.00\% & 42.31\% & 49.60\% \\
4-node & 95.69\% & 96.17\% & 65.10\% & 66.23\% \\
5-node & 97.61\% & 98.08\% & 79.33\% & 79.85\% \\
6-node & 93.26\% & 95.69\% & 76.67\% & 78.03\% \\ \hline \end{tabular}
\end{table}
Table 5: Applying smoothing method yields to higher accuracy for larger subgraphs.

 

While the entity grid works only on sequences of up to three adjacent sentences, we are able to model relationships of up to six non-adjacent sentences. We solve the sparsity problem of large subgraphs by adapting Kneser-Ney smoothing to graphs. Smoothing prevents LCG from losing performance with large subgraphs and leads to superior performance on the Pitler and Nenkova (2008) dataset and to a first reasonable state-of-the-art on the De Clercq et al. (2014) dataset.

In future work we want to apply LCG to essay scoring as well. Also, we see that our adaption of Kneser-Ney smoothing to graphs may be useful for research in subgraph mining in general.

## Acknowledgments

This work has been funded by the Klaus Tschira Foundation, Heidelberg, Germany. The first author has been supported by a HITS Ph.D. scholarship. We would like to thank Orphee De Clercq who provided the _De Clercq_ dataset. We also appreciate Andreas Spitz' comments on graph processing.

## References

* Barzilay and Lapata (2008) Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. _Computational Linguistics_, 34(1):1-34.
* Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveren Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created grand database for structuring human knowledge. In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, Vancouver, B.C., Canada, 10-12 June 2008, pages 1247-1250.
* Danes (1974) Frantisek Danes. 1974. Functional sentence perspective and the organization of the text. In F. Danes, editor, _Papers on Functional Sentence Perspective_, pages 106-128. Prague: Academia.
* De Clercq et al. (2014) Orphee De Clercq, Veronique Hoste, Bart Desmet, Philip Van Oosten, Martine De Cock, and Lieve Macken. 2014. Using the crowd for readability prediction. _Natural Language Engineering_, 20(3):293-325.
* Denoyer and Gallinari (2006) Ludovic Denoyer and Patrick Gallinari. 2006. The Wikipedia XML corpus. _ACM SIGIR Forum_, 40(1):64-69.
* Guinaudeau and Strube (2013) Camille Guinaudeau and Michael Strube. 2013. Graph-based local coherence modeling. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, Sofia, Bulgaria, 4-9 August 2013, pages 93-103.
* Halliday and Hasan (1976) M. A. K. Halliday and Ruqaiya Hasan. 1976. _Cohesion in English_. London, U.K.: Longman.
* Heafield et al. (2013) Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, Sofia, Bulgaria, 4-9 August 2013, pages 690-696.
* Hoffarf et al. (2013) Johannes Hoffarf, Fabian M. Suchanek, Klaus Berberich, and Gerhard Weikum. 2013. YAGO2: A spatially and temporally enhanced knowledge based from Wikipedia. _Artificial Intelligence_, 194:28-61.
* Kusner et al. (2015) Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From word embeddings to document distances. In _Proceedings of the 32nd International Conference on Machine Learning_, Lille, France, 6-11 July 2015, pages 918-927.
* Lee et al. (2013) Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. _Computational Linguistics_, 39(4):885-916.
* Manning and Schutze (1999) Christopher D. Manning and Hinrich Schutze. 1999. _Foundations of Statistical Natural Language Processing_. MIT Press, Cambridge, Mass.
* Mesgar and Strube (2014) Mohsen Mesgar and Michael Strube. 2014. Normalized entity graph for computing local coherence. In _Proceedings of TextGraphs-9: Graph-based Methods for Natural Language Processing, Workshop at EMNLP 2014, Doha, Qatar, 29 October 2014, pages 1-5.
* Mesgar and Strube (2015) Mohsen Mesgar and Michael Strube. 2015. Graph-based coherence modeling for assessing readability. In _Proceedings of STARSEM 2015: The Fourth Joint Conference on Lexical and Computational Semantics_, Denver, Col., 4-5 June 2015, pages 309-318.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Proceedings of Advances in Neural Information Processing Systems 26_. Lake Tahoe, New., 5-8 December 2013, pages 3111-3119.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25-29 October 2014, pages 1532-1543.
* Pitler and Nenkova (2008) Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, 25-27 October 2008, pages 186-195 

* [10]G. A. Bjorstad, M. J. K. McFarland, and R. D. S. Murray (2007) The role of the social network in social networks. In Proceedings of the 3rd International Conference on Social Information Processing, Florence, Italy, pp. 45-68. Cited by: SS1.
* [11]M. C. Miller, P. H. Lee, and T. E. Smith (2009) An empirical study of social networks. In Proceedings of the 2nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1-10. Cited by: SS1.
* [12]C. N. Newman and G. E. Hinton (2002) Visualizing data for large graphs. Nature & Machine Intelligence3 (1), pp. 1-12. External Links: ISSN 0037-469X, Document, Link Cited by: SS1.
* [13]A. V. Mnih, L. Kaiser, and I. Sutskever (2015) On the impact of neural networks on graph learning. arXiv preprint arXiv:1502.01665. Cited by: SS1.
* [14]S. O. N. Srivastava, Y. Bengio, and J. Salomon (2014) Learning to be done with deep convolutional neural networks. In Advances in Neural Information Processing Systems, NIPS'14, New Orleans, Louisiana, USA, pp. 1-10. External Links: ISBN 978-1-4693-3861-1, Link, Document Cited by: SS1.
* 5 June 2015, pp. 1087-1096. Cited by: SS1.