# A Discourse Coherence Analysis Method Combining Sentence Embedding and Dimension Grid

Authors: Lanlan Jiang; Shengjun Yuan; Jun Li
Journal: Complexity
 (2021)

---

## Abstract

Discourse coherence is strongly associated with text quality, making it important to natural language generation and understanding. However, existing coherence models focus on measuring individual aspects of coherence, such as lexical overlap, entity centralization, rhetorical structure, etc., lacking measurement of the semantics of text. In this paper, we propose a discourse coherence analysis method combining sentence embedding and the dimension grid, we obtain sentence-level vector representation by deep learning, and we introduce a coherence model that captures the fine-grained semantic transitions in text. Our work is based on the hypothesis that each dimension in the embedding vector is exactly assigned a stated certainty and specific semantic. We take every dimension as an equal grid and compute its transition probabilities. The document feature vector is also enriched to model the coherence. Finally, the experimental results demonstrate that our method achieves excellent performance on two coherence-related tasks.


---

## Body

## 1. Introduction

A well-written text is indeed the rigorous flow of logic. In a well-written text, sentences are structured to convey the author’s purpose, and it is ensured that each new piece is interpretable given the preceding context, which is so-called “coherence.” Modelling text coherence has been an important issue along with multidocument summarization (MDS) and retrieval-based question answering (QA), and currently, it has intersected with essay scoring [1, 2] and text autogeneration [3, 4]. Given a text composed of sentences, we aim to evaluate its rationality and determine the coherence quality.To our knowledge, previous work on modelling coherence quality has mostly used two approaches: one is dimensionality reduction techniques such as latent semantic analysis (LSA), and the other is entity grid and its extension models based on Markov theory. The essence of LSA is singular value decomposition (SVD). After SVD, the document-term matrix is transformed into three matrices, which are the document-topic relation matrix, the topic-keyword relation matrix, and the keyword-term relation matrix. Once we perform SVD, the homonym and text classification could be completed at the same time. However, some number values, especially negatives, in the generated matrix lack a human interpretation, and the polysemy and the word order cannot be handled properly. Therefore, the entity grid model [5], which is well interpretable and has a strong theoretical base, has been the method that researchers have gradually adopted. The entity grid is a statistical model based on centralization theory [6], and it models the text by focusing on the transition of syntactic roles of important entities and computing the text probability by repeating the transition process.While the entity grid has been popular for its simplicity and practicality, there are still some problems that could not be properly handled: (1) It lacks fuzziness such that the entity grid is weak when processing similar sentences. For example, assuming sentenceB following sentence A is coherent with sentence A, the local coherence of A⟶B would be judged as coherent. However, if we now substitute B with a similar sentence C in which the entities are similar to B but not the same, the local coherence of A⟶C would not be judged as coherent according to its rigid entity transition computing. (2) It only considers the coherence of the syntactic level and not the semantic level. The entity grid is based on centralization theory, taking only the entity word into account; that is, the sentence is represented only by the syntactic roles of entity words. This results in the bias that if sentences have the same entities and the entities have the same syntactic roles, their coherence should be the same. Consider two different cases, as shown in Figure 1.Figure 1
Illustration of the inconsistency of the discourse coherence expressed by the same entities. Intuitively, Case 1 is more coherent than Case 2 on the semantic level based on sentiment. However, the coherence of both Case 1 and Case 2 would be the same according to the entity grid, leading to the fact that illogical sentences could be judged as coherent. An important reason is that the entity grid does not consider other factors except for entities.Currently, deep learning has been widely applied to text research and has achieved some great performance. Text units (e.g., words, sentences, and paragraphs) are always transformed into a low dimensional vector and then used for subsequent computing. In this paper, we represent the text with distributed representations generated by deep learning tools (sentence2vec, i.e., “sentence to vector”). Then, inspired by the entity grid, these distributed representations are utilized to build the model. There have been few substantial improvements to the entity model framework. This paper presents an extension of the original method that changes the theory on which it is based.


## 2. Related Works

Since Barzilay et al. [5, 7] proposed the entity grid, many extensions to the method have arisen. Nie et al. [8] propose a cluster-adjacency based method to order sentences for multidocument summarization tasks. They map the summary sentences to a theme in source documents, and the adjacency of pairs of sentences is learned from the source documents based on the adjacency of the clusters to which they belong. Then, the ordering of the summary sentences can be derived using the first sentence determined. Elsner et al. [9, 10] change the selection method of an entity word by considering the effect of a modifier word and including all entities in the grid to build the model. Burstein et al. [2] apply the entity grid model to student essays, showing that, by combining entity-based features with features related to grammatical errors and word usage, one can greatly improve the performance of automated coherence prediction for student essays for different populations. Zhang et al. [11] propose an event-enriched model to alleviate the semantic deficiency problem in the IR-style text processing and apply it to sentence ordering for multidocument news summarization. An ordering algorithm is built on event and entity coherence, both locally and globally. Lin et al. [12] compute the coherent quality by analysing the discourse relation transitions. Yannakoudakis and Briscoe [13] analyse several methods for assessing coherence under the automated assessment (AA) framework by measuring the effect on performance when combined with an AA system that achieves competitive results. Sukumar and Gayathri [14] use WordNet synsets to determine the semantic relationships between the sentences in a summary and build an entailment model that infers the logical relationships. The graph method is used for ranking the sentences, where nodes represent the sentences and edges represent the preference value of one sentence over another sentence. Lin et al. [15] propose a hierarchical recurrent neural network language model for document modelling. After establishing an RNN to capture the coherence between the sentences in a document, their model integrates it as the sentence history information in the word level RNN to predict the word sequence with cross-sentence contextual information. Zhang et al. [16] use the graph-based model and external knowledge to capture semantic relatedness, and then they incorporate this semantic relatedness as additional features in the entity-based model. Chen et al. [17] collect a large corpus of academic texts and derive a data driven method to analyse the coherence problem. Ermakova [18] use the similarity between adjacent sentences to measure text coherence, and their work shows that the methods based on word similarities have very high agreement with human judgement. Li and Jurafsky [19] use both discriminative and generative models to distinguish coherent from incoherent discourse and to produce coherent text. Logeswaran et al. [20] propose an end-to-end unsupervised deep learning approach based on the set-to-sequence framework to model the structure of coherent texts. Their work shows that proper representations of text can order sentences effectively. Hu and Wang [21] propose a proximal policy optimization based on reinforcement learning, and the method could be used for semantic and discourse coherence recognition.The contributions of this paper can be summarized as follows: (1) We regard the dimension as the grid unit, reducing the granularity from a large entity size to a fine-grained single dimension. In addition, we use all the dimensions for the computations, considering all the information a sentence provides and promote the coherence assessment up to the semantic level from the syntactic level. (2) We compute the transition both within one dimension and across different dimensions to consider more relatable information. In addition, we compute the grid transition probability by clustering the dimension transition pairs and using the cluster occupation percentage to which the dimension transition pair belongs to represent the probability, making the model represent fuzzy semantics. Furthermore, we use the document probability based on different transition conditions as the feature vector to rank models, enriching the feature space.


## 3. Methodology

In this section, we introduce the method of discourse coherence analysis combing sentence embedding and the dimension grid. Sentence embedding includes entity grid construction and word2vec/sentence2vec representation. Dimension grid contains dimension grid model construction and document feature vectors extraction.


### 3.1. Entity Grid Construction

Barzilay et al. [5, 7] propose the entity grid method to capture the local coherence of text. In the method, a document is represented as a matrix with rows for sentences and columns for entities. It is based on centralization theory [6], which mainly states that text could be represented by some important nouns; if two adjacent sentences contain the same nouns, they are more likely coherent.Entryrij in the representation matrix describes the syntactic role of entity j in sentence i. For rij, there are four markers of the syntactic role to be assigned to the entity: S for subject, O for object, X for other role, and − for an entity that does not appear. Then, the probability of the syntactic role transition (e.g., S⟶O or O⟶O) is computed across adjacent sentences. Given a specific entity, its transition probability can be estimated by calculating the proportions of different entity transition types within a text. These probability values are then used as features to build a coherence model. Barzilay also uses feature selection (i.e., syntax, coreference, and salient) to tag an entity in a text. After these processing steps, they use the document representations as vectors to input into SVM-rank to build a rank model. Figure 2 shows a summary augmented with syntactic annotations for grid computation. In addition, Figure 3 illustrates a fragment of an entity grid constructed for the text in Figure 2.Figure 2
Summary augmented with syntactic annotations for grid computation.Figure 3
A fragment of an entity grid. The grid cells correspond to grammatical roles: subjects (S), objects (O), or neither (X).


### 3.2. Word2vec and Sentence2vec Representation

Word2vec is a natural language processing (NLP) tool presented by Tomas Mikolov et al. [22, 23] group of Google in 2013 that can could vectorize words by using neural network and Huffman trees. We also call this vectorization process word embedding, and the result is a distributed representation, distinguishing it from the traditional one-hot representation in which a word is described using only 0 or 1 in serial. Word2vec has two training methods: the CBOW and skip-gram. In the CBOW, context words are used to predict one specific word, and in the skip-gram, a specific word is utilized to predict the context words. Once the distributed representation is obtained after training using a large enough corpus, we could conveniently measure the relationships between words and obtain results such as “king” − “queen” ≈ “man” − “woman.”Following the success of word2vec, Mikolov’s group proposed vectorizing sentences and then proposed sentence2vec. Sentence2vec is an extension of word2vec to some extent and promotes the embedding from the word level to the sentence level. Different from word2vec, they added the paragraph ID to identify every sentence. The paragraph ID and the word vectors are then concatenated to train and predict the sentence vector. In addition, the CBOW and skip-gram are also in sentence2vec.Practical applications have verified the reliability of these vectors. We propose the model on the premise that once the vector length is set, each dimension is exactly assigned with a stated certainty and specific semantics, though we cannot explain it in human language. These certainties are stable and our focus.


### 3.3. Dimension Grid Model

In this paper, there are several training corpora and models. To avoid confusion, we endeavor to clearly explain every training process. First, we train sentence2vec with a large enough open corpus, Wikipedia, to obtain the sentence embedding model; then, any sentence could be embedded to continue the following computations. Note that the corpus just mentioned is the corpus used to train and obtain sentence2vec, and once the sentence2vec model is trained, this corpus is useless and could be discarded since the embedding model is what is needed. Additionally, there are training corpora that comprise documents, and each document comprises sentences for coherence computing. We introduce a random document in the training corpus for coherence computing, and other documents could be processed in the same way. Assuming there arem sentences in a document and sentence2vec generates n-dimensional vectors for each sentence, then this document could be represented as a two-dimensional matrix of m×n, and here we denote it as the Val matrix.We compute the whole transition probability by computing the detailed transition probability from every dimension of sentenceT to every dimension of its next sentence i+1. Note that, in the entity grid method, there are only 4 cases (S, O, X, and −) for each entity, and the proportion of each case is always a normal fractional number to some extent. These fractions are used as the transition probability of each entity from one sentence to the next to conduct the following computing. However, for a distributed representation, the range of the dimension value is the whole real space. If we still use the traditional method to calculate the transition pair, a new problem will arise, which is that the denominator would be too large due to the large number of number pairs. In addition, the numerator would be singular because even if the value difference is less than 0.0001, two numbers will be still considered different, and the number pairs will be considered more different. This will cause underflow because the fraction value will be too small. The transition probability is computed in Algorithm 1, and the input of the algorithm is the document set T, which contains all h documents. Each document abbreviated as doc is a list of sentences, and each sentence is represented by a n dimension sentence vector. The output of the algorithm is the transition probability of T from the r dimension to the k dimension in a cluster r denoted as PTrj,kr. More details are shown in the following pseudocode.Algorithm 1: Compute transition probability.
Input Document set T containing all h documentsOutput The transition probability of T(1)
generate the matrix of docs, Val[i][j] = docs·get(i)[j](2)
generate the coordinate list of Val,Cor[j][k] append (Val[i][j], Val[i + 1] [k])(3)
cluster every coordinate, set number of clusters asR, p = [ ][ ][ ]cluster the points in Cor[j][k] and obtain R clustersnum_r = count points in clusters,num_all = count points in Cor[j][k]p[j][k][r] = num_r/num_all(4)
returnpTherefore, we design a method to compute the probability of a transition pair. For the transition from one dimension of sentencei to one dimension of the next sentence i+1, this value pair can be regarded as a point in a two-dimensional coordinate system. That is, the two number values are the coordinate values on the X-axis and Y-axis. Thus, coordinate systems are obtained because there are n×n kinds of transitions, and for each coordinate system, there are m−1 points projected on it. This is just for one document, and we analyse the other documents in the training corpus of coherence computing in the same way, taking the new value pairs as coordinate points and concatenating them to the coordinate systems mentioned above.Specifically, we mark a coordinate system as0,0, indicating that the coordinate system contains the transition information from the 0−th dimension to the 0−th dimension in the training text. Beginning with the 0−th sentence in document doci, we extract the 0−th dimension value in the 0−th sentence Val00 and the 0−th dimension value in the 1−th sentence Val10 to form a point Val00,Val10, and this point will be projected on the coordinate system 0,0. From the 0−th sentence to the end of a document, we will iteratively project m−1 points on this coordinate system 0,0. Then, we can obtain coordinate systems 0,10,2, …, 0,n−1, 1,0, …, n−1,n−1, which are in total n×n coordinate systems with points projected onto them. Based on the coordinate systems just constructed, the new coming points of the other document are concatenated to the corresponding coordinate system.Assume that all the documents are projected onn×n coordinate systems and there are a total of s points in each coordinate system. We then perform clustering for every coordinate system, where the number of clusters is uniformly set for all systems; that is, all coordinate systems use the same number of clusters R for clustering. For each coordinate system, we record the clustering results, including the number of clusters R, each cluster’s label r with its centre point Centroidr, the number of points Countr, and the number of all points in the coordinate system Countall, as our training parameters. The proportion of points in cluster r0≤r≤R−1 accounting for total points in the coordinate system Countr/Countall is used to represent the transition probability of the points contained in r, which is denoted as PTrr. In this way, we will perform the clustering process corresponding to n×n coordinate systems. All this statistical information is stored as model to perform the following computing.


### 3.4. Document Feature Vectors

To represent text in more effective ways, we design the document feature vector quite differently compared to the entity grid. Here, we define the vector that is formed by the document transition probability of every coordinate system. That is, for a training documentdoci, its feature vector is(1)Φdoci=p0,0doci,p0,1doci,…,pn−1,n−1doci,where p is counted iteratively from coordinate system 0,0 to n−1,n−1, and each p is the transition probability of doci on the transition coordinate system corresponding to its subscript.We now discuss how to obtain eachp. Given a training document doci with m sentences, we first obtain sentence vectors using sentence2vec, and then these vectors further make doci, which is represented as a matrix Valm×n with m rows and n columns, as described in the last chapter. Beginning with the 0−th sentence in doci, we obtain the point ptVal00,Val10 and find which cluster it belongs to in the coordinate system 00. Here, assuming that ptVal00,Val10 belongs to cluster rk in coordinate system 00, we assign it the transition probability PTr0,0rk as its probability. Here, we only consider the transition from the 0−th dimension to the 0−th dimension as an example. We analyse the whole document and sum up all the transition probabilities in 00. Then, we normalize the result and use the result as the document transition probability of doci on the coordinate system 00. That is, we assign a normalized result to p0,0doci. The document feature vector is obtained by Algorithm 2, in which we use Φdocij∗n+k to represent pj,kdoci. Then, we can compute the other dimensions in Φdoci in the same way. The input of the algorithm is a document docq that has docq⋅length sentences, and each sentence is represented by n dimension sentence vectors. The transition probability of the training document set from the j−th dimension to the k−th dimension in cluster r is denoted as PTrj,kr, and Corjk denotes the trained coordinate system. The output of the algorithm is the document feature vector Φdocq.Algorithm 2: Compute document feature vectors.
Input document doc_q which has doc_q.length sentences;Output the document feature vector φ(doc_q).(1)
generic the matrix of doc, Val[i] [j] = doc_q.get(i)[j](2)
generic the feature vector of doc,dist (r) = distance of cluster centre r to point (Val[i][j], Val[i + 1] [k])r′ = argmin dist (r)p = p + PTr [j, k] (r′)(3)
normalize,p = p/(doc·length − 1),φdoc_qj∗n+k=p(4)
returnφ(doc_q)Using this method, any document can be represented as a feature vector. We then use these vectors to train the rank-SVM. Note that now the training data are used for the rank-SVM and are slightly different from the original training documents because some self-permutation files are generated. We rearrange the sentences in the original training documents, obtaining each candidate set that consists of a source document and permutations of its sentences, and we use this method to transform them into feature vectors. We assume that the original document is more coherent than any of its permutations, that is,w∗Φdoci>w∗Φdoci−permuation, and this is the constraint to train the rank-SVM. When testing, the test documents with their permutations are used. If w∗Φdoci>w∗Φdoci−permuation, it means that the prediction is correct.


## 3.1. Entity Grid Construction

Barzilay et al. [5, 7] propose the entity grid method to capture the local coherence of text. In the method, a document is represented as a matrix with rows for sentences and columns for entities. It is based on centralization theory [6], which mainly states that text could be represented by some important nouns; if two adjacent sentences contain the same nouns, they are more likely coherent.Entryrij in the representation matrix describes the syntactic role of entity j in sentence i. For rij, there are four markers of the syntactic role to be assigned to the entity: S for subject, O for object, X for other role, and − for an entity that does not appear. Then, the probability of the syntactic role transition (e.g., S⟶O or O⟶O) is computed across adjacent sentences. Given a specific entity, its transition probability can be estimated by calculating the proportions of different entity transition types within a text. These probability values are then used as features to build a coherence model. Barzilay also uses feature selection (i.e., syntax, coreference, and salient) to tag an entity in a text. After these processing steps, they use the document representations as vectors to input into SVM-rank to build a rank model. Figure 2 shows a summary augmented with syntactic annotations for grid computation. In addition, Figure 3 illustrates a fragment of an entity grid constructed for the text in Figure 2.Figure 2
Summary augmented with syntactic annotations for grid computation.Figure 3
A fragment of an entity grid. The grid cells correspond to grammatical roles: subjects (S), objects (O), or neither (X).


## 3.2. Word2vec and Sentence2vec Representation

Word2vec is a natural language processing (NLP) tool presented by Tomas Mikolov et al. [22, 23] group of Google in 2013 that can could vectorize words by using neural network and Huffman trees. We also call this vectorization process word embedding, and the result is a distributed representation, distinguishing it from the traditional one-hot representation in which a word is described using only 0 or 1 in serial. Word2vec has two training methods: the CBOW and skip-gram. In the CBOW, context words are used to predict one specific word, and in the skip-gram, a specific word is utilized to predict the context words. Once the distributed representation is obtained after training using a large enough corpus, we could conveniently measure the relationships between words and obtain results such as “king” − “queen” ≈ “man” − “woman.”Following the success of word2vec, Mikolov’s group proposed vectorizing sentences and then proposed sentence2vec. Sentence2vec is an extension of word2vec to some extent and promotes the embedding from the word level to the sentence level. Different from word2vec, they added the paragraph ID to identify every sentence. The paragraph ID and the word vectors are then concatenated to train and predict the sentence vector. In addition, the CBOW and skip-gram are also in sentence2vec.Practical applications have verified the reliability of these vectors. We propose the model on the premise that once the vector length is set, each dimension is exactly assigned with a stated certainty and specific semantics, though we cannot explain it in human language. These certainties are stable and our focus.


## 3.3. Dimension Grid Model

In this paper, there are several training corpora and models. To avoid confusion, we endeavor to clearly explain every training process. First, we train sentence2vec with a large enough open corpus, Wikipedia, to obtain the sentence embedding model; then, any sentence could be embedded to continue the following computations. Note that the corpus just mentioned is the corpus used to train and obtain sentence2vec, and once the sentence2vec model is trained, this corpus is useless and could be discarded since the embedding model is what is needed. Additionally, there are training corpora that comprise documents, and each document comprises sentences for coherence computing. We introduce a random document in the training corpus for coherence computing, and other documents could be processed in the same way. Assuming there arem sentences in a document and sentence2vec generates n-dimensional vectors for each sentence, then this document could be represented as a two-dimensional matrix of m×n, and here we denote it as the Val matrix.We compute the whole transition probability by computing the detailed transition probability from every dimension of sentenceT to every dimension of its next sentence i+1. Note that, in the entity grid method, there are only 4 cases (S, O, X, and −) for each entity, and the proportion of each case is always a normal fractional number to some extent. These fractions are used as the transition probability of each entity from one sentence to the next to conduct the following computing. However, for a distributed representation, the range of the dimension value is the whole real space. If we still use the traditional method to calculate the transition pair, a new problem will arise, which is that the denominator would be too large due to the large number of number pairs. In addition, the numerator would be singular because even if the value difference is less than 0.0001, two numbers will be still considered different, and the number pairs will be considered more different. This will cause underflow because the fraction value will be too small. The transition probability is computed in Algorithm 1, and the input of the algorithm is the document set T, which contains all h documents. Each document abbreviated as doc is a list of sentences, and each sentence is represented by a n dimension sentence vector. The output of the algorithm is the transition probability of T from the r dimension to the k dimension in a cluster r denoted as PTrj,kr. More details are shown in the following pseudocode.Algorithm 1: Compute transition probability.
Input Document set T containing all h documentsOutput The transition probability of T(1)
generate the matrix of docs, Val[i][j] = docs·get(i)[j](2)
generate the coordinate list of Val,Cor[j][k] append (Val[i][j], Val[i + 1] [k])(3)
cluster every coordinate, set number of clusters asR, p = [ ][ ][ ]cluster the points in Cor[j][k] and obtain R clustersnum_r = count points in clusters,num_all = count points in Cor[j][k]p[j][k][r] = num_r/num_all(4)
returnpTherefore, we design a method to compute the probability of a transition pair. For the transition from one dimension of sentencei to one dimension of the next sentence i+1, this value pair can be regarded as a point in a two-dimensional coordinate system. That is, the two number values are the coordinate values on the X-axis and Y-axis. Thus, coordinate systems are obtained because there are n×n kinds of transitions, and for each coordinate system, there are m−1 points projected on it. This is just for one document, and we analyse the other documents in the training corpus of coherence computing in the same way, taking the new value pairs as coordinate points and concatenating them to the coordinate systems mentioned above.Specifically, we mark a coordinate system as0,0, indicating that the coordinate system contains the transition information from the 0−th dimension to the 0−th dimension in the training text. Beginning with the 0−th sentence in document doci, we extract the 0−th dimension value in the 0−th sentence Val00 and the 0−th dimension value in the 1−th sentence Val10 to form a point Val00,Val10, and this point will be projected on the coordinate system 0,0. From the 0−th sentence to the end of a document, we will iteratively project m−1 points on this coordinate system 0,0. Then, we can obtain coordinate systems 0,10,2, …, 0,n−1, 1,0, …, n−1,n−1, which are in total n×n coordinate systems with points projected onto them. Based on the coordinate systems just constructed, the new coming points of the other document are concatenated to the corresponding coordinate system.Assume that all the documents are projected onn×n coordinate systems and there are a total of s points in each coordinate system. We then perform clustering for every coordinate system, where the number of clusters is uniformly set for all systems; that is, all coordinate systems use the same number of clusters R for clustering. For each coordinate system, we record the clustering results, including the number of clusters R, each cluster’s label r with its centre point Centroidr, the number of points Countr, and the number of all points in the coordinate system Countall, as our training parameters. The proportion of points in cluster r0≤r≤R−1 accounting for total points in the coordinate system Countr/Countall is used to represent the transition probability of the points contained in r, which is denoted as PTrr. In this way, we will perform the clustering process corresponding to n×n coordinate systems. All this statistical information is stored as model to perform the following computing.


## 3.4. Document Feature Vectors

To represent text in more effective ways, we design the document feature vector quite differently compared to the entity grid. Here, we define the vector that is formed by the document transition probability of every coordinate system. That is, for a training documentdoci, its feature vector is(1)Φdoci=p0,0doci,p0,1doci,…,pn−1,n−1doci,where p is counted iteratively from coordinate system 0,0 to n−1,n−1, and each p is the transition probability of doci on the transition coordinate system corresponding to its subscript.We now discuss how to obtain eachp. Given a training document doci with m sentences, we first obtain sentence vectors using sentence2vec, and then these vectors further make doci, which is represented as a matrix Valm×n with m rows and n columns, as described in the last chapter. Beginning with the 0−th sentence in doci, we obtain the point ptVal00,Val10 and find which cluster it belongs to in the coordinate system 00. Here, assuming that ptVal00,Val10 belongs to cluster rk in coordinate system 00, we assign it the transition probability PTr0,0rk as its probability. Here, we only consider the transition from the 0−th dimension to the 0−th dimension as an example. We analyse the whole document and sum up all the transition probabilities in 00. Then, we normalize the result and use the result as the document transition probability of doci on the coordinate system 00. That is, we assign a normalized result to p0,0doci. The document feature vector is obtained by Algorithm 2, in which we use Φdocij∗n+k to represent pj,kdoci. Then, we can compute the other dimensions in Φdoci in the same way. The input of the algorithm is a document docq that has docq⋅length sentences, and each sentence is represented by n dimension sentence vectors. The transition probability of the training document set from the j−th dimension to the k−th dimension in cluster r is denoted as PTrj,kr, and Corjk denotes the trained coordinate system. The output of the algorithm is the document feature vector Φdocq.Algorithm 2: Compute document feature vectors.
Input document doc_q which has doc_q.length sentences;Output the document feature vector φ(doc_q).(1)
generic the matrix of doc, Val[i] [j] = doc_q.get(i)[j](2)
generic the feature vector of doc,dist (r) = distance of cluster centre r to point (Val[i][j], Val[i + 1] [k])r′ = argmin dist (r)p = p + PTr [j, k] (r′)(3)
normalize,p = p/(doc·length − 1),φdoc_qj∗n+k=p(4)
returnφ(doc_q)Using this method, any document can be represented as a feature vector. We then use these vectors to train the rank-SVM. Note that now the training data are used for the rank-SVM and are slightly different from the original training documents because some self-permutation files are generated. We rearrange the sentences in the original training documents, obtaining each candidate set that consists of a source document and permutations of its sentences, and we use this method to transform them into feature vectors. We assume that the original document is more coherent than any of its permutations, that is,w∗Φdoci>w∗Φdoci−permuation, and this is the constraint to train the rank-SVM. When testing, the test documents with their permutations are used. If w∗Φdoci>w∗Φdoci−permuation, it means that the prediction is correct.


## 4. Experiment and Analysis

This section may be divided by subheadings. It should provide a concise and precise description of the experimental results and their interpretation, as well as the experimental conclusions that can be drawn.


### 4.1. Environment Setup

The experiment is deployed on a computer configured as follows: an 8-core processor with a speed of 3.2 GHz, 32 GB of RAM, a 1 TB HDD, and the Ubuntu 16.04 (64 bit) operating system. For sentence2vec, we use its Python implementation provided on GITHUB (https://github.com/klb3713/sentence2vec), and the Python running environment is PyCharm. Our proposed model is implemented using Java, and the running environment is Eclipse in Luna version 4.4.First, we use the Wikipedia corpus (https://dumps.wikimedia.org/jawiki/latest/) [24], which is approximately 14 GB, to train sentence2vec. When performing sentence embedding, the number of dimensions to set needs to be carefully considered. The setting affects the computing complexity on a quadratic order of magnitude and greatly affects the following processes because there are several traversing computing processes that follow. If the number of dimensions is set equal to or greater than 100, 10 k more intermediate results will occur, which will cause a dimensional disaster and greatly increase the workload; therefore, we set the number of dimensions of the sentence embedding as 50 to reduce the computing costs. Sentence2vec contains the word2Vvec process, so the majority of the training time when we train sentence2vec is due to word2vec. After we obtain the sentence2vec model, any sentence could be represented as a vector of 50 dimensions. When performing clustering, we use the Weka (https://www.cs.waikato.ac.nz/ml/weka/) [25] API as the clustering tool, and the data are transformed into the arff format that is accepted by Weka. Weka is a machine learning package written in Java, and we can use its API to conveniently run the algorithm. Here, the k-means clustering method is chosen, and the number of clusters is set to 5, 10, 15, 20, 25, and 30.


### 4.2. Sentence Ordering Experiment

The first experiment consists of ranking alternative sentence orderings of a document, as proposed by Barzilay and Lapata [5] and Elsner [9].


#### 4.2.1. Experimental Data

Here, we use the same sentence ordering corpus [5] as in the entity grid. This corpus contains 2 categories of articles: accidents and earthquakes. For the training set, the accident article set contains 100 articles about accidents, up to 20 random permutations for each article, and 2200 total pairs (a copy of the original article is also contained in the permutations). Furthermore, the earthquake article set also contains 100 articles about earthquakes, up to 20 random permutations of each article, and 2135 total pairs. These are the details of the training set, and the testing set is formed in the same way. Finally, the total numbers of pairs in the testing set are 2087 and 2056.


#### 4.2.2. Results and Analysis

The following baselines are used for comparison.Entity Grid. The entity grid [5] started the grid approaches for modelling text, and many grid-related models later followed. The entity grid uses NLP techniques such as coreference resolution, expressive syntactic information, and salience-based features to obtain its best performance.HMM. The hidden Markov approach [26] seeks to model the cluster transition probability in the coherent context using syntactic features. Sentences need to be clustered in advance, where the number of clusters is tuned as a parameter. We directly take the results from Louis and Nenkova’s paper and report the best results among different combinations of parameter and feature settings.Graph. The graph approach [27] extends the entity grid model to a bipartite graph to represent the text and computes the local coherence of the entity transition in the bipartite graph. This approach outperforms the original entity approach for feature settings.Window Network. The window network approach [28] applies two neural network approaches, recurrent and recursive networks, to the sentence ordering task. It learns syntactic and semantic representation for sentences automatically and has excellent performance in sentence ordering tasks.Tables1 and 2 show the results of our approach compared with others. We can see that the entity grid model presents a strong baseline on the accidents dataset. Our method’s performance is optimal when the number of clusters is set to 20. The experimental results for our method on the accident dataset slightly outperform those of the entity grid model. Furthermore, on the earthquake dataset, our model outperforms the entity grid and graph approaches by a large margin, performs at the same level as the HMM and window (recurrent) method, and performs inferior to the window (recursive) method by approximately 0.02. To sum up, the overall results achieve an excellent performance on the two kinds of data.Table 1
Mean accuracy comparison on the accident and earthquakes for the order discrimination task.
DatasetEntity gridHMMGraphWindow (recurrent)Window (recursive)Accidents0.9040.8420.8460.840.864Earthquakes0.8720.9570.6350.9510.976Table 2
Mean accuracy of our approach by setting different numbers of clusters.
Dataset51015202530Accidents0.8440.8760.9020.9120.9040.900Earthquakes0.8870.9160.9280.9550.9350.926To further verify the efficiency of the algorithm, we compared the training time and response time of different models on two datasets, and the results are shown in Figures4 and 5, which shows our algorithm has the least training time and response time on both datasets.Figure 4
Comparison of the training time of different models on two datasets.Figure 5
Comparison of the response time of different models on two datasets.


### 4.3. Summary Coherence Rating Experiment

To test the effectiveness of our model, a summary coherence rating experiment is also performed. To accomplish this, we use the same corpus from Barzilay and Lapata [5]. This dataset accounts not only for the text coherence but also for other dimensions of the summaries: content selection, fluency, etc. Starting with a pair of summaries in which one is slightly more coherent than the other, we aim to order the two summaries according to local coherence.


#### 4.3.1. Experimental Data

Different from the sentence ordering experiment above, this dataset is taken from the Document Understanding Conference 5 [29] (DUC 2003). The dataset contains 16 document clusters, each with 5 machine-generated summaries and 1 human-written summary; therefore, there are a total of 6 × 16 = 96 summaries, and every 6 summaries include the same information with variations. All 96 summaries are judged and scored by 177 unpaid volunteers who are all native English speakers, and the rating scores range from 1 to 7 for each summary. Every 2 summaries within the 6 are formed as a summary pair, as in the sentence ordering experiment above; therefore, there are 240 pairwise rankings. By eliminating some pairs of summaries in which the human summary scores are identical, we obtained 144 pairwise rankings for training, 80 pairwise rankings for testing, and 6 pairwise rankings for development. Even though all summaries are approximately of the same length (114.2 words on average), their sentence length can vary considerably. Indeed, more coherent summaries tend to have more sentences and contain fewer entities.


#### 4.3.2. Results and Analysis

The baselines chosen here are the entity grid and graph methods introduced in the sentence ordering experiment, and Figures6 and 7 show the comparisons of the accuracy and recall of several methods, respectively. The abscissas indicate the k in the k-means, which we set as 5, 10, 15, 20, 25, or 30.Figure 6
Comparison of the accuracies of different methods on the DUC 2003 dataset. The abscissa indicates thek in k-means.Figure 7
Comparison of the recalls of different methods on the DUC 2003 dataset. The abscissa indicates thek in k-means.The accuracy of our method drops slightly when the number of clusters increases from 5 to 10. The reason is that, different from the previous dataset in which document pairs consist of the same sentence but vary in sentence orders, the dataset in the summary coherence rating experiment is generated from different perspectives, that is, different summaries may use different expressions to convey the same thing. Although 5 is less than 10, it provides better fuzziness and inclusiveness. Overall, the best performance is achieved when the number of clusters is 15, where it balances the inclusiveness of the semantic fuzziness and the richness of the document features. On the one hand, the smaller the number of clusters we set, the more semantics one cluster includes, which is semantic fuzziness. On the other hand, the number of clusters affects the document feature vector because the larger we set the number, the larger the probability space each transfer will have, that is, the richer the document feature vectors. Therefore, in this experiment, 15 is a balanced point, and once this point is exceeded, increasing the number of clusters does not improve the results.


## 4.1. Environment Setup

The experiment is deployed on a computer configured as follows: an 8-core processor with a speed of 3.2 GHz, 32 GB of RAM, a 1 TB HDD, and the Ubuntu 16.04 (64 bit) operating system. For sentence2vec, we use its Python implementation provided on GITHUB (https://github.com/klb3713/sentence2vec), and the Python running environment is PyCharm. Our proposed model is implemented using Java, and the running environment is Eclipse in Luna version 4.4.First, we use the Wikipedia corpus (https://dumps.wikimedia.org/jawiki/latest/) [24], which is approximately 14 GB, to train sentence2vec. When performing sentence embedding, the number of dimensions to set needs to be carefully considered. The setting affects the computing complexity on a quadratic order of magnitude and greatly affects the following processes because there are several traversing computing processes that follow. If the number of dimensions is set equal to or greater than 100, 10 k more intermediate results will occur, which will cause a dimensional disaster and greatly increase the workload; therefore, we set the number of dimensions of the sentence embedding as 50 to reduce the computing costs. Sentence2vec contains the word2Vvec process, so the majority of the training time when we train sentence2vec is due to word2vec. After we obtain the sentence2vec model, any sentence could be represented as a vector of 50 dimensions. When performing clustering, we use the Weka (https://www.cs.waikato.ac.nz/ml/weka/) [25] API as the clustering tool, and the data are transformed into the arff format that is accepted by Weka. Weka is a machine learning package written in Java, and we can use its API to conveniently run the algorithm. Here, the k-means clustering method is chosen, and the number of clusters is set to 5, 10, 15, 20, 25, and 30.


## 4.2. Sentence Ordering Experiment

The first experiment consists of ranking alternative sentence orderings of a document, as proposed by Barzilay and Lapata [5] and Elsner [9].


### 4.2.1. Experimental Data

Here, we use the same sentence ordering corpus [5] as in the entity grid. This corpus contains 2 categories of articles: accidents and earthquakes. For the training set, the accident article set contains 100 articles about accidents, up to 20 random permutations for each article, and 2200 total pairs (a copy of the original article is also contained in the permutations). Furthermore, the earthquake article set also contains 100 articles about earthquakes, up to 20 random permutations of each article, and 2135 total pairs. These are the details of the training set, and the testing set is formed in the same way. Finally, the total numbers of pairs in the testing set are 2087 and 2056.


### 4.2.2. Results and Analysis

The following baselines are used for comparison.Entity Grid. The entity grid [5] started the grid approaches for modelling text, and many grid-related models later followed. The entity grid uses NLP techniques such as coreference resolution, expressive syntactic information, and salience-based features to obtain its best performance.HMM. The hidden Markov approach [26] seeks to model the cluster transition probability in the coherent context using syntactic features. Sentences need to be clustered in advance, where the number of clusters is tuned as a parameter. We directly take the results from Louis and Nenkova’s paper and report the best results among different combinations of parameter and feature settings.Graph. The graph approach [27] extends the entity grid model to a bipartite graph to represent the text and computes the local coherence of the entity transition in the bipartite graph. This approach outperforms the original entity approach for feature settings.Window Network. The window network approach [28] applies two neural network approaches, recurrent and recursive networks, to the sentence ordering task. It learns syntactic and semantic representation for sentences automatically and has excellent performance in sentence ordering tasks.Tables1 and 2 show the results of our approach compared with others. We can see that the entity grid model presents a strong baseline on the accidents dataset. Our method’s performance is optimal when the number of clusters is set to 20. The experimental results for our method on the accident dataset slightly outperform those of the entity grid model. Furthermore, on the earthquake dataset, our model outperforms the entity grid and graph approaches by a large margin, performs at the same level as the HMM and window (recurrent) method, and performs inferior to the window (recursive) method by approximately 0.02. To sum up, the overall results achieve an excellent performance on the two kinds of data.Table 1
Mean accuracy comparison on the accident and earthquakes for the order discrimination task.
DatasetEntity gridHMMGraphWindow (recurrent)Window (recursive)Accidents0.9040.8420.8460.840.864Earthquakes0.8720.9570.6350.9510.976Table 2
Mean accuracy of our approach by setting different numbers of clusters.
Dataset51015202530Accidents0.8440.8760.9020.9120.9040.900Earthquakes0.8870.9160.9280.9550.9350.926To further verify the efficiency of the algorithm, we compared the training time and response time of different models on two datasets, and the results are shown in Figures4 and 5, which shows our algorithm has the least training time and response time on both datasets.Figure 4
Comparison of the training time of different models on two datasets.Figure 5
Comparison of the response time of different models on two datasets.


## 4.2.1. Experimental Data

Here, we use the same sentence ordering corpus [5] as in the entity grid. This corpus contains 2 categories of articles: accidents and earthquakes. For the training set, the accident article set contains 100 articles about accidents, up to 20 random permutations for each article, and 2200 total pairs (a copy of the original article is also contained in the permutations). Furthermore, the earthquake article set also contains 100 articles about earthquakes, up to 20 random permutations of each article, and 2135 total pairs. These are the details of the training set, and the testing set is formed in the same way. Finally, the total numbers of pairs in the testing set are 2087 and 2056.


## 4.2.2. Results and Analysis

The following baselines are used for comparison.Entity Grid. The entity grid [5] started the grid approaches for modelling text, and many grid-related models later followed. The entity grid uses NLP techniques such as coreference resolution, expressive syntactic information, and salience-based features to obtain its best performance.HMM. The hidden Markov approach [26] seeks to model the cluster transition probability in the coherent context using syntactic features. Sentences need to be clustered in advance, where the number of clusters is tuned as a parameter. We directly take the results from Louis and Nenkova’s paper and report the best results among different combinations of parameter and feature settings.Graph. The graph approach [27] extends the entity grid model to a bipartite graph to represent the text and computes the local coherence of the entity transition in the bipartite graph. This approach outperforms the original entity approach for feature settings.Window Network. The window network approach [28] applies two neural network approaches, recurrent and recursive networks, to the sentence ordering task. It learns syntactic and semantic representation for sentences automatically and has excellent performance in sentence ordering tasks.Tables1 and 2 show the results of our approach compared with others. We can see that the entity grid model presents a strong baseline on the accidents dataset. Our method’s performance is optimal when the number of clusters is set to 20. The experimental results for our method on the accident dataset slightly outperform those of the entity grid model. Furthermore, on the earthquake dataset, our model outperforms the entity grid and graph approaches by a large margin, performs at the same level as the HMM and window (recurrent) method, and performs inferior to the window (recursive) method by approximately 0.02. To sum up, the overall results achieve an excellent performance on the two kinds of data.Table 1
Mean accuracy comparison on the accident and earthquakes for the order discrimination task.
DatasetEntity gridHMMGraphWindow (recurrent)Window (recursive)Accidents0.9040.8420.8460.840.864Earthquakes0.8720.9570.6350.9510.976Table 2
Mean accuracy of our approach by setting different numbers of clusters.
Dataset51015202530Accidents0.8440.8760.9020.9120.9040.900Earthquakes0.8870.9160.9280.9550.9350.926To further verify the efficiency of the algorithm, we compared the training time and response time of different models on two datasets, and the results are shown in Figures4 and 5, which shows our algorithm has the least training time and response time on both datasets.Figure 4
Comparison of the training time of different models on two datasets.Figure 5
Comparison of the response time of different models on two datasets.


## 4.3. Summary Coherence Rating Experiment

To test the effectiveness of our model, a summary coherence rating experiment is also performed. To accomplish this, we use the same corpus from Barzilay and Lapata [5]. This dataset accounts not only for the text coherence but also for other dimensions of the summaries: content selection, fluency, etc. Starting with a pair of summaries in which one is slightly more coherent than the other, we aim to order the two summaries according to local coherence.


### 4.3.1. Experimental Data

Different from the sentence ordering experiment above, this dataset is taken from the Document Understanding Conference 5 [29] (DUC 2003). The dataset contains 16 document clusters, each with 5 machine-generated summaries and 1 human-written summary; therefore, there are a total of 6 × 16 = 96 summaries, and every 6 summaries include the same information with variations. All 96 summaries are judged and scored by 177 unpaid volunteers who are all native English speakers, and the rating scores range from 1 to 7 for each summary. Every 2 summaries within the 6 are formed as a summary pair, as in the sentence ordering experiment above; therefore, there are 240 pairwise rankings. By eliminating some pairs of summaries in which the human summary scores are identical, we obtained 144 pairwise rankings for training, 80 pairwise rankings for testing, and 6 pairwise rankings for development. Even though all summaries are approximately of the same length (114.2 words on average), their sentence length can vary considerably. Indeed, more coherent summaries tend to have more sentences and contain fewer entities.


### 4.3.2. Results and Analysis

The baselines chosen here are the entity grid and graph methods introduced in the sentence ordering experiment, and Figures6 and 7 show the comparisons of the accuracy and recall of several methods, respectively. The abscissas indicate the k in the k-means, which we set as 5, 10, 15, 20, 25, or 30.Figure 6
Comparison of the accuracies of different methods on the DUC 2003 dataset. The abscissa indicates thek in k-means.Figure 7
Comparison of the recalls of different methods on the DUC 2003 dataset. The abscissa indicates thek in k-means.The accuracy of our method drops slightly when the number of clusters increases from 5 to 10. The reason is that, different from the previous dataset in which document pairs consist of the same sentence but vary in sentence orders, the dataset in the summary coherence rating experiment is generated from different perspectives, that is, different summaries may use different expressions to convey the same thing. Although 5 is less than 10, it provides better fuzziness and inclusiveness. Overall, the best performance is achieved when the number of clusters is 15, where it balances the inclusiveness of the semantic fuzziness and the richness of the document features. On the one hand, the smaller the number of clusters we set, the more semantics one cluster includes, which is semantic fuzziness. On the other hand, the number of clusters affects the document feature vector because the larger we set the number, the larger the probability space each transfer will have, that is, the richer the document feature vectors. Therefore, in this experiment, 15 is a balanced point, and once this point is exceeded, increasing the number of clusters does not improve the results.


## 4.3.1. Experimental Data

Different from the sentence ordering experiment above, this dataset is taken from the Document Understanding Conference 5 [29] (DUC 2003). The dataset contains 16 document clusters, each with 5 machine-generated summaries and 1 human-written summary; therefore, there are a total of 6 × 16 = 96 summaries, and every 6 summaries include the same information with variations. All 96 summaries are judged and scored by 177 unpaid volunteers who are all native English speakers, and the rating scores range from 1 to 7 for each summary. Every 2 summaries within the 6 are formed as a summary pair, as in the sentence ordering experiment above; therefore, there are 240 pairwise rankings. By eliminating some pairs of summaries in which the human summary scores are identical, we obtained 144 pairwise rankings for training, 80 pairwise rankings for testing, and 6 pairwise rankings for development. Even though all summaries are approximately of the same length (114.2 words on average), their sentence length can vary considerably. Indeed, more coherent summaries tend to have more sentences and contain fewer entities.


## 4.3.2. Results and Analysis

The baselines chosen here are the entity grid and graph methods introduced in the sentence ordering experiment, and Figures6 and 7 show the comparisons of the accuracy and recall of several methods, respectively. The abscissas indicate the k in the k-means, which we set as 5, 10, 15, 20, 25, or 30.Figure 6
Comparison of the accuracies of different methods on the DUC 2003 dataset. The abscissa indicates thek in k-means.Figure 7
Comparison of the recalls of different methods on the DUC 2003 dataset. The abscissa indicates thek in k-means.The accuracy of our method drops slightly when the number of clusters increases from 5 to 10. The reason is that, different from the previous dataset in which document pairs consist of the same sentence but vary in sentence orders, the dataset in the summary coherence rating experiment is generated from different perspectives, that is, different summaries may use different expressions to convey the same thing. Although 5 is less than 10, it provides better fuzziness and inclusiveness. Overall, the best performance is achieved when the number of clusters is 15, where it balances the inclusiveness of the semantic fuzziness and the richness of the document features. On the one hand, the smaller the number of clusters we set, the more semantics one cluster includes, which is semantic fuzziness. On the other hand, the number of clusters affects the document feature vector because the larger we set the number, the larger the probability space each transfer will have, that is, the richer the document feature vectors. Therefore, in this experiment, 15 is a balanced point, and once this point is exceeded, increasing the number of clusters does not improve the results.


## 5. Conclusions

We propose a dimension grid approach, which incorporates sentence embedding representations, to model text coherence. Each dimension of the sentence is taken as grid to refine the semantic granularity, and the clustering method is applied to enhance the fuzziness tolerance of computing the semantic transition probabilities. The key point is that we change the fundamental theory that is the basis of the original entity grid approach; that is, we consider all grids instead of some salient factors. The reliability has been verified and good performance is achieved through experiments. This paper involves more statistical computing and less NLP techniques; therefore, in future research, NLP techniques such as coreference resolution and feature selection could be incorporated, and the impacts that different clustering methods have on results could be researched. We believe that this work gives inspiration to other coherence-related issues such as text generation, QA systems, multidocument summarization, and essay scoring.



---

*Source: 6654925-2021-11-05.xml*