Modeling Structural Similarities between Documents for Coherence Assessment with Graph Convolutional Networks
Year: 2023

Abstract:
Coherence is an important aspect of text quality, and various approaches have been applied to coherence modeling. However, existing methods solely focus on a single document’s coherence patterns, ignoring the underlying correlation between documents. We investigate a GCN-based coherence model that is capable of capturing structural similarities between documents. Our model first creates a graph structure for each document, from where we mine different subgraph patterns. We then construct a heterogeneous graph for the training corpus, connecting documents based on their shared subgraphs. Finally, a GCN is applied to the heterogeneous graph to model the connectivity relationships. We evaluate our method on two tasks, assessing discourse coherence and automated essay scoring. Results show that our GCN-based model outperforms all baselines, achieving a new state-of-the-art on both tasks.

heading: 1 Introduction
text: Coherence describes the relationship between sentences that makes a group of sentences logically connected rather than just a random collection of them (Jurafsky and Martin, 2021). It is an important aspect of text quality (McNamara et al., 2010), and its modeling has been applied in many downstream tasks, including summarization (Parveen et al., 2015; Wu and Hu, 2018), dialogue generation (Mesgar et al., 2020; Xu et al., 2021), machine translation (Xiong et al., 2019; Tan et al., 2019) and document-level text generation (Wang et al., 2021; Diao et al., 2021). Given the importance of the task, there is a long line of methods proposed for coherence modeling.
Previous models leverage linguistic features to solve the problem. For example, entity grid-based methods (Barzilay and Lapata, 2005; Elsner and Charniak, 2011) capture the entity transition between adjacent sentences of a text to model local
coherence; in contrast, graph-based models (Guinaudeau and Strube, 2013; Mesgar and Strube, 2015) measure coherence using the entity-graph of a document. Recently, neural network models (Li and Hovy, 2014; Li and Jurafsky, 2017; Mesgar and Strube, 2018; Xu et al., 2019b; Farag and Yannakoudakis, 2019; Moon et al., 2019; Jeon and Strube, 2020a,b; Mesgar et al., 2021) have been applied to the task due to their strength in representation learning and feature combination. Those models learn a document’s representation from word embeddings or pre-trained language models, giving significantly better performance than previous statistical methods.
However, one drawback of existing neural-based methods is that they solely focus on extracting features within a single document, ignoring the underlying correlations between documents. Coherence describes how sentences of a text connect to each other (Reinhart, 1980; Foltz et al., 1998; Schwarz, 2001). Theoretically, documents with similar connection structures should tend to have similar degrees of coherence, which can be useful prior knowledge for coherence modeling. For example, a model is more likely to accurately assess a new document’s coherence if it can refer to the labels of known documents with a similar organizational structure (see Appendix E.1 for an example).
To fill this gap, we investigate a graph-based approach to model the correlation between docu-
7792
ments from the perspective of structural similarity. The main idea is to connect structurally similar documents through a graph and capture those connectivity relationships using Graph Convolutional Networks (GCN) (Kipf and Welling, 2017). In particular, inspired by Guinaudeau and Strube (2013), we first represent a document as a sentence graph, where nodes are sentences and two nodes will be connected if they contain semantically related nouns. Our method further converts each sentence graph into a subgraph set as it proves to be an efficient approach for measuring the topological similarity between graphs (Shervashidze et al., 2009; Kondor et al., 2009). Then, we construct a heterogeneous graph for the training corpus, containing document and subgraph nodes, based on subgraphs shared between documents. In this way, structurally-similar documents are explicitly linked through the subgraphs (shown in Figure 1). Finally, a GCN is applied to the heterogeneous graph to learn the representation of document nodes while considering the connections between them.
We evaluate our method on two benchmark tasks1: assessing discourse coherence and automatic essay scoring. Experimental results show that our method significantly outperforms a baseline model that does not consider structural similarities between documents, achieving a new state-ofthe-art performance on both tasks. In addition, we provide a comprehensive comparison and detailed analysis, which empirically confirm that structural similarity information helps to mitigate the effects of uneven label distributions in datasets and improve the model’s robustness across documents with different lengths.

heading: 2 Related Work
text: Our work is related to text coherence modeling and graph neural networks (GNN)-based methods for natural language processing (NLP). Coherence Modeling. Inspired by Centering Theory (Grosz et al., 1995), Barzilay and Lapata (2005, 2008) propose an entity-based approach (the entity grid) to assess coherence by considering entity transitions between adjacent sentences of a text. The entity grid model has been improved by grouping entities based on their semantic relatedness (Filippova and Strube, 2007), incorporating entity-specific features (Elsner and Charniak, 2011), replacing grammatical roles with discourse
1https://github.com/liuwei1206/StruSim
roles (Lin et al., 2011). On the other hand, Guinaudeau and Strube (2013) propose an entity graph to capture entity transitions between not only adjacent sentences but also non-adjacent ones. Motivated by the functional sentence perspective of text coherence (Danes, 1974), Mesgar and Strube (2015, 2016) improve the entity graph with graphbased features extracted from text structures. Similarly, we also leverage the structural features of texts. However, instead of feeding individual documents’ structure into the model as coherence patterns, we use them to capture the underlying correlation between documents.
With the advent of deep learning, neural networks have been applied to coherence modeling. Li and Hovy (2014); Xu et al. (2019b) learn to assess coherence by training a model to distinguish coherent texts from incoherent ones using different neural encoders. Tien Nguyen and Joty (2017); Joty et al. (2018) extend the entity grid model with a convolutional neural network. Moon et al. (2019) propose to enrich the coherence features of a document by considering discourse relations and syntactic patterns within it. Jeon and Strube (2020a) design structure-guided inter-sentence attention to learn a document’s local and global coherence patterns. Inspired by human reading habits, Jeon and Strube (2020b) investigate a model to measure a document’s coherence by incrementally interpreting sentences. Our work is in line with the above approaches to learning a coherence model based on neural networks. The main difference is that the above neural models focus on extracting features within a single document, whereas our graph-based approach aims to study the effectiveness of correlations between documents. Specifically, rooting on the linguistic definition of text coherence, we model the correlation from the perspective of structural similarities between documents.
GNN-based Methods for NLP. Graph neural networks are a family of neural networks that operate naturally on graphs. Many NLP problems can be expressed with a graph structure, so there is a surge of interest in applying GNNs for NLP tasks. Marcheggiani and Titov (2017) present a Syntactic GCN to learn latent feature representations of words in a sentence over dependency trees for Semantic Role Labeling. Yasunaga et al. (2017) propose a GCN-based multi-document summarization system that exploits the sentence relation information encoded in graph representations of document
clusters. Yao et al. (2019) build a graph containing documents and words as nodes and used the Text GCN to learn embeddings of words and documents. Lv et al. (2020) design a graph-based approach to encode structural information from ConceptNet for commonsense question answering. Compared with existing work, our graph-based method is different in both motivation and graph construction. For example, we specially design subgraph nodes to connect documents with a similar structure for capturing the structural correlations between samples.

heading: 3 Method
text: Figure 2 shows an overview of our proposed method. We describe step-by-step how to capture the structural similarities between documents, including i) identifying the structure of a document (Section 3.1); ii) representing the sentence graph of a document as a subgraph set (Section 3.2); iii) building a corpus-level heterogeneous graph to connect structurally similar documents based on the shared subgraphs (Section 3.3); iv) applying a GCN encoder to capture connectivity relationships between document nodes (Section 3.4).

heading: 3.1 Sentence Graph
text: To model the structural similarities between documents, we need to identify each document’s structure. We follow Guinaudeau and Strube (2013) to represent a document as a directed sentence graph but with some modifications in graph construction. Specifically, in our implementation, two sentences are semantically connected if there are strong semantic relations between nouns in the two sentences. We use nouns instead of entities (Guin-
audeau and Strube, 2013) because the former shows better performance than the latter in modeling semantic connection between sentences (Elsner and Charniak, 2011; Tien Nguyen and Joty, 2017).
Given a document, we use the Stanza toolkit (Qi et al., 2020) to segment it into sentences {s1, s2, ..., sL} and recognize all nouns in each sentence. For a pair of sentences su and sv (u < v), we compute the similarity score for each pair of nouns from them (one noun from su and the other from sv) and use the maximum similarity score to measure their semantic connection. The score between two nouns is obtained by calculating the cosine value of their embedding. If the maximum similarity score is greater than the preset threshold δ, then the two sentences are considered semantically connected, and we add a directed edge between them (from su to sv). After computing all combinations of su and sv (u < v) in the document, we can build a directed graph containing sentences as nodes (refer to Algorithm 1 in Appendix A).

heading: 3.2 Subgraph Set
text: After obtaining the graph structure of documents, we represent each sentence graph as a subgraph set. The subgraph set is an efficient way to compare topological similarities between graphs (Shervashidze et al., 2009), which we can employ to compare documents in terms of structure.
Graph g is a subgraph of graph G if the nodes in g can be mapped to the nodes in G and the connection relations within the two sets of nodes are the same. If the subgraph contains k nodes, we call it a k-node subgraph. In our method, we only consider subgraphs without backward edges. This
is because when constructing the sentence graph, we process the document from left to right and never look back. We use weakly connected and disconnected subgraphs (shown in Figure 3) since we empirically find they both reflect the properties of a document in terms of coherence.
Given a sentence graph Gi of a document di, we first mine the contained k-node subgraphs by enumerating all combinations of k nodes and corresponding edges in Gi. Subgraphs with intersentence distances greater than w are filtered out because far-distant sentences are less likely to be related. It also reduces the search space when mining subgraphs. In the retained subgraphs, two can have the same structure but only differ in node IDs. We consider them as the same subgraph since they are isomorphic in graph theory. Then, we count the frequency of each k-node subgraph and identify the isomorphic subgraphs using the pynauty tool. Consequently, a sentence graph is represented as a k-node subgraph set (refer to Algorithm 2 in Appendix A).

heading: 3.3 Doc-subgraph Graph
text: A graph is an efficient way to model the correlation between items and has been used in different domains, such as knowledge graphs (Carlson et al., 2010) and social networks (Tang and Liu, 2009). We build a corpus-level undirected graph (on the training dataset), named doc-subgraph graph, to explicitly connect structurally similar documents through their shared subgraphs (shown in Figure 2). The graph contains document nodes and subgraph nodes, and the total number of nodes is the sum of the number of documents (N ) and the number of k-node subgraph types (M ) mined in Section 3.2. We design two types of edges in the graph, including edges between document and subgraph, and edges between subgraphs. We build the first type of edge if a document’s subgraph set contains a subgraph, and set its weight as the product of the subgraph’s normalized frequency in the subgraph set and the subgraph’s inverse document frequency in the corpus. The definition of inverse document
frequency is adopted from TF-IDF, but here it represents how common a subgraph is across subgraph sets of all documents. As for the second kind of edge, we construct it between two subgraphs that appear in the same subgraph set of a document, and its weight is the co-occurrence probability of these two subgraphs. We model the co-occurrence information between subgraphs because it has been shown helpful for comparing similar structures between graphs (Kondor et al., 2009).
Formally, we denote documents in a training corpus as D = {d1, d2, ..., dN} and all types of k-node subgraphs mined from the corpus as SubG = {g1, g2, ..., gM}. We use Gi to denote the sentence graph of document di and Fi = {fi1, fi2, ..., fiM} to denote the k-node subgraph set mined from Gi, where fij denotes the frequency of subgraph gj . We represent nodes in the doc-subgraph graph as V = {v1, ..., vN , vN+1, ..., vN+M}, in which {v1, ..., vN} are documents D and {vN+1, ..., vN+M} are k-node subgraphs SubG.
For any pair of document node vi (i ≤ N) and subgraph node vN+j (j ≤ M), we build an edge between them if gj appears in the subgraph set of di, i.e. fij > 0, and define the edge’s weight as:
Ai,N+j = fij∑M j′=1 fij′ · log N|d ∈ D : gj ∈ d| (1)
where the first term is the normalized frequency of subgraph gj in subgraph set Fi, and the second term is an inverse document frequency factor, which diminishes the weight of subgraphs that occur frequently in subgraph sets and increases the weight of subgraphs that occur rarely. |d ∈ D : gj ∈ d| represents the number of documents whose subgraph set contains subgraph gj . A denotes the adjacency matrix of the doc-subgraph graph with shape (N + M) × (N + M) and is initialized as zero matrix. To make the graph symmetrical, we set the value of AN+j,i to be the same as Ai,N+j .
We also construct edges between any pair of subgraph nodes vN+j and vN+j′ (j ≤ M, j′ ≤ M, j ̸= j′) if gj and gj′ co-occur in the subgraph set of a document, i.e. ∃ di ∈ D : fij > 0, fij′ > 0. The weight is set as the Pointwise Mutual Information (PMI) of these two subgraphs, which is a popular way (Ghazvininejad et al., 2016; Yao et al., 2019) to measure co-occurrence information:
AN+j,N+j′ = log p(j, j′)
p(j) p(j′) (2)
p(j) = |d ∈ D : gj ∈ d|
N
p(j, j′) = |d ∈ D : gj ∈ d, gj′ ∈ d|
N
(3)
The PMI can be positive or negative, we follow previous work to clip negative PMI at 0 since this strategy works well across many tasks (Kiela and Clark, 2014; Milajevs et al., 2016; Salle and Villavicencio, 2019).

heading: 3.4 GCN Encoder
text: We adopt a GCN (Kipf and Welling, 2017) to encode the built doc-subgraph graph. GCN is a graph neural network which directly operates on graphstructured data. By integrating the normalized adjacency matrix, the GCN learns node representations based on both connectivity patterns and feature attributes of the graph (Li et al., 2018).
Formally, given the built graph with (N +M) nodes, we represent the graph with an (N +M)× (N + M) adjacency matrix A. We first follow Kipf and Welling (2017) to add self-connections for each node:
Ã = A+ IN+M (4)
where IN+M is an identity matrix. Then, a twolayer GCN is applied on the graph, and the convolution computation at the l-th layer is defined as:
H(l) = σ ( D̃− 1 2 ÃD̃− 1 2H(l−1)W(l−1) ) (5)
Here, D̃ is the degree matrix (i.e. D̃i,i = ∑
j Ãi,j) and W(l−1) is a layer-specific trainable weight matrix. σ is an activation function, such as ReLU. H(l) is the output of l-th GCN layer; H(0) = X , which is a matrix of node features. We use representations from the pre-trained model as features of document nodes due to its excellent performance on document-level tasks (Guo and Nguyen, 2020; Yin et al., 2021; Zhou et al., 2021). For subgraph nodes, since they have no textual contents, we set their features to zero vectors, which is a common setting in heterogeneous graphs (Ji et al., 2021). Finally, we feed the outputs of the two-layer GCN into a softmax classifier:
P = softmax(H(2)) (6)
and train the model by minimizing the CrossEntropy loss over document nodes:
L = − N∑
i=1
C∑
c=1
Yi,c · log (Pi,c) (7)
where Yi is the label of document node vi with a one-hot scheme, C is the number of classes.
While evaluating, for each document in the test corpus, we add it to the doc-subgraph graph, normalize the adjacent matrix of the new graph, and predict its label (refer to Appendix B).

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate the proposed method on two benchmark tasks, assessing discourse coherence (ADC) and automated essay scoring (AES). The descriptive statistics of the dataset for each task are shown in Appendix C. Assessing Discourse Coherence. ADC is the task of measuring the coherence of a given text. The benchmark dataset for this task is the Grammarly Corpus of Discourse Coherence (GCDC) dataset (Lai and Tetreault, 2018). Specifically, GCDC contains texts from four domains, including Yahoo online forum posts, emails from Hillary Clinton’s office, emails from Enron, and Yelp online business reviews. It is annotated by expert raters with a coherence score ∈ {1, 2, 3}, representing low, medium, and high levels of coherence, respectively. Automated Essay Scoring. AES is a task to assign scores for essays, which has been used to evaluate coherence models (Burstein et al., 2010; Jeon and Strube, 2020b). We follow previous work (Jeon and Strube, 2020b) to employ the Test of English as a Foreign Language (TOEFL) dataset (Blanchard et al., 2014) in our experiments. The corpus contains essays from eight prompts along with score levels (low/medium/high) for each essay.

heading: 4.2 Experimental Settings
text: We implement our method based on the Pytorch library. The pre-trained embedding we use to calculate the similarity between nouns is GloVe (Pennington et al., 2014), and we set the similarity threshold δ to 0.65. For the subgraph set construction, we use 4-node subgraphs as basic units for the ADC task and 5-node subgraphs for the AES task, and limit the maximum sentence distance w to 8 for both tasks. The two-layer GCN is employed in our method, with ReLU as the activation function. We follow previous work (Jeon and Strube, 2020b) to use the representation from XLNetbase as document node features, and initialize XLNet using the pre-trained checkpoint from Huggingface. We use XLNet instead of other pre-trained models,
such as BERT, because the TOEFL dataset contains long texts. For example, some essays have more than 800 words (maybe more than 1000 subwords). Autoencoding-based pre-trained models, such as BERT, limit input text length (usually 512 subwords), whereas XLNet can handle any input sequence length.
For the GCDC dataset, we follow the setting in Lai and Tetreault (2018) to perform 10-fold crossvalidation over the training dataset. As for the TOEFL corpus, we conduct 5-fold cross-validation on the dataset of each prompt, which is a common setting for the AES task (Taghipour and Ng, 2016). Consistent with previous work (Lai and Tetreault, 2018; Jeon and Strube, 2020b), we use mean accuracy (%) as the evaluation metric. For more detailed settings and hyperparameters, please refer to Appendix D. Baselines. To investigate the effectiveness of structural similarities between documents for coherence modeling, we empirically compare our method with a baseline model that does not use this knowledge. We call this baseline XLNet+DNN, which inputs document representations from XLNet as features, learns document embeddings with a twolayer deep neural network (DNN), and uses a softmax layer as the classifier. The only difference between the XLNet+DNN baseline and our method in terms of mathematical form is whether the regularized adjacency matrix D̃− 1 2 ÃD̃− 1 2 is applied (Li et al., 2018). We configure this baseline to have the same number of parameters as our method for a fair comparison.
We also compare with Mesgar and Strube (2016), which feeds subgraphs as input features. For a fair comparison, we input document representations from XLNet to this model, equip it with a two-layer DNN and softmax layer for feature extraction and classification. Furthermore, we compare our method against existing state-of-the-art models for each task to evaluate the effectiveness
of our approach.

heading: 4.3 Overall Results
text: Assessing Discourse Coherence. Table 1 shows the experimental results on GCDC dataset2. The first three rows (Li and Jurafsky, 2017; Mesgar and Strube, 2018; Lai and Tetreault, 2018) in the first block show the performance of embeddingbased models, and the last four rows (Mesgar and Strube, 2016; Moon et al., 2019; Jeon and Strube, 2020a,b) in the same block are the state-of-theart models based on XLNet. With the pre-trained model as the encoder, the latter four models outperform embedding-based methods by a large margin.
We present the performance of the XLNet+DNN baseline and our method in the last two blocks of Table 1. As shown in the table, structural similarity information between documents is helpful for coherence assessment, which improves the average accuracy from 59.10% of the XLNet+DNN baseline to 61.23% of our method. Subgraphs as input features (Mesgar and Strube, 2016) can also enhance performance, but the improvement is much smaller than our method. We speculate that simply concatenating subgraph features cannot efficiently capture structural similarities between documents. By contrast, our method explicitly connects structurally similar documents via a graph, thereby fully utilizing this information. Surprisingly, our simple baseline outperforms previous state-of-the-art models, which are also built on XLNet. This is likely because the GCDC dataset mainly contains short and informal texts, whereas previous sota models were designed to handle long and well-formed documents. By contrast, our method works well on the corpus, achieving the best performance. Automated Essay Scoring. As mentioned in Section 4.1, AES is a task for scoring the quality of essays and has been used to evaluate coherence
2In Tables 1, 2, † denotes that the same XLNet as our method is employed in the model.
models. Hence, to better illustrate the effectiveness of our approach, we report the performance of both existing coherence models (Mesgar and Strube, 2018; Moon et al., 2019; Jeon and Strube, 2020a,b) and models designed to solve the AES task. For the latter, we report the result of Dong et al. (2017), which is a state-of-the-art method for the AES task.
Results on the TOEFL dataset are shown in Table 2. Previous coherence models and the XLNet+DNN baseline give significantly better performance than the AES model in Dong et al. (2017). Similar to the results on the GCDC dataset, subgraphs as input features can slightly improve the performance. However, the XLNet+DNN baseline can not beat the state-of-the-art coherence models on the TOEFL dataset. The results are reasonable because those coherence models are not only based on XLNet but also consider the characteristics of long documents. Consistent with observations on GCDC, our method, considering the structural similarities between documents, outperforms the XLNet+DNN baseline on the TOEFL dataset, giving state-of-the-art results.

heading: 4.4 Performance Analysis
text: To understand how structural similarity works for coherence modeling, we compare our model with the XLNet+DNN baseline in terms of the predicted label distribution and document length.
Predicted Label Distribution. Figure 4 shows the distributions of predicted essay scores from the XLNet+DNN baseline and our model on the TOEFL P1 dataset. The XLNet+DNN’s predictions are strongly biased, with about 60% of essays predicted as medium scores. We speculate this is caused by the uneven label distribution in the TOEFL P1 dataset (10.3%/53.8%/35.9% of low/medium/highscoring essays). By contrast, our model is less affected by the uneven distribution, making more low and high score predictions. We also collect the prediction accuracy of the two models for each essay score. The prediction accuracy of the XLNet+DNN model for low, medium, high scores is 35.29%, 83.71%, 76.47%, and that of our method is 50.00%, 82.02%, 84.87%. XLNet+DNN mainly predicts medium scores, so this label’s recall value is high. Compared with the baseline, our method makes relatively accurate predictions for all essay scores, suggesting that capturing structural similarities between essays helps mitigate the effects of uneven label distribution and thus focuses on learning coherence patterns.
To better verify this, we report the performance of the XLNet+DNN baseline and our method using F1-Macro as the evaluation metric. F1-Macro computes the accuracy for each class independently and then takes the average at the class level. Intuitively, if our model’s predictions are more uniform and accurate, the F1-Macro performance gap between our method and the XLNet+DNN baseline should be no smaller than the gap in terms of accuracy. Table 3 shows the F1-Macro results of the XLNet+DNN baseline and our model on the GCDC dataset. Our method achieves much better F1-Macro results than the XLNet+DNN baseline, and the gap between the two models in F1-Macro is larger than the gap in accuracy, which further demonstrates that our
model makes more even and accurate predictions. Document Length. Figure 6 shows the accuracy trends of the baseline and our method on the TOEFL P1 dataset as essays become longer. The curve of XLNet+DNN generally shows a downward trend, accuracy decreasing as essays’ length increases. The result is not surprising since long documents contain more complicated semantics and thus are more challenging. Our method performs similarly to the XLNet+DNN baseline over short documents (length <= 200). But when essays become longer, our model gives relatively high accuracy and even presents a slight increase (200 < length <= 400). This suggests that structural similarity information helps to improve the model’s robustness when document length increases.

heading: 4.5 Ablation Study
text: We analyze the effectiveness of each type of edge in our method. To this end, we test the performance of our approach by first removing edges between subgraph nodes (ESS) and then removing edges between document node and subgraph node (EDS). Note that if all edges are removed (i.e. each doc-
ument is an isolated node), our method degrades into the XLNet+DNN baseline.
Table 4 shows the results on the GCDC Clinton and TOEFL P1 datasets. We can observe from the table that eliminating any type of edges would hurt the performance. The decline in performance is more significant when removing the EDS than eliminating the ESS. The results are reasonable because edges between documents and subgraphs are the key to connecting documents with similar structures, while edges between subgraphs are considered to further assist it (Kondor et al., 2009).

heading: 4.6 Subgraph Analysis
text: In this section, we statistically investigate which subgraphs (sentence connection styles) mostly correlate to each level of coherence3. Specifically, we calculate the Pearson correlation coefficient between each subgraph and per label, and test the significance of the correlation. Figure 5 shows the top two results on GCDC Clinton and TOEFL P1.
In general, subgraphs positively correlated with higher coherence tend to contain more edges. This is somewhat aligned with the previous finding (Guinaudeau and Strube, 2013) that coherence correlates with the average out-degree of sentence graphs. Weakly connected subgraphs are more
3We perform this analysis on the whole corpus and show readable text examples in Appendix E.2
likely to reflect higher coherence than disconnected ones. Taking results on GCDC Clinton as an example, the top two most correlated subgraphs for low coherence contain isolated nodes or components while nodes in subgraphs for high coherence are (weakly) connected. Furthermore, subgraphs with more connections between adjacent sentences seem more correlated with high coherence. For example, there is an almost linear subgraph (or contains linear structure) in the high category of both datasets. We also find that the subgraph results per coherence level on the GCDC Clinton dataset differ from that on the TOEFL P1 dataset. This could be due to two reasons. First, the two datasets contain texts from various domains with domain-specific writing styles and structures. Second, they are built by different annotators, who may have different preferences for text organization styles.

heading: 5 Conclusion
text: In this paper, we investigated the effectiveness of structural similarity information between documents for coherence modeling. We proposed a graph-based method to connect structurally similar documents based on shared subgraphs, and model the connectivity relations with a GCN. Experiments on two benchmark tasks show that our method consistently outperforms the baseline model, achieving state-of-the-art results on both tasks.

heading: 6 Limitations
text: Despite showing impressive performance, our graph-based approach still has several limitations. The first one is related to the construction of the sentence graph. At present, we consider two sentences to be semantically related if they share similar nouns. But coherence can be achieved not only by describing similar entities but also by discourse (rhetorical) relations (Jurafsky and Martin, 2021). So it will be an exciting direction to incorporate discourse relations into the construction of a graph. The second one is that we implemented our method using only a plain GCN. Recent work has pointed out that the original GCN can be further improved with more advanced aggregation functions (Xu et al., 2019a) or attention mechanisms (Velickovic et al., 2018). So another interesting direction is to explore the benefits of more powerful graph neural networks for our method, which we leave for future study.

heading: Acknowledgements
text: The authors would like to thank the four anonymous reviewers for their comments. This work has been funded by the Klaus Tschira Foundation, Heidelberg, Germany. The first and second authors have been supported by a Heidelberg Institute for Theoretical Studies Ph.D. scholarship.

heading: A Graph Construction
text: Algorithm 1 Constructing sentence graph Input: Document d, threshold δ Output: Sentence graph G
1: S,NS ← stanza(d) ▷ Sentences and nouns 2: L← len(S) 3: G← zeros(L, L) ▷ Init adjacency matrix 4: for u← 1 to L− 1 do 5: for v ← u+ 1 to L do 6: un, vn← len(NSu), len(NSv) 7: sim_scores← [ ] 8: for a← 1 to un do 9: for b← 1 to vn do
10: ea ← embed(NSu,a) 11: eb ← embed(NSv,b) 12: score← cos_sim(ea, eb) 13: Append(score, sim_scores) 14: end for 15: end for 16: max_score← max(sim_scores) 17: if max_score > δ then 18: Gu,v ← 1 19: end if 20: end for 21: end for
Algorithm 2 Counting Subgraph Frequency Input: Sentence graph G, subgraph size k, max
sentence distance w Output: subgraph set freq
1: freq ← {} ▷ frequency of each subgraph 2: nodes← G.nodes() 3: i, n← 0, len(nodes) 4: while i < (n− k + 1) do 5: w_n← nodes[i : i+ w] ▷ distance < w 6: k_node_combs← combinations(w_n, k) 7: for k_nodes in k_node_combs do 8: subgraph← extract(G, k_nodes) 9: signature← pynauty(subgraph)
10: Add(freq[signature], 1) 11: end for 12: i← i+ (w − k + 1) 13: end while

heading: B Train and Evaluation
text: Vanilla GCN is a transductive method in which both training and test data are presented to the model during training. This, however, is not applicable in practice since we do not know the eval-
Algorithm 3 Evaluation Input: Test corpus TC, Doc-subgraph graph G,
Trained GCN Output: Predictions preds
1: preds← [] 2: N ← len(TC) 3: for i← 1 to N do 4: di ← TC[i] 5: G∗ ← Add(di, G) ▷ Add document 6: G∗ ← Norm(G∗) ▷ Norm graph 7: li ← GCN(G∗) ▷ Predict label 8: Append(li, preds) 9: end for
uation documents in advance. To overcome this drawback, we implement an inductive GCN inspired by the work in fast GCN (Chen et al., 2018). Specifically, we first construct the doc-subgraph graph based on the training corpus (Section 3.3) and train GCN on this graph (Section 3.4). While evaluating, for each document in the test corpus, we add it to the doc-subgraph graph, normalize the adjacency matrix of the new graph, and predict its label (refer to Algorithm 3). Consequently, our method is in a pure inductive setting. That is, our model does not see the test corpus during training, and its evaluation is performed on individual documents without using the information of other samples in the test corpus. Note that when calculating weights for edges between the newly added document node and subgraph nodes, the inverse document frequency we used in equation (1) is the one we computed using only the training corpus.

heading: C Dataset Description
text: The statistics of the GCDC and TOEFL datasets is shown in Table 5.

heading: D Detailed Experimental Setting
text: For the GCDC dataset, we perform 10-fold crossvalidation over the training dataset following previous work (Lai and Tetreault, 2018). The dimensionality of the two-layer GCN is set to be 240 for Clinton and Enron domains, and 360 for Yahoo and Yelp domains. We use the Adam optimizer with an initial learning rate of 0.01 on Clinton and Enron, 0.008 on Yahoo and Yelp. As for the TOEFL corpus, we conduct 5-fold cross-validation on the dataset of each prompt, which is the common evaluation setting for the AES task (Taghipour and Ng, 2016). A two-layer GCN with dimension size 240 and the Adam optimizer with an initial learning rate of 0.05 is employed for every prompt dataset. Dropout with a rate of 0.5 is applied to both tasks. And we train the model for 160 epochs on the GCDC dataset and 400 on the TOEFL dataset. For the XLNet+DNN baseline, we configure it with the same trainable parameters as our method. As for Mesgar and Strube (2016), we concatenate the document presentation from XLNet and subgraphs mining from the document’s sentence graph as input, and also use a two-layer DNN and a softmax layer. Note that this model has more trainable parameters since its input dimension becomes larger (the concatenation of subgraphs and XLNet representation). For other baseline models, we apply the same experimental setting and XLNet to them as our method and tune their hyperparameters according to the performance on the Dev set. We conduct all experiments on a single Tesla P40 GPU with 24GB memory. It takes about 0.5 days to train our model on the GCDC dataset and 1.5 days on the
TOEFL dataset. We follow previous works (Lai and Tetreault, 2018; Taghipour and Ng, 2016) to use the mean of multi-run accuracy (std) as the evaluation metric.

heading: E Examples
text: E.1 Text Example Coherence describes how sentences of a text connect to each other (Reinhart, 1980; Foltz et al., 1998; Schwarz, 2001). Theoretically, documents with similar connection structures should tend to have similar degrees of coherence. To help readers understand it, we show two texts in Figure 7. Although the two texts have different content, they share very similar connection structures. For example, the first text first talks about Africa, then discusses specific African countries, and finally makes a conclusion. The second text starts with exercise, then goes to certain daily sports, and finally makes a summary. Based on the linguistic definition of text coherence, the two texts should have a similar degree of coherence due to their similar organizational structures. This could be a very useful prior knowledge when we measure a text’s coherence. For example, in Figure 7, we can easily assess the coherence of one text by referring to the label of the other one since they have very similar organizational structures.
E.2 Subgraph Examples We show several text pieces with constructed subgraphs in Figure 8 (from the GCDC Clinton dataset) and Figure 9 (from the TOEFL P1 dataset). In each example, the corresponding subgraph is
shown on the left. We use blue boxes to mark the recognized nouns in each sentence and link semantically related nouns between different sentences by a directed edge between two boxes. Two sentences will be connected if there are semantically related nouns between them.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In Section 6.
7 A2. Did you discuss any potential risks of your work? This is an entirely technical paper. We don’t think it has any risk of bias or otherwise.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In Abstract section and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: In Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In Section 4.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We are the member of LDC, so we can use those corpora.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Those corpora are created for research purpose.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Those corpora have been widely used in the field for a long time. We don’t think it contains any offensive content.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? In Section 4.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Appendix C.
C 3 Did you run computational experiments?

heading: In Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? In Appendix D.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In Appendix D.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In Section 4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In Section 3.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: So another interesting direction is to explore the benefits of more powerful graph neural networks for our method, which we leave for future study