COHESENTIA: A Novel Benchmark of Incremental versus Holistic Assessment of Coherence in Generated Texts
Authors: Aviya Maimon
Reut Tsarfaty
Venue: EMNLP 2023 Main (2023)
Keywords: coherence
benchmark
cohesion
consistency
relevance
linguistic theory
gpt
nlp applications

Abstract: Coherence is a linguistic term that refers to the relations between small textual units (sentences, propositions), which make the text logically consistent and meaningful to the reader. With the advances of generative foundational models in NLP, there is a pressing need to automatically assess the human-perceived coherence of automatically generated texts.
Up until now, little work has been done on explicitly assessing the coherence of generated texts and analyzing the factors contributing to (in)coherence. 
Previous work on the topic used other tasks, e.g., sentence reordering, as proxies of coherence, rather than approaching coherence detection heads on. In this paper, we introduce {\sc CoheSentia}, a novel benchmark of human-perceived coherence of automatically generated texts. 
Our annotation protocol reflects two perspectives; one is global, assigning a single coherence score, and the other is incremental, scoring sentence by sentence. The incremental method produces an (in)coherence score for each text fragment and also pinpoints reasons for incoherence at that point. Our benchmark contains 500 automatically-generated and human-annotated paragraphs, each annotated in both methods, by multiple raters. Our analysis shows that the inter-annotator agreement in the incremental mode is higher than in the holistic alternative, and our experiments show that standard LMs fine-tuned for coherence detection show varied performance on the different factors contributing to (in)coherence. All in all, these models yield unsatisfactory performance, emphasizing the need for developing more reliable methods for coherence assessment.

COHESENTIA:
A Novel Benchmark of Incremental versus Holistic
Assessment of Coherence in Generated Texts
Aviya Maimon and Reut Tsarfaty
Department of Computer Science, Bar Ilan University aaviyamm,reut.ssarfin (sarfaty) gmail.com
Abstract
Coherence detection plays a pivotal role in NLP tasks or applications that are required to generate human-like extended texts, or otherwise measure the text quality; these include text summarization, story generation, long-form question-answering, and automatic essay scoring, to name just a few. In this era of large language models (LLMs), gen- erating texts is at the forefront of NLP research, and the generation capacity of LLMs has an un- precedented celebrated success. However, using LLM--genrrted texts blindly is not always viable option since texts may ultimately be rendered inco- herent, inconsistent, or both. These texts typically require human judgment regarding the meaningful- ness and quality of the generated text.
Coherence is linguistic term that refers to the relations between small textual units (sentences, propositions), which make the text logically consistent and meaningful to the reader. With the advances of generative foundational models in NLP, there is pressing need to automati- cally assess the human-perceived coherence of automatically generated lexts. Up until now, lit- tle work has been done explicitly assessing the coherence of generated texts and analyz- ing the factors contributing to (in)coherence. Previous work on the topic used other tasks, e.g., sentence reordering, as proxies of coher- ence, rather than approaching coherence de- tection heads on. In this paper, we introduce COHESENTIA, novel benchmark of human- perceived coherence of automatically gener- ated texts. Our annotation protocol reflects two perspectives: one global, assigning single coherence score, and the other is incremental, scoring sentence by sentence. The incremen- tal method produces an (in)coherence score for each text fragment and also pinpoints rea- sons for incoherence at that point. Our bench- mark contains 500 automatically--eneettt human-annotated paragraphs, each annotated both methods, by multiple raters. Our analy- sis shows that the anter-annotator agreement in the incremental mode is higher than in the holis- tic alternative, and our experiments show that standard LMs fine-tuned for coherence detec- tion show varied performance on the different factors contributing (in)coherence. All in all, these models yield unsatisfactory performance, emphasizing the need for developing more reli- able methods for coherence assessment.
To reduce the substantial manual effort by hu- mans, required for validating the quality of the text in order use it, ideally automatic model for coherence assessment will be made available. This would reduce the reliance on monn-mnnnuinng and expensive human judgments, allowing for faster and more reproducible evaluations. However, no standard metric or automatic model currently exist or are readily available for this task.
and
In the pre-neural era, large body of work on models for coherence was inspired by linguistic theories. For instance, entity-based local models (Barzilay and Lapata, 2008; Elsner and Charniak, 2011) consider syntactic realization of entities in adjacent sentences, inspired by the centering theory (Grosz et al., 1995). Another line of work, inspired by discourse theories such as Rhetorical Structure Theory (RST) (Mann and Thompson, 1988), is using discourse relations between sentences to pre- dict local coherence (Pitler and Nenkova, 2008; Lin al., 2011), 1). Other notable methods include word co-occurrence-basee local models (Soricut and Marcu, 2006), content (or topic distribution)- based global models (Barzilay and Lee, 2004), and syntax- vyntax-based local and global models (Louis and Nenkova, 2012).
1 Introduction
Coherence is an essential property of well-written texts that refers the way different textual units relate to one another. Thanks to these relations, the text uppears to be logically consistent and semanti- cally meaningful to human readers.

---

Due the fact that properties of coherence re- main ambiguous and hard to formalize, much NLP work on detecting coherence done by creating models that are based on proxy tasks such as sum- marization, essay scoring (Burstein et al., 1998) and, most commonly, the sentence reordering task (Barzilay and Lapata, 2008). Such proxies, while related to coherence, do not necessarily capture coherence fully, resulting in coherence models that are sensitive to task-specific urtefacts and can be misleading. So far, to the best of our knowledge, the only dataset annotated explicitly for human- rated coherence is GCDC (Lai and Tetreault, 2018) which contains coherence scores for real-world texts in four domains. However, with the increased use of automatically-geeerated text by e.g.. the GPT models' family, coherence scoring bench- mark based on generated texts is called for.
In sum, the contribution of this paper is threefold. First, we propose a novel coherence-annotalie protocol that obtains higher agreement on human judgments than previous rating practices. Second, we present a benchmark labeled with this scheme, which is to the best of our knowledge the first benchmark for assessing the quality of models for detecting coherence of generated texts. Finally, we provide first glimpse into the coherence factors that can be learned or identified by contemporary LLMs, versus factors that are harder to detect.
Dissecting the Task of Coherence
2.1 Linguistic Fundamentals of Coherence
Coherence is clusive concept, however, funda- mental work in linguistics sheds light on its com- ponents. The seminal work of Reinhart (1980), asserts that for a text to be coherent it has to meet three conditions: (i) cohesion, (ii) consistency and (iii) relevance, capturing formal, logical and prag- matic aspects of the text, respectively. Let us briefly introduce each condition in turn.'
In this work, we set out fill the gap and ex- plicitly address the human-based coherence of au- tomatically generated texts. Concretely, we present COHESENTIA, new benchmark for human as- sessment of automatically generated texts, which are generated by a generative LLM, annotated by human workers for rating the coherence score of the artificially-gnnrratedd text. Crucially, instead of using a single holistic score of coherence for each text, we propose novel incremental, sentence-by- sentence, labeling scheme that manifests the con- ceived coherence of the text fragments, shedding light on both the specific factors that tend to break the coherence for human readers, and the specific points where the coherence broke. This benchmark and scheme provide insights into discourse coher- ence, allowing to dissect and investigate the spe- cific factors contributing to coherence towards advanced computational modeling of coherence.
(i) Cohesion is matter of the linear concatena- tion of sentences. It requires that sentences in the text will be formally connected. They can be con- nected by either a referential link (co-reference, bridging anaphora) or by a semantic connector (a.k.a,, discourse relations as in the PDTB (Milt- sakaki et al., 2004)). This condition is essentially syntactic, and has to do with the form of the text. (ii) Consistency a logical requirement wherein every sentence must align with the preceding sen- tences. Formally it requires that the current and previous sentences can be true within a (one and the same) specific world, taking into account the assumptions and limitations of that world. This condition is formally semantic, and has to do with the ways we interpret texts to construct meanings. (iii) Relevance restricts not only the relations between the sentences but also between those sen- tences and the underlining discourse topic, and their relations with the global context of the utter- ance. This condition subscribes to Grice's prag- matic maxims of relevance and manner, and sumes collaborative speaker (Grice, 1975).
We compare the standard holistic coherence scor- ing with our incremental scheme and show that the incremental method does not only enable us to pin- point the incoherence causes and positions where it breaks, but also, the global score obtained for via the incremental methods shows higher inter- annotator agreement than when using the holistic method. In addition, for LLMs fine-tuned on this task, we show that different size LMs have similar Fl score, corroborating our assumption that while LLMs capture many important aspects they still lack the ability to assess coherence. We also as- sess the detection of different coherence factors by LLMs. Their performance is, at best, varied.
According to Reinhart, this set of conditions is not only verifiable for already coherent texts, but can further be vehicle for determining whether given text is coherent not, in way which is
'Examples for these condition are given in Appendix A.

---

determined in accordance with human speakers in- tuitions. We will use this functional equivalence to dissect the task of coherence detection and scoring, while also unveiling the reasons for (in)cceeeence and detecting problematic segments in the text.
#Sentences
#Stories
#Words
#Stories
1-5
169
1-50
20
6-8
205
51-100
167
9-10
83
101-150
177
11-15
46
151-200
97
201-300
42
2.2 Practical Considerations of Coherence: Holistic versus Incremental Detection
Table I: The number of sentences and words per story.
3.1 Story Generation and Text Cleaning
Determining the coherence score of a text poses a complex challenge because individuals interpret the text differently based on their knowledge, pre- conceived notions, and their ability to make dis- course coherent even when it is not inherently so. Consequently, assigning a coherence score to an entire paragraph is a daunting task, and its results may vary among different human annotators.
We first created list of story titles using GPT-3 by providing the prompt "Provide story title" and later, carefully selected the final list of story titles through manual curation. Then, we employed GPT-3 generate story for each title on the list. In all cases we used the prompt: "Write me co- herent story with the title <title>"
To address this, propose study coherence phenomenon being accumulated in a paragraph incrementally, sentence by sentence, rather than a holistic score. At each iteration, an extra sentence is presented and one needs to decide the coherence status of the paragraph thus far. Moreover, when taking this incremental perspective, at each point of detecting incoherence, specific reason vis- à-vis Reinhart's conditions can be more easily pointed out (up to multiplicity of factors con- tributing the coherence of the overall text). After scanning all sentences in the paragraph, one can assign a final coherence score for the paragraph.
We used three versions of GPT-3: most sto- ries were generated using "text-davinci-003" oth- using "text-curie-001" and "text-bahbage-001" The temperature was mostly 0.7, maximum length was 256 tokens for all stories. We also ran- domly changed the frequency penalty and presence penalty for different stories in order to create more versatile text and change the course of the story. We wanted each text to be long enough to exhibit a range of characteristics of local and global coher- ence, but not too long so that the labeling process does not become too tedious. Using this method, created 500 stories with an average number of 150 tokens of different lengths, as seen in Table 1. To ensure that annotators focused solely on as- sessing the coherence level of the text and to elim- inate any confusion caused by superficial errors such as grammar mistakes and punctuation issues, manually cleaned all texts before presenting them to the raters. We cleaned the text by first using regex for recognizing contiguous multiple punctuation, lack of space between sentences and words, invalid characters in text, and more. After- wards, we thoroughly reviewed the text and manu- ally corrected the punctuation and grammar issues to ensure proper text form.
In our endeavor, we held two annotation proto- cols, where every text was annotated holistically as well as incrementally, in order assess the differ- ence between those two perspectives.
3 Data Creation and Curation
The purpose of creating the coherence benchmark twofold. First, due to the increased use of encoder-decoder models for generating text auto- matically, we aim to assess the coherence of such generated texts, as these texts are often considered with high quality and are used blindly but in reality may lack coherence. Second, we aimed to enable a comparison between human annotations for coher- ence scoring, when employing holistic versus incre- mental annotation approaches, in order to provide an optimal validated protocol for future use. The stages of benchmark creation have been as follows: (1) story generation and text cleaning (Section 3),, and (2) coherence annotation (Section 3.2). We now elaborate on each of these stages.
3.2 Coherence Annotation
We collected coherence judgments via Amazon Me- chanical Turk (MTurk), from 7 human raters with prior experience annotation for NLP tasks. To plore and compare the factors impacting coherence scoring, we developed two annotation protocols.
The first involves a holistic approach, where raters evaluate the overall coherence score of the story.

---

The second is incremental, where raters assess the coherence of each sentence in the story before as- signing final score to the entire story.
This method was developed to investigate the reasons behind the annotator's decision on lower coherence in some stories, to identify common is- sues in generation models, with the face towards comparing the ability of coherence detection mod- els in sentence level vs. paragraph level. Addition- ally, we aimed to assess whether annotating each sentence individually enhances the reliability of the final coherence score, the rater had time comprehend each part of the passage fully. Once all sentences in the story were evaluated, the an- notator provided a coherence score for the entire paragraph.
We collected MTTurk worker judgments for each text, using both methods, from multiple raters, where each rater has undergone qualification train- ing for both approaches before rating any one of the texts in any method. We trained the raters for both methods in multiple batches ensure they pos- sessed an understanding of coherence and could serve as experts, ensuring the benchmark's quality, Previous work showed that MTurk raters as- signed the task of coherence scoring without train- ing showed low inter- annotator agreement (Lai and Tetreault, 2018). Our approach allows us exam- ine if certain annotation protocol may yield better anter-annotator agreement.
Data Analysis
4.1
The COHESENTIA Benchmark
The COHESENTIA benchmark we deliver consists of 500 generated stories with average of 6.5 sentences per story, each of which has been as- sessed for coherence by a minimum of MTurk raters. A consensus label was determined for each method. To compute the consensus label, the co- herence ratings from the MTurk workers were av- eraged and then clustered into 5 groups ranging from 1 to 5. The incremental acquisition method involved annotating approximately 3.5k sentences for (in)coherence reasons. An example of the sulting annotation is provided in Appendix D. The dataset offers fresh view of coherence patterns in generated texts, offering insights into discourse structure, stylistic choices, and the impact of coher- ence on the overall text quality.
3.2.1 The Holistic Protocol
The first protocol involves a simple approach where the human annotator reads the text and has to as- sign coherence score to the story on a scale of 0-100, given instructions which are based on prior coherence annotation efforts: "A text with high co- herence is one that you understand what the author tried to pass on, it is well organized text and con- tains relevant details to the main subjects in the text. Try ignore grammar or spelling errors." (Barzi- lay and Lapata, 2008; Louis and Nenkova, 2012). An example of the holistic method annotation page is given in Appendix B.
3.2.2 The Incremental Protocol
The second annotation protocol is a sentence-by- sentence annotation scheme, where at each step the rater decides whether the current sentence is coherent individually and with regards to the pre- viously presented sentences. If the rater deems sentence coherent, they proceed to the next sen- tence. Otherwise, list of reasons is presented and the rater is required to choose the reasons for the detected incoherence at that particular point. The pool of reasons we provided is based on Reinhart's linguistic conditions for incoherence, but narrated in non-linguistic, colloquial and understandable manner. The reasons and associated conditions are presented in Table 2. Raters were not able to view the full story before assessing every sentence's co- herence and only after completing the sentence-by- sentence process did they assign final coherence score to the entire passage. The incremental anno- tation page is given in Appendix C.
4.2 Properties of the Dataset
4.2.1 Final Score
The stories final coherence score was calculated by a weighted average among the annotators, per each annotation method. The dataset is unbalanced as there are more stories annotated with high coher- ence than lower coherence. The distribution of the final scores for each method is in Figure la.
4.2.2 Incoherence Reasons Distribution
Figure lb shows the fraction of unique incoherence reasons of each reason type annotated across all dataset. The majority of the incoherence reasons are due to cohesion errors while relevance issues are less frequent.
4.2.3 Inter-Annotator Agreement
In order to know whether individuals' intuition for coherence scoring is in agreement with one another,

---

Condition
Reason
'The sentence doesn't make sense"
Cohesion
"The new sentence discusses an entity which has not been introduced yet'
The relation between this sentence and previous ones doesn't tt make sense"
Consistency
The new senience contains information inconsistent with previous presented data"
Thee new sentence contains information inconsistent with your knowledge about the world'
Relevance
The new sentence relevant to the title'
'The new sentence relevant to previous data the story' 'Other'
Table Reasons for incoherence based on Reinhart's conditions
Protocol
ICC
krip.
Group
Correlation
Holistic
0.804
0.694
0.66
Total
-0.89
Incremental
0.968
0.827
0.86
Cohesion
-0.81
Consistency
-0,63
Table 3: Inter-annotator Agreement in both methods.
Relevance -0.44
Group
ICC
6
Krip.
Table 5: Pearson's correlation between the number of incoherence reasons to final coherence score per incoherence reasons group.
Coherence
0.96
0.87
0.90
Cohesion
0.96
0.87
0.88
Consistency
0.91
0.81
0.86
Relevance
0.95
0.69
0.76
4.3 Main Statistic
Table 4: Inter-annotator agreement sentence score for incoherent sentences and each possible group of reasons
Human Annotators Assessing Coherence Focus Language Form more than Content: To un- derstand which aspects of the paragraph contribute to the final coherence score, we compute the corre- lation between the number of incoherence reasons of cach group with the overall coherence score (shown in Table 5). It shows that the total number of errors is correlated with the overall coherence score, but annotators tend to weigh incoherence with cohesion issues more than consistency issues and a lot more than with relevance issues.
we report the agreement for intraclass correlation (ICC), quadratic weighted Cohen's к and Krippen- dorff's a (Krippendorff, 2004) for an ordinal scale. For all future references, k refers to weighted Co- hen and krip. refers to Krippendorff's a.
Paragraph-level Agreement is shown in Table 3. We can see that the agreement for the incremental method is higher than for the holistic method, on all measures. This suggests that when annotators carefully consider each sentence in relation to the entire paragraph, there are less dispersed views and greater consensus. Table 10 further shows the agreement on the number incoherent sentences for each paragraph, and the agrcement on the num- ber of incoherent sentences from each type.
Holistic and Incremental Score Comparison: Comparing the final coherence scores obtained the two methods, we see that there generally little difference in the final scores when the coherence is high. However, as the coherence score decreases, larger discrepancy is observed between the two methods. Notably, for lower coherence scores in the incremental method, the coherence score tends to be higher compared to the holistic method. This phenomenon can be attributed the annotator's thorough examination each sentence in the para- graph, attempting to extract as much understanding as possible and exhaust the available information.
Sentence-level Agreement is shown in Table where we check how much the annotators agreed per sentence on the incoherence status and inco- herence types. We can see that annotators tend to agree lot more on incoherence because of co- hesion rather than relevance. This indicates that humans tend to consider which data is relevant to story differently, possibly imposing semantic and discourse links that could make such details rele- vant. Additional agreement values on the number incoherent sentences and associated reasons, are presented in Appendix E.
Reasons for Incoherence: As coherence scores decrease, it is observed that the number of reasons for incoherence increases.
In Figure we can see that once a cohesion-type error appears paragraph, the model identifies more sentences as incoherent due to the same type of reason, rather than when an 'irrelevant' sentence

---

cohesiin consisttancy
200
relevance
Narter Phrappoto
350
300
100
Score
Distribution coherence scores
Figure 2:
The number of paragraphs containing the number of incoherent sentences from the same group. E.g., there are 49 paragraphs with irrelevant sentences.
200
1. The Coherence Scoring Task: We aim to de- vise a model that delivers global coherence score for each story. That is, given story P with title T, the model produces the final coherence score C. This task is designed evaluate how well standard Pre-trained LLMs capture coherence.
330
007
corahiiency
(b) Distribution of incoherence reason classes clus-
tered into groups
SETIE
Cata consantency
ertttty
knowleddge
2. The Coherence Reasoning Task: We aim devise model that can detect, for each sentence in the text, its (in)coherence status and the reason for incoherence. Formally, let D = d1, d2, du be the accumulative para- graphs sequence such that for each i, d, con- tains all the sentences in the paragraph up to and including i-1. Then, S = 81, $2: ..... 8, is the sequence of the entire paragraph. Given di and 8;, the model aims predict whether d, cohesive, consistent and relevant based on 8;. This will allow us mark the hardest capture causes for incoherence the text and see how well PLM models capture each varied coherence aspect.
300
200
Distribution of incoherence classes
Figure/ 1: Distribution of final score, incoherent reasons, and the reasons clustered into groups
is identified. This implies that even when the model produces sentence unrelated to the context, it demonstrates a tendency to maintain some level of story relevance. However, the appearance of non- cohesive sentence will affect the final score less than the effect of one irrelevant sentence. (Further data on this presented in Appendix F).
Data: For the Coherence Scoring task we used the consensus tic and incremental methods separately. For the Coherence Reasoning task the data we use is the (in (in)coherence reasons for each sentence, as acquired in our incremental annotation variant.
single label, for the holis-
5 Experiments
Goal: Using the new benchmark, out as- sess the performance standard contemporary LLMs on two kinds of tasks:
In our annotation setup, each condition of Rein- hart is spelled out as 2-3 reasons that the rater may select. In our experiments, however, the classifi- cation labels we use are the coherent conditions attributed to Reinhart, rather than the spelled out reasons, as there was higher agreement on the conditions themselves than the individually spelled out reasons (can be seen in Appendix E).
1. The Coherence Scoring Task
2. The Coherence Reasoning Task
The second of which is, to the best of our knowl- edge, first of its kind in the NLP literature. Let us define in detail each of these tasks.

---

Given that multiple annotations are assigned to each paragraph, sentence may have a reason cho- sen by one annotator but not by others. To mark a reason from a specific group, we require the agree- ment of more than half of the annotators selecting reasons from that group.
Results
6.1 Fine-Tuned Experiments
6.1.1 The Coherence Scoring Task
Figure 3 shows the results of the classification- based and ceneration-basee architectures for the different models, for both the holistic protocol and the incremental protocol. We can see that the re- sults for all methods are relatively low, thus corrob- orating our hypothesis that while such models learn important text features, coherence scoring contains intricate features that are much harder for such a model to capture. Interestingly, the outcomes for larger LLMs exhibit only small improvements com- pared with their smaller counterparts, suggesting that both models have limitations in comprehend- ing and identifying coherence, and that size and scaling are not major factors contributing to mod- els' ability of capturing coherence.
For both tasks, the data was split into train/dev/test with a 80/10/10 ratio of the stories. For each story, the title was concatenated at as the first sentence at the beginning of the story.
Models: We designed two neural models based on two types of LLMs, an encoder-only model with classification head, and a generative model with carefully designed prompt, either of which produces coherence score for the input paragraph.
1. Classification -Bsed Modeling: We checked several models including the base and large versions of BERT (Devlin et al., 2019) and DeBERTa (He et al., 2023). The output of each prediction is either one of classes for the coherence score task, or 2 labels (yes/no) for each group- cohesion, consistency and rel- evance in the Coherence Reasoning Task. The input the encoder in the coherence scoring task is the title concatenated with the text "T, p". In coherence reasoning it is the previous text and the current sentence "d, <SEP> s;".
Additionally, results on the incremental data are higher than the ones from the holistic method, which suggests that the model learns better repre- sentations of coherence from the incremental coher- ence scoring, possibly due decomposing the large context the shorter contexts included in the task. Interestingly, when we finetuned GPT-3, the model used to generate the text in the first place, on coherence scoring, the results were not superior to the other models.
6.1.2 The Coherence Reasoning Task
2. Generation-Base Based Modeling: We use Flan-T5 (Chung et al., 2022) and GPT-3 (davinci ver- sion), using specific prompts for the task's input and completions as the task's output. The model output is either coherence score (as a number) for the coherence scoring task or a yes/no answer for the coherence reason- ing task. Details of the prompts and comple- tions designed for T5 and GPT are given in Appendix G and H respectively. During infer- ence, we generate output the most probable label among the possible labels for each task.
We see in Figure 4 that the lack of improved per- formance of the LLMs compared to their smaller versions even more noticeable this case than in the coherence scoring task.
The models also demonstrate relatively good pre- cision but low recall for cohesion features. Further- more, most models struggle to capture the concept of relevance, as indicated by their insufficient per- formance in relevance reasoning.
Apart from that, the models struggle under- stand the reasons behind incoherence, especially relevance reasons. This highlights a common issuc in generational models, where generated data may lack overall coherence even if the model perceives coherence at the sentence level.
Evaluation We evaluated the models' results us- ing the standard precision, recall, and F1 score on the global score for coherence scoring task well the outcome of each individual classifier in the coherence reasoning task.
Notably, in contrast to the coherence score task. GPT-3 performs significantly better in recognizing the reasons for incoherence and especially the rel- evance reasons, which could explain its strength as a generational model at the sentence level and drawbacks in longer text.
'Due to computational constraints, and for GPT-3 also the cost of the API, we could not use gross-validation

---

Docal
05:00
300-TA
FUM JP2C
XQBEX
205
S42
5521
Hald
Figure 3: Coherence Score comparisons single point represents the same performance)
Metrie (%)
P
R
F1
Metrie (%)
P
R
FI
Holistic
24
24
24
coherence
75
70
72
Incremental
23
23
cohesion
72
consistency
60
68
62
Table 6: Coherence scoring with Zero shot setting
relevance
56
75
59
Table 7: Zero shot on GPT-3.5: Results on coherence
These results further highlight the dissonance there between the success in capturing sentence level coherence features compared paragraph level coherence score.
detection and reason detection
Related Work
Coherence, a critical property of language, is not new to NLP research. In particular, it is relevant to any language generation task. Previous stud- ies in language generation developed metrics such as BLEU (Papineni et al., 2002) and MoverScore (Zhao al., 2019) measure the lexical overlap or semantic entailment between generated samples and references. However, these metrics correlate poorly with human judgments.
6.2 Zero-Shot Experiments
Since the data was generated using GPT-3 and in light of the success of this model in generating text we checked the tasks in zero shot setting with the same prompt described in the previous sections GPT-3.5 model.
6.2.1 The Coherence Scoring Task
The evaluation in Table shows the zero-shot co- herence assessment results for each method. reveals that despite the text being evaluated on GPT- 3.5 which is considered to be better than GPT-3, its ability to score coherence is very low. Furthermore, in contrast to the results obtained when fine-tuning LLMs, the holistic score performs similarly to the incremental method in this case.
To date, coherence research has mostly relied the sentence reordering task, assuming that well-formed and coherent text is achieved by main- taining the original sentence order. A coherence model expected to reconstruct the most coherent order of sentences when presented with a random permutation. The advantage of this task its un- supervised nature, allowing for cheap and rapid dataset construction.
6.2.2 Sentence-Level Coherence Reasoning Table presents the results of the zero-shot experi- ments on the coherence reasoning task. It demon- strates that while GPT performs poorly on coher- ence scoring of the entire paragraph, it a lot better in identifying sentence-level coherence even in zero-shot settings. On recognizing the incoher- ence reasons, the model performs better than most fine-tuned models. This further strengthens our conclusion that GPT already has the capability to assess coherence sentence level, but struggles when comes to handling coherence in longer, global contexts.
Existing datasets for the sentence ordering task consist of professionally written and extensively edited texts, including datasets like Accidents and Earthquakes (Burzilay and Lee, 2004), the Wall Street Journal (Elsner and Charniak, 2008), and Wikipedia (Li and Jurafsky, 2017). However, while model trained on the reordering task captures relevant features related coherence, it is not guar- anteed have high correlation with coherence.
Another task, automated essay scoring, involves assessing the quality of an essay and grading it (Feng et al., 2014; Somasundaran et al., 2014;

---

Docal
8
SAB
FLAI
EDI
POECTA
1002
5257
Figure 4: Sentence level results (a single point represents the same performance)
Burstein et al., 2013). This line of work extends beyond sentence ordering to evaluate the properties low-coherence texts, but has mainly focused on test-taker essays. One notable dataset in this domain is the GCDC dataset (Lai and Tetreault, 2018), but it focuses naturally occurring rather than automatically-gennet content.
protocol may be done with diverse models and more diverse prompts.
Our proposed taxonomy of reasons for (in)coherence, based on (Reinhart, 1980)'s work, also may not cover errors made by text generation models, as focus on coherence in human language. Moreover, the results of the LLMs we use, and the analyses we can provide, are English specific, and thus may not carry over to other languages.
8 Conclusion
This work addresses the foundation of automatic assessment of discourse coherence, by presenting a new benchmark, COHESENTIA, containing human- rated generated stories and presented alongside (axonomy of possible
Moreover, when inconsistency reasons occur, the notion of 'what is the status of the world' to is dif- ferent for different people. Our specific raters pool could conceivably have biases in the respect that affect the benchmark and the fine-tuned models. In addition, the benchmark composed of short stories, and while this is a good first step, gener- ation and coherence detection of longer texts is subsequently called for.
a linguistccally reasons for incoherence. We further present two annotation protocols, holistic versus incremental. Through the incremental method, get better inter-annotator agreement, as well as valuable in- sights into the fundamental components that con- tribute to coherence and key challenges of gener- ated text. Consequently, we are better equipped to explore more advanced methods for generating texts in more refined manner, ultimately leading to more coherent generated texts. We make this benchmark, annotation guidelines, and code, pub- licly available, encourage future work on the topic of automatic coherence 3
Ethics Statement
It is important to note that in the generation pro- cess, the GPT input was the title of the story and while the titles were chosen in manner that is discriminatory or harmful in any way, the gen- erated stories were not checked to make sure it is the case. Furthermore, since the MTurk anno- tators were chosen from the workers pool with- out knowledge about their background or belief system, it is possible that the aggregation of the annotations suffers from the fact that different an- notators might precise logical/semantic consistency and pragmatic relevance differently (given that their common knowledge and world knowledge differ, not only different between the annotators but also for a single annotator overtime).
Limitations
Although we view this work as an important step towards better understanding of the notion of co- herence and better automatic evaluation of coher- ence of artificially generated texts, we acknowledge there a lot more to be done. In this work, we col- lect annotations and analyze coherence errors only in stories generated from GPT-3, with relatively simple prompt. Follow up experiments using our
Phttps: wwt..//waaa//ooeeeeeeeeooo

---

Acknowledgements
for Computational Linguisties: Human Language Technologies: Short Papers, 2(11): 129.
We thank Valentina Pyatkin as well as the audience of the NLP-BIU seminar for thoughtful comments and discussion of this work. This work was funded by the European Research Council (ERC-SIG grant number 677352), the Ministry of Science and Tech- nology (MOST grant number 3-17992) and the Israeli Innovation Authority (IIA, KAMIN grant), for which we are grateful.
Vanessa Wei Feng, Ziheng Lin. and Graeme Hirst. 2014. The impact of deep hierarchical discourse structures in the evaluation of text coherence. Pro- ceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, 1:940-949.
Herbert P Grice. 1975. Logic and conversation. Speech acts, pages 41-58.
Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics. 11(2::6--1.227-8.22
References
Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Compu- tational Linguistics. 34(1):1-34.
Pengcheng He. Jianfeng Gao, and Weizhu Chen. 2023, DeBERTav3: Improving deBERTa using ELECTRA- style pre-traiming with gradiiet-disenttane ding sharing. In The Eleventh International Confer- ence Learning Representations.
embed-
Regina Barzilay and Lillian Lee. 2004, Catching the drift: Probabilistic content models, with applications generation and summarization. HLT- NAACL
2004: Main Proceedings. 120.
Klaus Krippendorff. 2004. Content Analysis: An In- troduction to Its Methodology. Sage Publications. Beverly Hills, CA: Sage.
Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lu, Martin Chodorow, Lisa Braden-Harder, and Mary Dee Harris. 1998. Automated scoring using a hybrid feature identification technique. Proceedings of the 36th Annual Meeting of the Association for Computational Linguisties and 17th International Conference on Computational Linguistics, 1:206 210.
Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: A dataset, evaluation and methods. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. Association for Computa- tional Linguistics, 1:214-223
Jiwei Li and Dan Jurafsky. 2017. Neural net models of open-domain discourse coherence. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Com- putcational Linguistics, 209.
Jill Burstein, Joel Tetreault, and Martin Chodorow. 2013. Lholistic discourse coherence annotation for noisy essay writing. Dialogue & Discourse, 4(2):34-52.
Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling ciiructioo-finetunne language mod- els.
Ziheng Lin, Hwee Tou Ng. and Min-Yes Kan. 2011, Automatically evaluating text coherence using dis- course relations. Proceedings of the 49th Annual Meeting of the Association for Computational Lin- guistics: Human Language Technologies. Associa- tion for Computational Linguistics. 1:997-1006.
Annie Louis and Ani Nenkova. 2012. A coherence model based on syntactic patterns. Proceedings of the 2012 Joint Conference on Empirical Meth- ods in Natural Language Processing and Computa- tional Natural Language Learning, EMNLP-CONLL, Stroudsburg. PA, USA. Association for Computa- tional Linguistics, 1:1157-1168.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, 1:4171-4186
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward functional the- ory of text organization. Text Interdisciplinary Jour- nal for the Study of Discourse, 1:243-281.
Micha Elsner and Eugene Charniak. 2008. Coreference- inspired coherence modeling. Proceedings of ACL- 08: HLT. Short Papers. Association for Computa- tional Linguistics, 1:41-44.
Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The Penn Discourse Tree- bank. In Proceedings of the Fourth International Conference on Language Resources and Evaluation 04), Lisbon, Portugal. European Language Resources Association (ELRA).
Micha Elsner and Bugene Charniak. 2011. Extending the entity grid with aatity-specifie features. Proceed- ings of the 49th Annual Meeting of the Association

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. Proceedings of the 40th anmual meeting of the Association for Compu- lational Linguistics, 1:311-318.
Emily Pitler and Ani Nenkova. 2008. Revisiting read- ability: A unified framework for predicting text qual- ity. Proceedings of the 2008 Conference on Empiri- cal Methods in Natural Language Processing, pages 186-195, Honolulu, Hawaii. Association for Compu- tational Linguistics. 1:186-195.
Tanya Reinhart. 1980. Conditions for text coherence. Poetics Today 1(4): 16t-180, volume 1.
Swapna Somasundaran, Jill Burstein, and Martin Chodorow. 2014. Lexical chaining for measuring discourse coherence quality test-taker essays. Pro- ceedings of COLING 2014, the 25th International Conference Computational Linguistics: Technical Papers, 1:950 961.
Radu Soricut and Daniel Marcu. 2006. Discourse gen- eration using utility-trained coherence models. Pro- ceedings of the COLING/ACL 2006 Main Conference Poster Sessions. Association for Computational Lin- guistics, Sydney, Australia, 1:803-810,
Wei Zhao. Maxime Peyrard, Fei Liu, Yang Gao, Chris tian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized em- beddings and earth mover distance. Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP) 1:563-578.

---

A Coherence Violations
Here some examples for coherence violations for each one Reinhart conditions.
Reason
Cohesion
Entity Reference
"John hid Bill's car keys. was drunk."
Discourse connector
"Dan is a tough guy. So, was easy for him to cry at the movie."
Consistency
previous data knowledge of the world
"She just went home. She is here playing games."
"My father is dead now. That's why he has decided to smoke pipe."
Relevance
Irrelevance
"I poured some chemical into beaker. Since I poured slowly, the chemical fell on my hand. The professor immediately book me to the emergency bath. was shaking from the hot water. I thankfully came without any injuries. Then, T learned he would imitate my whistles."
Table 8: Examples for incoherent sentences for each reason based on Reinhart linguistic theory
B
Holistic Method UI
Here a figure of the UI used for the holistic method for acquiring coherence score only.
"Father Son"
Paragraph:
When father worked, he would came home and tell his son about what he had done during the day, Son would listen patiently, but secretly hoped that when Father came often held give
Sometimes was tough going through life with just one parent around: however, as long as Father waen't unaveilable, Sons should be happy just to have him in their lives, One bright autumn morning Son woke this year they
angry
there entre house! they finished their chorekin even started piring and Baby John todeled into the room scampering behind his old Brookdyn scarf Papaclly said "Behave yourself now Baby Fagscrunches
because this
sweeping
Jahn stopped darting Fathers Instruction etc muttered under breath until Daddy got back from work After dineeCrowcspurzed Perenting Tips For New Perents joint responsibiiities can dlear Saturcayshow all calm first Papaelly supervised me Homeraade each way w/o including Freezing On The Stairs does anyone have anyoxperiences?
pretanding
Soup2 FREEZERS
Methods
How coherent the story?
Please, the silder below this question (0 definitlly not coherent;; 100 definitely coherent).
Not
Na
Maybu yea, maybe no
Yoa
Dafmnall
100
Your is: 51
Figure& 5: An example to annotation page for the holistic method
Incremental Method UI
Here is a figure the UI used for the incremental method for acquiring coherence score with reasons for (in)coherence per sentence in each paragraph.

---

Title: "Father and
Paragraph:
(There are 6 sentences)
1. When father worked, he would come home and tell his son about what he had done during the day.
Is the sentence coherent?
No
(a) first sentence
"Father
Paragraph: (There sentences)
worked,
son about during day.
Son would listen patiently, but secretly hoped that when Father came home tired or angry - which was often he'd give him good beating.
Is the last sentence coherent with all previous sentences?
(b) second sentence
"Father and Son"
Paragraph: (Thern
would Heten patiently, excretty that when Father came tired which held him good beating.
Is the last sentence coherent with all prestous sentences?
sertence dozan't make Jasse
The new sentence discusses ertty which has not been property incroduced in the previous centenoes
sentence cornains information incorsiitett with previous date promected story knowladge about world
pentence contann information incorandent
The new sentenic not refevent to previous ditt in the story
Comment teat.
(c) reasons for second senience
When
licad
horee firet any which wes oiten lecd give tiee good booting, tenadiics non lough gurng through ihe with jost uee eroudd foncects do hes Pelk
snvers eut of them
they cuerping matoo knourd wan cuher hai lin siarte and fubty ti todid tao
scarpenng savied sid arapdyy scaff rassedy "aaaanc
cat's cappoct daring tonan aggecrunclee snnch aga meeciingg casr sway sanurcaution is cain first Popsally expervise the servaross Soupa sia Moslzdio wared each way all ne sied out Monnis uic. scing freecing On Stare does ampons have styoppeteros?
Do soile, Fauk camanent
elary!
Peees, move kide peinee anoivar thes queetion (D Bally nos cohernet 200 oetrrially comverent
Ho cheerrt
Hok heneee
Moboosblee coheror
300
World like communt?
(d) final sereen after all sentences annotation
Figure 6: An example to annotation page for the incremental method
In (a) is the start screen of the UT, after filling the coherence detection on the bold sentence (in this example the sentence is coherent), the UI moves to (b) the next sentence where the annotator is required to annotate the coherence of the next sentence in regards to what he read thus far. this example, the

---

annotator chose it is not coherent, and then (c) he needs decide what are the reasons for the incoherence. After going this way over all sentences in the paragraph (d) is presented and the annotator required decide the final coherence score of the entire paragraph. change his answer only for the current sentence and can't 't go back. This was done so that he wouldn't 't be able to change.
important to note, that the annotator can
Annotation Output
Here is an example of annotation output:
Annotator
Amnotaorr
Title: The Untorgivnng Road
The unforgiving road led to nowhere,
2 long and winding path that forover.
3 waB full potholee and bumps, and was always covered in dust.
4 The sun never shone on the unforghving road, end the wind elways blew il away.
X
P33
R3
R33
5 He also felt angyy all The time with his friend and just wanted to move pest his anger
X
F77
R6,
R7
R7
133 The row sanie diec mere willyy wcich noi boen proparty priodd and Ihe precice verfercwe
Final
Final
93 The ascat or newwar
Figure 7: An example of final annotation for incremental method
E
Agreement
Additional agreement values are presented here: Because of the importance recognizing what is the first sentence that is not coherent in story, we checked the agreement (Table 9) in detecting the first incoherent sentence a paragraph.
Group
ICC
AC
Krip. a
All data
0.94
0.84
0.90
with incoherence
0.99
0.81
0.93
Table Inter annotator agreement on incoherence detection out stories and out all stories that contain at least one incoherence sentence
In addition, the agreement on a number of incoherent sentences for each paragraph and the agreement on the number of incoherent sentences from each group type is presented in Table 10.
Group
ICC
K
Krip. a
Incoherence
0.98
0.76
0.92
in-cohesion
0.98
0.72
0.90
in-consistency
0.98
0.73
0.84
irrelevance 0.92
0.70
0.77
Table 10: Inter annotator agreement number of incoherence sentences and number of reasons paragraph
Lastly, we presented the paper the agreement among annotators in the sentence-level of the reasons for incoherence clustered into groups. In Table 11 is the agreement among annotators per reason. As can seen, the agreement is much lower per reason. The reason IDx are based on the list in Sect.

---

Reason
ICC
Krip.
rl
0.95
0.76
0.83
r2
0.87
0.63
0.66
r3
0.81
0.55
0.60
r4
0.60
0.74
0.80
r5
0.72
0.65
0.73
r6 0.78
0.53
0.63
r7
0.73
0.53
0.62
Table 11: Inter annotator agreement on incoherence reason
F Reasons Per Score
Based on the Table 12 you can for example that for score (low coherence score), the chance that there is at least in-cohesive sentence 98% while for score (high coherence score) 9%. For relevance, however, for score there is 41 % that least one irrelevant sentence will appear, and for score just 2%.
number sentences
Cohesion
Consistency
Relevance
2
1
2
3
4
5
2
3
4
5
1
11
24
50
85
58
26
38
44
78
35
13
25
31
23
12
2
20
28
29
37
17
18
34
23
2
14
14
16
12
2
3
18
22
34
13
16
13
3
0
3
3
1
0
4
10
16
3
9
5
0
0
0
0
5
18
4
5
1
0
0
0
>5
26
12
2
1
1
0
0
ratio (%)
98
92
84
72
54
67
64
58
5
41
38
33
20
2
Table Number paragraphs containing number incoherent sentences the same group per score. the last row is the ratio between the number of paragraphs with the number of incoherence reasons and the total number of paragraphs
Additionally, there is notable trend where sentence is deemed incoherent because of in- cohesiveness, it is likely that other incoherent sentences will be incoherent because of cohesive error. As can be shown in Figure 8 in most of the stories where there are at least two incoherent sentences with one of them being in-cohesive the rest of the sentences will be in-cohesive well.
Irrelevant sentences, however, does not act this way and it is possible that irrelevant sentence will appear number of times regardless of number of incoherent sentences.
T5 Settings
For fine tuning T5 model for both tasks prompt and completion were constructed. P represent the story text, T' is the story title and C is the required output. The task's input put with a prefix based the task:
Coherence Scoring Task:
"coherence score: title: <T> paragraph: <P>"
Sentence Level (In)Coherence and Reasons Detection Task:
For incoherence detection:
"incoherence detection: previous data: <d,> new sentence: <8;>"

---

160
Cohesion
140
Consistency
Relevance
100
80
60
Ratio
Figure 8: Histogram number stories with ratio of amount of innohssieeiicooniseetummmbaaaaa out of incoherent sentences in the story (normalised with the stories with least nccoessie/nccossseen//meeeaattt respectively)
For reasons:
<ロ reason: previous data: <d,> new sentence: <s,>"
Where is "cohesion", " "consistent" or "relevance" for the different group reason.
The task output is 5 multi-clas classification output and binary classification task for the first and second tasks respectively.
GPT-3 Settings
For fine tuning model for both tasks a prompt and completion constructed. P represent the story text, T is the story title and C is the required output.
Coherence Scoring Task:
"coherence score: title: <T> paragraph: <P> Task: Give just discrete coherence score between ] to 5 for the paragraph as a number not coherent, 5 fully coherent)?"
Sentence Level (In)Coherence and Reasons Detection Task: For incoherence detection:
"incoherence detection: previous data: <d,> sentence: <s,> Task: Is the sentence coherence in regards the previous data? give yes or no answer"
For reasons:
<イン>>
reason detection: previous data: <d,> new sentence: <si> Task: Is the new sentence <gj> in regards to the previous data? give or no answer"
Where 9j is "cohesion". "consistent" "relevance" for the different group reason.
The task output is number for the first task and yes/no completion for the (In)Coherence Reasons Task.


Reviews:
[{"review_id": "QJwXBDRn4T", "reviewer": "Reviewer_1hwA", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review, as well as the clear and accurate summary that implies that our submission was well-understood. \nThe fact that you found our benchmark “challenging enough to foster future work in enabling LLMs to handle and interpret coherence at a more convincing Level” is exactly what we aimed for in constructing this dataset. \n\nWe acknowledge your  suggested improvements related to typos and presentation and we will make sure to address them in the camera ready version.\n\nThanks again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, the methodology strong, and the gathered annotation provides a valuable tool for measuring the ability of automatic systems to model coherence.\n\n", "reasons_to_reject": "No reasons to reject", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper \"COHESENTIA: A Novel Benchmark of Incremental versus Holistic Assessment of Coherence in Generated Texts\" introduces a dataset meant as a benchmark to asses coherence in generated texts. It discusses the linguistic aspects of coherence, where authors show a deep theoretical background, and presents two alternative and parallel annotation methods, holistic and incremental, showing that the latter produces higher agreement among annotators. \n\n\nThe paper is complemented with a set of experiments, organised into two subtasks (i.e. Coherence Scoring and Coehrence Reasoning), which show that the COHESENTIA benchmark is challenging enough to foster future work in enabling LLMs to handle and interpret coherence at a more convincing level. \n\n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\nI have just some comments:\n- move all refenrences into brackets\n- neutral pronoun at 310 instead of \"she\"\n- caption for figure 8 needs rephrasing\n- few typos: (\"otherwose\"\" at 045, \"this\" at 554, \"(i)\" at 496)"}}, {"review_id": "KNyiVeKRE8", "reviewer": "Reviewer_8yo1", "summary": "", "questions": "[Q1] LM are basically probability distributions over text sequences: as such, LMs are unfit to deal with logical reasoning or matters such as coherence. In this sense, one is not surprised for the poor perfomance attained in the evaluation Section. One would thus be curious to know how do authors expect that whatever LM trained on their data this may generalize to previously unseen stories. How a device such as a LM is expected to help in higher-order tasks, such as deciding whether 'The relation between this sentence and previous ones doesn’t make sense'? \n\n[Q2] Putting at the heart of the system a closed resource such as GPT* may be detrimental to investigate inner mechanisms and implementation details. Are the Authors willing to elaborate on these issues?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort, we appreciate your thorough evaluation of our paper. \nWe are pleased to learn that you find our paper interesting and fundamental for understanding LMs’ generative aspects.\n\nRegarding your questions: \n1. Re: “One would thus be curious to know how do authors expect that whatever LM trained on their data this may generalize to previously unseen stories. How a device such as a LM is expected to help in higher-order tasks, such as deciding whether 'The relation between this sentence and previous ones doesn’t make sense'?”\nAnswer:\nYou are right, LMs, being probabilistic, don’t focus the training on logical reasoning. That said, LLMs are frequently used for tasks such as generation and summarization, which entails production of longer texts that are at least in principle required to be inherently coherent. It has also been shown that LLMs, trained extensively, exhibit some logical reasoning. For example, the paper by Honovich et al 2021 (https://arxiv.org/pdf/2104.08202.pdf) shows how fine-tuning LLMs using NLI tasks improves their ability in capturing consistency.\nHence, our benchmark facilitates the evaluation of generation-based models, which is vital for high-quality long generated texts. This paves the way for potential applications such as identifying incoherent sections for enhanced text generation or serving as a tool for human-in-the-loop editing. \n\n\n2. Re: “Putting at the heart of the system a closed resource such as GPT* may be detrimental to investigate inner mechanisms and implementation details. Are the Authors willing to elaborate on these issues?”\nAnswer:\nThe fact that GPT* is a closed resource is part of the challenge that this dataset is in fact trying to mitigate. Specifically, while we can't access its internal details, the dataset aims to provide a methodological way to probe its resulting generated text for human-like discourse coherence. Crucially, we make this dataset and accompanying annotations available for coherence assessment of text generated by *any* model, and not necessarily by GPT*.\n\nWe wish to kindly ask, in what sense you found the work incremental? Our purpose was to create a theoretical foundation (Reinhart) and a first-of-a-kind resource that allows to benchmark models for automatic coherence detection. The models we show are proof of the usability of this resource - this is a starting point for the community to improve upon, on this important topic.\nIf there are particular elements that you find insufficient, we'd very much like to improve on that.\n\nWe thank and acknowledge your comments on typos, grammar and other notes, which we will make sure to address in the camera ready version.\nWe hope we addressed all of your concerns to your satisfaction. \n\nThank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes an interesting work on a relevant issue, that is explicitly assessing the coherence of generated texts. This is a relevant theme to NLP, since it lies at the base of our understanding of LMs and their generative features.\n", "reasons_to_reject": "- The computational approaches employed in the experiments seem to require some further (higher-level) mechanism to deal with the task at hand: or, equivalently, the task might not be appropriate to be dealt with Language Models.", "questions_to_authors": "[Q1] LM are basically probability distributions over text sequences: as such, LMs are unfit to deal with logical reasoning or matters such as coherence. In this sense, one is not surprised for the poor perfomance attained in the evaluation Section. One would thus be curious to know how do authors expect that whatever LM trained on their data this may generalize to previously unseen stories. How a device such as a LM is expected to help in higher-order tasks, such as deciding whether 'The relation between this sentence and previous ones doesn’t make sense'? \n\n[Q2] Putting at the heart of the system a closed resource such as GPT* may be detrimental to investigate inner mechanisms and implementation details. Are the Authors willing to elaborate on these issues?", "paper_topic_and_main_contributions": "This paper proposes an annotated resource, COHESENTIA, concerned with explicitly assessing the coherence of generated texts and with the analysis of incoherence and its underlying factors. Texts in the benchmark were annotated in two ways: a holistic assessment (an overall evaluation scored in 0-100), and an incremental assessment, where human raters considered text sentence by sentence, in incremental fashion. In this setting, also the reasons at the base of incoherence judgements were annotated (based on literature, namely from Reinhart's conditions), which may provide helpful information to learning systems. Comparison between the two annotation schemes shows that the IAA is higher in the incremental setting. The work proposes an experimental section, the Authors fine-tuned LMs to detect incoherence. \n\nObtained accuracy is not fully satisfying (\"Coherence scoring contains intricate features that are much harder for a model to capture\"); another outcome is that \"Larger LMs exhibit only small improvements compared to their smaller counterparts, suggesting that both models have limitations in comprehending and identifying coherence\". \n\nAlso considering the reported experimental results I have a question, and I'd solicit Authors to elaborate on this point. LM are basically probability distributions over text sequences: as such, LMs are unfit to deal with logical reasoning or matters such as coherence. In this sense, one is not surprised for the poor perfomance attained in the evaluation Section. One would thus be curious to know how do authors expect that whatever LM trained on their data this may generalize to previously unseen stories. How a device such as a LM is expected to help in higher-order tasks, such as deciding whether 'The relation between this sentence and previous ones doesn’t make sense'? \nPutting at the heart of the system a closed resource such as GPT* may be detrimental to investigate inner mechanisms and implementation details. Are the Authors willing to elaborate on these issues?\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Minor issues & Typos\n\n- References should be corrected: please check a guide on text references, such as https://cljournal.org/style_guide_refs.html\n- Figure 1 is too tiny, nearly unreadable, should anybody ever try to print the paper...\n- line 424 \"In Figure 2 **you** can see that once a cohesion error appears\" -> in scientific literature it is rather unusual that Authors directly refer to their readership; impersonal forms should be preferred -> \"it can be seen\" or even 'we can see' would be acceptable\n- line 496: 'concatinated' -> 'concatenated'\n- line 657: 'generatio' -> 'generation'\n- line 825 broken reference: Sect. ??\n- Figure 4 is too tiny, nearly unreadable\n\n\n"}}, {"review_id": "hgoqmWCKRl", "reviewer": "Reviewer_8DsP", "summary": "", "questions": "1. Will the dataset be released? I could not find any related statement.\n2. In the Experiments section, is there a rationale for not using a tenfold cross-validation? I can imagine that this is due to the actual cost of the API and/or computational reasons. If so, it might be helpful for the reader to make the rationale behind the split decision explicit.\n3. In both Figure 3 and Figure 4, GPT is associated with a single point. Is this because the performance is the same (holistic vs incremental in Figure 3 and coherent vs incoherent for Figure 4)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive feedback and we are glad you found the benchmark one “that could be easily plugged in a framework designed to assess LLM” as this is an important part of the role of these data we envision in the field. \n\nRegarding your questions:\n1. Re: “Will the dataset be released?“ \nAnswer: Of course, the dataset along with all the necessary materials required for reproducibility will be made openly available once the paper is published. In the camera ready version, a link containing the dataset as well as the full code will be added to the paper. (We provide the anonymized version of these materials in the supplementary material of this submission). \n\n\n2. Re: “In the Experiments section, is there a rationale for not using a ten-fold cross-validation? I can imagine that this is due to the actual cost of the API and/or computational reasons.”\nAnswer: Yes, you are right. The reasons are the cost of the API and computational constraints. We will explicitly clarify the rationale in the camera ready version. \n\n\n3. Re: “ In both Figure 3 and Figure 4, GPT is associated with a single point. Is this because the performance is the same?”\nAnswer: Yes, a single point does indicate the same results. We will clarify it in the caption.\n\nWe will of course make changes in the paper regarding your style and presentation improvements notes.  \nThank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is interesting, the paper is well-written and easy to follow. The paper is self-contained. The authors provide clear motivations underlying the work. The paper introduces a novel benchmark that could be easily plugged in a framework designed to assess Large Language models so as to provide linguistically grounded intuitions on how such models  understand text coherence.", "reasons_to_reject": "I feel no reason to reject the paper.", "questions_to_authors": "1. Will the dataset be released? I could not find any related statement.\n2. In the Experiments section, is there a rationale for not using a tenfold cross-validation? I can imagine that this is due to the actual cost of the API and/or computational reasons. If so, it might be helpful for the reader to make the rationale behind the split decision explicit.\n3. In both Figure 3 and Figure 4, GPT is associated with a single point. Is this because the performance is the same (holistic vs incremental in Figure 3 and coherent vs incoherent for Figure 4)?", "paper_topic_and_main_contributions": "The paper presents a novel benchmark for assessing coherence in automatically generated texts. The authors, together with the novel dataset, introduced a two-fold labelling schema: the holistic setting, where each text is featured with a global coherence score (i.e. considering the text as a whole), and the incremental setting, where each sentence of the paragraph is provided with a coherence score together with the reasons for incoherence of the fragment. The benchmark consists of 500 automatically generated texts annotated by humans through the Amazon Mechanical Turk system.\nThe annotators were requested to provide coherence scores for both experimental settings: in the holistic protocol the reader is asked to assess the coherence of the whole text by providing a score in the range 0-100 while in the incremental protocol the rater decides whether the current sentence is coherent individually and with regards to the previously presented sentences; if the annotator deems a sentence incoherent, a list of reasons is presented and the rater is required to choose the reasons for the detected incoherence at that particular point.\nFinally, the authors assessed two different types of language models on the novel benchmark: encoder models provided with a classification head (BERT and DeBERTa), and generative models (Flan T5 and GPT3).\nThe results point out that results on the incremental setting are higher than those obtained on the more classical holistic setting. Additionally, the performance of large language models seem to be only a small improvement over the lower-sized models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please find some minor comments below:\n- please consider rewriting she on line 321 as she/he;\n- please consider rewriting appendix on line 321 as Appendix;\n- please consider rewriting cohen k on line 372 as Cohen k;\n- In Figure 4, should the last model be GPT3?\n- please consider rewriting si on lines 845, 847, 859 and 862 as s_i."}}]

Author rebuttal:

Meta-review: The paper presents a benchmark to assess human-perceived coherence (through crowd annotators on Amazon Mechanical Turk) in generated texts, which contains 500 automatically generated texts. The authors  discuss linguistic aspects of coherence including its deep theoretical background. They present two alternative annotation methods – holistic – the annotators were asked to provide a global coherence score for the whole text, and incremental – the annotators scored each sentence also providing a reason for the detected incoherence cases. The results show that the incremental method guarantees higher agreement among annotators. 
Apart from the annotation procedures, the authors complemented the paper with further experiments organised into two subtasks  - Coherence scoring and coherence reasoning evaluating different language models using the resulting benchmark - encoder models (BERT and DeBERTa) and generative models (Flan T5 and GPT3). Here, the results were also better for the incremental setting. The results also show that performance of large language models seem to be only a small improvement over the lower-sized models.
Strong sides (pros): (1) definitely, an interesting topic for the community; (2) novel benchmark for coherence assessment which can be used in further studies, and also for texts generated with any model; (3) the presented methodology is sound; (4) the paper is clearly and well-written, it is self-contained.
I would not say that there are cons, but rather some flaws in the paper. They were all pointed in reviewers’ comments and include typos, formatting, etc.
Decision: Accept-Main