A Multi-Task Dataset for Assessing Discourse Coherence in Chinese Essays: Structure, Theme, and Logic Analysis
Authors: Hongyi Wu
Xinshu Shen
Man Lan
Shaoguang Mao
Xiaopeng Bai
Yuanbin Wu
Venue: EMNLP 2023 Main (2023)
Keywords: discourse coherence assessment
TL;DR: We present a high quality dataset for the analysis of discourse coherence.

Abstract: This paper introduces the \textbf{C}hinese \textbf{E}ssay \textbf{D}iscourse \textbf{C}oherence \textbf{C}orpus (\textbf{CEDCC}), a multi-task dataset for assessing discourse coherence. Existing research tends to focus on isolated dimensions of discourse coherence, a gap which the CEDCC addresses by integrating coherence grading, topical continuity, and discourse relations. This approach, alongside detailed annotations, captures the subtleties of real-world texts and stimulates progress in Chinese discourse coherence analysis. Our contributions include the development of the CEDCC, the establishment of baselines for further research, and the demonstration of the impact of coherence on discourse relation recognition and automated essay scoring. The dataset and related codes is available at \url{https://github.com/cubenlp/CEDCC_corpus}.

A Multi-Task Corpus for Assessing Discourse Coherence in Chinese Essays: Structure, Theme, and Logic Analysis
Hongyi Wu', Xinshu Shen', Man L2, 12, Yuanbin Wu', Xiaopeng Bai", Shaoguang Mao "Scooo of Computer Science and Technology, East China Normal University, Shanghai, China
*Shanghai Institute of AI for Education, East China Normal University. Shanghai, China
3 DDepartment of Chinese Language and Literature, East China Normal University, Shanghai, China Microsoft Research Asia, Beijing, China
(hongyiwu, xinnuussnnnhuhheeees testu ecnu, edu. cn, {mlan) ybwu) ecnu. ecnu.edu.co cn bobai@zzwx.co ecnu. edu. cn, shaoguang mawemi cwwpoft.rrrooootcomommmmoo com
Abstract
Each of these elements highlights a distinct as- pect of discourse coherence. Specifically, for co- herence grading, Lai and Tetrcault (2018) assert that inherent coherence is fundamental feature of human-writtee texts, crucial for the global compre- hension of an essay and its quality assessment. In terms of topical continuity, Amoualian et al. (2017) argue that a coherent text should maintain log- ical progression of ideas, eschewing abrupt topic transitions. They primarily utilize topic models extract and designate topics within a text, thereby quantifying topical coherence. Furthermore, Feng et al. (2014) suggest that text coherence is closely tied to its discourse structure and relations.
This paper introduces the Chinese Essay Discourse Coherence Corpus (CEDCC), multi-task dataset for assessing discourse co- herence. Existing research tends to focus on isolated dimensions of discourse coherence, a gap which the CEDCC addresses by in- tegrating coherence grading, topical continu- ity, and discourse relations. This approach, alongside detailed annotations, captures the subtleties real-world texts and stimulates progress in Chinese discourse coherence analy- sis. Our contributions include the development of the CEDCC, the establishment of baselines for further research, and the demonstration of the impact of coherence on discourse relation recognition and automated essay scoring. The dataset and related codes is available at https: //gi thub. com/ cuwenlp/CEDCC..C....//// corpus..
Despite the multifaceted nature of discourse co- herence, current research often examines isolated dimensions, overlooking their interplay. For exam- ple, most coherence grading studies (Shen et al., 2021; Flansmose Mikkelsen al., 2022) often perceive coherence as an automatic scoring task, neglecting aspects like topical continuity and dis- course relations, which impairs the interpretability of results. Although some studies aim to integrate these dimensions, progress has been hindered by the lack of a comprehensive dataset. For instance, methods proposed by Shrivastava al. (2018) and Lin al. (2011) operate in unsupervised arti- ficially structured contexts, but these fail to cap- ture the complexity of real-world texts. Further- more, the scarcity and limited diversity of Chinese discourse coherence datasets have somewhat con- strained advancements in this field.
1 Introduction
Discourse coherence, fundamental aspect of both language comprehension and generation, involves understanding how words, phrases, sentences, and paragraphs within a text are interconnected to cre- ate a cohesive and coherent message. Due to its sig- nificance, it has been widely incorporated into vari- ous natural language processing (NLP) tasks, such automated essay scoring (Farag et al., 2018), machine translation (Xiong al., 2019) and text generation (Guan al., 2021).
Existing discourse coherence analyses present various frameworks for understanding coherence realization, most of which can be grouped into three categories: (a) Overall Coherence Grading, which assigns a coherence score to given text, based on its overall logical flow and consistency; (b) Topical Continuity Modelling, which focuses on analysing the distribution and relationship of topies within discourses; and (c) Discourse Rela- tionship Recognition, which involves detecting semantic relations between two segments of a text.
To address the shortcomings of existing re- search, we introduce the Chinese Essay Discourse Coherence Corpus (CEDCC), a multi-task dataset designed to assess essay discourse coherence. The CEDCC, sourced from various middle schools, en- compasses diverse range of topics, genres, and regions. Each essay has been meticulously anno- tated by linguistic experts, as depicted in Figure I. The CEDCC addresses key limitations in prior

---

Chinese Essay (Excerpt)
(b) English Translation (Excerpt)
AP
The Spirit of Perseverance
Karl Marx once said, "Life like the oceen: only those with strong will can reach the other shore 'Bex Therefore, individuale who possees the spirit pereeverance those who can handle significant challenges, withstend etorne, and endure hardehips (deer [Omitted] They have no silver that can used for tribery ee. th. The oceen is vast because it admits all rivers. A wall stands a thousand feet high because yeelds to nothing jowa tt (are So, it can be said that the person who constantly self- improving, just they can rise against the wind the storm, these people have firm belief, relentless pursuit, extraordinery ambition, and unyielding spirit, therefore they are bound succeed pest vat
14 A,
[t:l]
(Bnc 11
06 2 M0 Bor
Bam
7
FLT Ag
A
3
1
10 13y.
(c) Fine-grained Annotation
Discourse Coherence Grade
Topic Sentence
Coherence Grade: (indicating low or poor coherence)
Paragraph Topic Sentences: Sent
and Sent (Highlighted red)
Primary Topic Sentence: Sent
(Marked in bold red
Factors influencing coherence;
Off-topic Section: Sent 12 (Underlined section in bluc)
Discourse Relations
Misused Connective: just (Highlighted in grey)
Inappropriate Clause: (Highlighted in purple box)
Paragraph
Sentence
#
Incoherent Logic: Sent II and Sent 12
(Highlighted in green above and in yellow below)
Para
Para 2
Sent
Sant
subordinate
subjective
An example of the CEDCC annotation: (a) Original Chinese Essay (Excerpt), (b) English Translation (Excerpt), and (c) Fine-grained Annotation about discourse coherence.
Figure 1:
work: firstly, it breaks from the isolationist ap- proach of previous studies. With the integration coherence grading, topical continuity, and discourse relations, it provides more comprehensive understanding of discourse coher- ence. Secondly, the detailed annotations, particu- larly evident in Figure I(c), capture the nuances of real-world texts. Factors such as off-topic sections, misused connectives, inappropriate clauses, and in- coherent logic that influence coherence grade are highlighted. Finally, by offering a diverse dataset for Chinese discourse coherence analysis, CEDCC stimulates progress in this area.
notations for automated essay scoring, encour- aging multi-dimensional discourse analysis.
Related Work
In this section, we delve into three core aspects of discourse coherence. For cach, we discuss its objectives, relevant datasets, and their limitations, establishing the backdrop for our proposed dataset.
2.1 Discourse Coherence Grading
Discourse Coherence Grading (DCG) in NLP mea- sures text coherence through assessing semantic, structural, and logical aspects. Several studies, such as those using the Grammarly Corpus of Dis- course Coherence (GCDC) by Lai and Tetreault (2018), the INSteD dataset for pre-trained language models by Shen et al. (2021), ) and DDisCo dataset comprising Danish texts by Flansmose Mikkelsen et al. (2022), have contributed significantly to this field by developing specific datasets.
Our contributions are summarised as follows:
We develop the CEDCC, a comprehensive multi-task dataset for discourse coherence as- sessment, enhancing understanding of Chi- nese middle school student essays.
We establish baselines for the CEDCC, setting reference point for future discourse coher- ence research.
However, these studies often focus grading overall coherence without pinpointing the specific factors affecting it. Consequently, the derived scores may lack interpretability and offer an in- complete view of discourse coherence. Moreover, most existing resources primarily target English or
Through insightful experiments, we illustrate the impact of coherence on discourse relation recognition and the value of fine- -raiind an-

---

other European languages, leaving a noticeable gap in Chinese discourse coherence analysis.
proposed using DRR for coherence assessment, exemplified by the works of Lin et al. (2011) and Feng et al. (2014), and the investigation into con- current explicit and implicit relations by Rohde et al. (2018).
Our CEDCC dataset aims to address these lim- itations. It predefines common factors impacting discourse coherence, such as off-topic sections, mis- used connectives, inappropriate clauses, and inco- herent logic. Besides providing overall coherence scores, our annotators meticulously annotate these factors, thereby improving the interpretability of the coherence grading. Finally, by focusing on Chi- nese middle school student essays, the CEDCC fills the gap in Chinese discourse coherence evaluation.
However, due to the absence of datasets featur- ing both discourse coherence grading and relations annotation, prior methods often resort to sentence ordering tasks for coherence assessment (Moham- madi et al., 2020). These tasks involve juxtaposing well-structured text with randomly arranged sen- tences, which, despite their usefulness, may not fully encapsulate the intricacies of real-world text coherence. To rectify this limitation, our CEDCC dataset includes both discourse relations annota- tions and coherence scores. This integrated ap- proach offers a more nuanced understanding of how discourse relations contribute to overall text coherence, addressing the deficiencies of previous methods.
2.2 Topical Continuity Modelling
Topical continuity modelling aims to quantify the consistency of topics within a document to ensure a cohesive narrative. Several techniques have been proposed for this task. For instance, Amoualian et al. (2017) introduced an LDA-based model to create topic-coherent segments within documents, while Shrivastava et al. (2018) proposed an unsu- pervised metric to evaluate topic coherence by ana- lyzing latent topic structures. More recently, pre- trained contextualized document embeddings have been used to enhance topic coherence in neural topic models (Bianchi et al., 2021). However, these methods often struggle to capture fine fine-grained topi- cal variations and require substantial computational resources for processing large volumes of text. In contrast, our CEDCC dataset addresses this issue by providing granular, sentence-level annota- tions for both paragraph and overall topics. These detailed annotations capture the nuanced topical variations within documents, adding depth to the task of topical continuity modelling. Furthermore, the focus these finer details reduces the reliance intensive prepprocessing and segmentation tech- niques, which often limit the effectiveness of cur- rent models. Hence, the CEDCC dataset can serve as a valuable tool for advancing research in topical continuity modelling, especially for tasks that re- quire an understanding of finer topical variations within a document.
Corpus Construction
This section delineates the process of collection and annotation for the Chinese Essay Discourse Coherence Corpus (CEDCC), designed for exten- sive discourse coherence analysis.
3.1 Data Collection
For the construction of the CEDCC, we collected 501 essays from secondary school students' exam compositions and daily practice. These essays, ranging from 603 to 1,600 tokens with an average of approximately 713.18 tokens, were meticulously selected based criteria such genre and teacher- assigned scores. As depicted in Figure 2(a), our dataset spans eight distinct genres, and the distri- bution of teacher-provided scores is illustrated in Figure 2(b).
We specifically chose secondary school compo- sitions for their significance in discourse coherence research. These essays offer genuine instances of both coherent and incoherent discourse combined with formal language usage. The teacher-ase aasigne scores further enable correlation between dis- course coherence ratings and the overall essay eval- uation, addressing primary challenge in Chinese automated essay scoring which often struggles integrate discourse coherence due to the absence of appropriate datasets like CEDCC.
2.3 Discourse Relation Recognition
Discourse Relation Recognition (DRR) is a signifi- cant area in NLP that seeks to identify and classify the relationships between text segments. Numer- ous efforts, such the works by Zhou et al. (2022) and Wu et al. (2023), have greatly advanced the DRR field. Given the intimate connection between discourse relations and coherence, researchers have

---

Letter
Scenes
(90,100)
[o.
Prose
[60,70)
Objects
Characterization
Narrative
(80,90)
Reflection
(70.80)
Arguments
(a) Essay genre distribution.
(b) Essay score distribution.
Figure 2: (a) displays distribution of the 501 essays used construct the dataset by genre, covering total essay genres. (b) shows the distribution of the essays used for annotation in terms of score.
3.2 Fine-grained Annotation Format
classifies discourse relations into explicit and im- plicit relations among paragraph and sentence pairs. based on the presence or absence of connectives. Annotators identified explicit connectives, pin- pointed their arguments, and labeled the discourse relation for explicit relations. For relations with- out explicit connectives, they inferred implicit lations. Our annotation scheme is tailored to ac- commodate the unique characteristics of Chinese discourse, such as complex sentence structures and common sentence grouping practices. For a de- tailed overview of our discourse relation annotation scheme, please refer to Appendix A.
For each essay in our corpus, our annota- tion contains three components, ie., discourse coherence grade, topic sentences and discourse relations.
Discourse Coherence Grade In our dataset, each essay-comprising its title and main text- unde-undegooss a coherence grading process. Ad- hering to conventions from prior studies (Lai and Tetreault, 2018; 8; Flansmose Mikkelsen et al., 2022), annotators use a three-tier coherence grading sys- tem: excellent (scored 2), average (scored 1), or poor (scored 0). They also identify and annotate specific issues affecting coherence, such as off- topic sections, misused connectives, and instances illogical flow. Given the multifaceted nature of discourse coherence, we are contemplating adopt- ing a broader grading scale, possibly Likert scale (Jebb et al., 2021), in future iterations.
3.3 Annotation Process
Our annotation team, consisting of language stu- dents and expert reviewers, underwent training session before starting the annotation process. The dataset was divided into five groups for efficient and consistent annotation. The whole process, in- volving grading discourse coherence, identifying topic sentences, and defining discourse relations, took three months, resulting in total of 501 an- notated essays. For detailed overview of annotation process, please refer Appendix B.
Topic Sentences Essays in our dataset are anno- tated with topic sentences that capture the main themes of their paragraphs, with primary topic sentence signifying the essay's overarching theme. These can be located anywhere within paragraph inferred if explicitly absent.
3,4 Data Statistics
In contrast to traditional topic continuity models, which emphasize topic words, we prioritize topic sentences for their richer thematic context. This approach suits the varied structures and styles of secondary school essays and has proven effective, detailed in 4.3.2.
We present an overview of our dataset's main char- acteristics, spanning coherence grades, topic sen- tences, and discourse relations. The distribution of coherence grades, including Excellent, Average, and Poor, across each genre, is depicted in Table 1. We also detail the frequency and distribution of specific issues impacting coherence in Table 2. Each essay averages to primary topic sen- tences. Considering essays typically have 7-8 para- graphs with 5-8 sentences each, we identified a
Discourse Relations Informed by resources such as the Rhetorical Structure Theory (Mann and Thompson, 1988) and the Penn Discourse Tree- Bank (Prasad et al., 2008), our annotation scheme

---

main topic sentence for every paragraph. While most of these topic sentences start their respective paragraphs, the overall distribution more diverse, as detailed in Table 3.
3.6 Ethical Issues
All data annotators and expert reviewers were com- pensated for their contributions. Furthermore, we have acquired explicit permission from both the authors of the essays and their guardians to use the essays for annotation and publication purposes. To safeguard the privacy of the students, all essays in the dataset have been anonymized, ensuring no per- sonal identifiers are present. We deeply appreciate the trust and support shown by all involved parties.
Finally, examine discourse relations in our corpus, considering both those between adjacent paragraphs and within sentences. The distribution of these explicit and implicit relations is presented Table These relations are further classified into thirteen fine-grained relations, which corre- spond to four coarse-graiin -ranined categories, as detailed in Figure 4.
4 Experiments
4.1 Tasks
3.5 Inner Annotator Agreements
Our annotated dataset serves as the foundation for three tasks, each delving into distinct facets of discourse coherence:
To ensure the annotation quality, Inter-Annotator Agreement (IAA) (Kremer al., 2014) was mea- sured across various tasks associated with identi- fying off-topic information, misuse connectives, inappropriate clause structures, illogical flow in text, sentence connectives and relations, as well paragraph connectives and relations. Table shows the IAA scores for these tasks across dif- ferent annotation batches. Further details on IAA calculation can be found in Appendix C.
Discourse Coherence Grading: This task assesses the overall coherence of essays, pro- viding holistic view of the textual flow and structure.
Topic Sentence Extraction: This task homes in the crux of each paragraph, pin- pointing the primary sentence that best embod- ies the central theme. operates at micro- level, spotlighting the nuances that contribute the broader coherence.
Genre
Coherence Grade (%)
Excellent
Average
Poor
Scenes
29.55
20.45
50.00
Objects
33.33
29.17
37.50
Discourse Relation Recognition: This task is dedicated to unveiling the semantic and logical relations both within sentences and among them. It's instrumental in under- standing the intricate interconnections that un- derpin the textual coherence.
Characterization
37.14
30.00
32.86
Arguments
48.89
31.11
20.00
Reflection
44.83
27.59
27.59
Narrative
37.61
30.77
31.62
Prose
27.27
36.36
36.36
Letter
50.00
19.05
30.95
Total
41.52
28.74
30.14
Each task, while distinct, contributes to our un- derstanding of discourse coherence. The global
Table Distribution of coherence grades across differ- ent genres, presented as percentages.
In Paragraph (%)
In Full-Text (%)
Genre
B
M
E
B
M
E
Secnes
75.71
5.31
18.98
52.27
22.73
25.00
Factor
Count
Avg/Essay
% of Total
Objects
75.95
5.70
18.35
33.33
8.33
58.33
Characterization
78.98
4.08
16.94
50.00
7.14
42.86
Off-Topic
43
0.09
2.40%
Arguments
75.71
5.31
18.98
41.48
25.93
32.59
Misused Conn.
237
0.47
13.21%
Reflection
73.96
4.43
21.61
53.45
13.79
32.76
Inapp. Clauses
1244
2.48
69.34%
Narrative
69.43
10.14
20.42
36.75
10.26
$2.99
Incoherent Logic
270
0.54
15.05%
Prose
6.41
16.67
18.18
63.64
18.18
35.71
40.48
Total
1794
3.58
100.00%
Total
19.37
42.51
19.62
38.32
Table 2: Distribution and average occurrences per essay
Table 3: Distribution of topic sentences' positions within paragraphs and full text, by genre. B, M, E represent beginning, middle, and end respectively.
factors impacting coherence.

---

coherence of an essay, for example, is influenced by its topic transitions and interwoven discourse relations. Topic sentence extraction, in turn, sheds light on both macro-level coherence and the un- derlying discourse relations. Together, these tasks position our dataset as a robust tool for probing discourse coherence, paving the path for advance- ments in NLP and pedagogical research.
ACLE are popular summarization baselines; Tex- (Mihalliaalccaa and Tarau, 2004) and LDA(Blei et al., 2003) as unsupervised methods; transformer- based models like BERT-ext and BERT-abs, which apply the extractive and abstractive summarization techniques, respectively, based on the code² from BERTSuumLLuu and Lapata, 2019). Additionally, include BARTsLewis et al., 2020), T5(Raffel et al., 2020), and ChatGPT, which employ end-to- end generation methods.
Fine-grained
Paragraph
Sentence
Exp.
Imp.
Exp.
Imp.
Discourse Relation Recognition: For this task, we focus on a variety of pre-trained models includ- ing BERT, RoBERTa, XLNet( (Yang et al., 2019), and ChatGPT, which have shown success in natural language understanding in prior work.
Coherence
47
279
197
375
Sequence
751
177
1718
Progression
130
256
269
Contrast
Concession
7
98
9
Tumnaouu
75
34
1096
83
Refinement
22
427
105
1651
For the evaluation, we employ metrics tailored to each task. For both the Discourse Coherence Grading and Discourse Relation Recognition tasks, the models' performance is assessed using Preci- sion (P), Recall (R), Fi-score (F1), and Accuracy (Acc). For the Topic Sentence Extraction task, we use ROUGE-I(R1), ROUGE-2( and ROUGE- (RL,n-grams overlap measures), BLEU (an- other n-grams overlap measure considering up to 4- 4-grams), and BERTSccoreZZhang et al., 2020) (a word overlap measure based on contextual BERT embeddings). This approach allows for task- focused examination of baselines and metrics, pro- viding clear distinctions between models and the rationale behind their selection and evaluation.
Generalization
49
618
57
754
Background
5
127
12
407
Objective Causal
11
15
309
172
Subjective Inference
4
1
37
38
Specific Conditionul
0
0
121
7
Hypothetical Conditional
3
2
108
9
Total
350
2403
2640
5566
Table 4: Distribution of Discourse Relations Para-
graph and Sentence Levels.
4.2 Baselines and Evaluation Metrics
We test several existing models each of our dis- course assessment tasks. These models comprise traditional NLP models, pre-trained transformer models such as BERT(Devlin et al., 2019) and RoBERTa(Liu et al., 2019), as well as large-scale language models like ChatGPT, specifically the gpt-3.5-turbe version'. We employ both zero-shot and few-shot learning for all tasks. The prompt and details for utilizing ChatGPT can be found in Appendix D.
Batch
Toprc
Uli
Misused Conn.
Inapp: classes
Illogical Flow
Printry Topr:
Sem. Relaioon
Para
Relorions
Par.
61.71
67.61
61.36
56.07
60.94
95.32 34
95.16
2
65.39
66.71
49.22
65.69
79.26
93.26
$4.35
92.23
65.32
71.99
57.41
79.39
78.94
93.59
79.25
92.76
70.41
87.11
57.20
64.51
60.97
96.1
Avg
Table 5: Consistency analysis results showing the inter- annotator agreement (IAA) scores (in percentage) across different aspects of text analysis for various data sub- mission batches (each batch represents round of anno- tations). The last row shows the average IAA scores for all batches.
Discourse Coherence Grading: In line with the work of (Lai and Tetreault, 2018), we test a variety of models: Entity-based Models (EGRID (Barzi- lay and Lapata, 2005) and EGRAPH (Guinaudeau and Strube, 2013)) and neural network models (CLIQUE (Li and Jurafsky, 2017), SENTAVG and PARSEQ (Lai and Tetreault, 2018) have shown promise in previous research for this task, and we further include the aforementioned pre-trained and large-scale language models.
4.3 Main Results and Analysis
In this section, we present and analyse the results of the benchmark model for each subtask in turn. We use the annotations from the first 401 compositions in the dataset the training set, and the annotations from the last 100 compositions as the test set.
Topic Sentence Extraction: Inspired by meth- ods from the extractive summarization field, we test models of different types: LEAD-3 and OR-
Pht httpgtuubb..oonlyyyyg/rrrrsmmmm

---

4.3.1 Discourse Coherence Grading
duction given in the prompt. However, its per- formance notably improves when provided with several examples, indicating the potential benefits of example-base fine-tuning in enhancing its task comprehension.
Table 6 showcases the performance of diverse models for the Discourse Coherence Grading task. The Transformer-basee models BERT and RoBERTa, especially the latter, displayed superior performance in distinguishing discourse coherence, reflecting their effective utilization of contextual
Model
R2
RL
BLEU
BERTScore
information in the text.
LEAD-3
67.58
62.06
63.79
43 13:00
85.33
ORACLE
93.91
92.78
93.45
91.52
97.16
TextRank
60.29
54.4
59.34
51.32
81.25
Model P(%)
R(%)
(((%)
Acc(%)
LDA
80.62
76.61
79.79
74.86
90.48
EGRID
35.83
35.12
33.92
40,00
BERT-ext
70.78
64.03
69.62
66.25
85.49
EGRAPH
27.89
28.36
27.92
43.00
BERT-abs
54.52
42.46
48.89
39.78
76.87
CLIQUE
33.29
33,86
28.45
29.57
BART
74.58
68.9
73.32
63.51
87,51
SENTAVG
33.9
34,2
33.99
45.00
T5
74.75
70.33
69.94
39.33
86.15
PARSEQ
35.16
33.58
31.26
34.00
ChaGGT
24.74
13.06
22.8
8.29
63.22
BERT
41.42
36.78
34.01
57.00
ChaiGPTa-ebot
41.05
30.65
38.58
25.07
70.92
RoBERTa
50.37
47.16
46.73
58.00
Table 7: Performance (in %) of various models on Topic
ChatGPT
34.13
30,26
31.15
55,00
Sentence Extraction.
ChaiGPT
41.37
42.40
41.49
54.00
Table 6: Comparative performance of different models for Discourse Coherence Grading.
Parngraphi: B 1eaaatr.eaaa aptertof 14
1-844-- K8H502 BiElle (1066) Eitri Bar tae
SEUJE (7 Ra0e,
Peesgrahh: 4-850 -idh XEXEEEEE 288T Mi
Entity-based models, namely EGRID and EGRAPH, scored lower than Transformer-based models but still managed to compete with Neu- ral Network models such SENTAVG, CLIQUE, and PARSEQ. This indicates that even simpler ap- proaches like entity transition patterns hold signif- icant importance in determining discourse coher- ence. The Large-scale Language Models, repre- sented by ChatGPT and ChatGPT, shot- showed commendable performance, with CharGPT, shot notably outperforming its zero-shot counterpart.
Cometiie 1831
Ryp Est
Garend Truik Relatee:
Prediction:
(0) Estay
Paragrophi:
people may
Lkmoiker, qualiyy
moverial belkoing dieerseeee weiling opueradiis [Dvinsd]
tuat dhre loes
onker absan then,
oppettatiio and adhove youncil
Paragoggh:: Tunntrr giua op may opportsity and bellewo in the say lng "There ras hos our covettor qudiitee, you
excese
Connectivr: Larefore
Rdeeiie lype: Expliil
Grozn Trisk Relatlas: Examittation
English Translation
Figure 3: Case study on explict discourse relation recog- nition among paragraphs
4.3.2 Topic Sentence Extraction
4.3.3 Discourse Relation Recognition
As Table illustrates, the ORACLE model, due to its utilization of ground truth labels, unsurpris- ingly achieves the highest scores across all metrics. Among Transformer-basee models, BART and T5 outperform others, demonstrating their effective- ness in the topic sentence extraction task. Interest- ingly, the less complex models, LEAD-3 and LDA, also yield competitive performance. This finding suggests the significant roles of initial sentences and topic coherency for this particular task.
Tables and 9 delincate the varied model per- formance in discourse relation recognition. At the paragraph level, BERT targe generally outper- forms, highlighting its aptitude in complex con- texts, yet certain categories are better addressed by RoBERTatarge and XLNetmid- underscoring the potential for specialized optimizations. On the sentence-level, BERT models take the lead, but RoBERTatarge exhibits prowess in Implicit Elab- oration and Causal relations, emphasizing its nu- anced relation handling.
In contrast, when considering pre-trained lan- guage models, it is found that ChatGPT and its few-shot variant perform less optimally compared other models. Despite the remarkable language understanding capabilities of ChatGPT, the model appears to lack a full comprehension of our topic sentence extraction task, despite a detailed intro-
An intriguing observation during inter- paragraph experiments was the underperformance of explicit relations compared to implicit ones, despite the presence of cues like so. Delving into this, our examination (Figure 3) revealed that the cue there- fore, typically signaling causality in inter-sentence

---

Elaboration
Reversal
Causal
Co-occurrence
Total Fi
Total Are
Exp.
Imp.
Exp.
Imp.
Exp.
Imp.
Exp.
Imp.
Exp.
Imp.
Exp.
Imp.
BERT
0
7.41
70.97
26.67
51.85
69.68
71.05
64.52
48.47
42.07
62.65
64.884
Barge
44.44
24.39
78.79
13.33
73.68
71.46
77.61
62.13
68.63
42.83
74.70
64.62
RoBERTa
0
66.67
75.85
75.29
74.65
35.49
37.63
61.45
72.53
RoBERTaterge
0
28.57
70.97
12.5
68
76.04
75.95
75,29
53.73
48.10
69.88
72.75
XLNet
0
7.69
41.67
0
61.02
75.54
76.92
73.22
44,90
39.11
63.86
71.65
XLNetmid
16.67
70.97
26.67
72.34
75.12
77.50
75.27
61.45
48.43
72.29
72.09
ChatGPT
0
6.06
47.06
5.19
34.48
12.71
67.35
60.99
37.22
21.24
55.42
41.76
ChatGPT3 shot
0
10.26
41.38
6.67
21.26
29.47
57.45
58.99
30.11
26.35
44.58
45.93
Table Performance of various models on the Coarse-grained Discourse Relation Recognition between paragraphs. Displayed are the F1 scores (%) for explicit (Exp.) and implicit (Imp.) relations for each type the CEDCC dataset, and overall F] scores and Ace for all Level-1 senses.
Elahoration Imp.
Reversal
Causal
Co-occurrence Exp.
Total Fi
Total Ace
Exp.
Exp.
Imp.
Exp.
Imp.
Imp.
Exp.
Imp.
Exp.
Imp.
BERT
87.87
38.14
92.39
12.9
76.92
73.82
79.87
61.66
84.26
46.63
86.76
64.29
BERTaagge
87.95
38.58
92.47
6.67
77.37
74.26
79.03
64.44
84,20
45.99
86.61
65.97
RoBERTa
85.71
29.3
92.12
8.33
72.73
73.93
80.00
63.69
82.64
43.81
85.57
65.58
RoBERTuyarge
81.51
42.33
91.98
8.33
68.71
77.95
80.49
63.87
80.67
75.25
68.15
40.48
XLNet
84.85
35.71
92.20
9.52
73.42
76.51
78.77
64.53
82.31
46.57
85.12
67.96
XLNctmid
85.15
36.18
92.15
18.75
75.18
76.97
78.29
66.84
82.69
49.68
85,42
68.15
ChatGPT
20.32
8,82
40.72
2.41
13.33
8.74
41.56
48.61
28.99
17.15
36.76
29.37
ChatGPT, 1133--shh shot
38.68
10.57
59.84
8.14
33.03
31.91
48.41
46.81
44.99
24.36
49.55
34.42
Table 9: Performance of various models on the Coarse-graings Discourse Relation Recognition between sentences. Displayed are the scores (%) for explicit (Exp.) and implicit (Imp.) relations each type on the CEDCC dataset, and overall Fj scores and Ace for all Level-1 senses.
contexts, often prefaced a summary or general- ized statement at the paragraph level, thus indi- cating generalization more than causation. This reflects the nuanced semantic intricacies between paragraphs and the associated modeling challenges, suggesting refinement in future recognition work accommodate these variances.
Explicit
Implicit
Overall
Model
Grade
Ace
Fl
Acc
Fj
Ace
F;
0
55.56
43.83
55.26
29.41
55.32
39.1
BERT
I
90.91
67.73
64.08
32.93
66.67
52.60
2
61.11
47.03
67.39
40.91
66.36
49.58
66.67
72.37
71.28
RoBERTa
54.55
32.05
66.99
34,66
65,79
34.40
2 61.11
35.23
74,64
38.65
72.42
38.22
0
61.11
43.48
71.05
45.12
69.15
48.98
ChatGPT variants show moderate results, sug- gesting enhancement opportunities. Conclusively, no single model excels consistently, pointing to- wards the merit of hybrid or specialized strategies. Further granular recognition details available in Appendix E.
XLNet
1
45.45
33.33
66.99
34.45
64,91
41.81
68.52
73.19
72.42
Table 10: Performance of different models discourse relation recognition tasks at different coherence levels.
for essays with the highest coherence level. This suggests that for certain models, essays with better coherence might facilitate improved discourse rela- tion recognition. However, for models like BERT and RoBERTa, the relationship between coherence and performance is less straightforward, indicat- ing further nuances in how text coherence might influence discourse recognition across models.
5 Discussion
5.1 The Impact of Coherence on Discourse Relation Recognition
According Table 10, while the performances vary across different coherence levels, there are some noticeable patterns. For instance, XLNet consis- tently achieves its best overall Acc and F, scores
This outcome underscores the importance of

---

considering text coherence in the application of discourse relation recognition models, hinting at possible enhancements from employing coherence grades, provided they are obtainable. Additionally, the need for more robust models capable of dealing with text of varying coherence levels is evident, crucial requirement considering the wide-ranging coherence grades present in real-world data.
text coherence in discourse relation recognition. Our findings set benchmark for future discourse analysis research Chinese, potentially sparking further advancements in the field.
Limitations
The limitations of our corpus include:
Interestingly, the recognition of implicit rela-
Limited by data scale: While our dataset is among the largest in this field, its size is still constrained. The diversity and complexity of discourse coherence phenomena imply that the larger the dataset, the more comprehen- sive its coverage of these phenomena. Conse- quently, the current size of our dataset might limit the performance and generalizability of models trained on it.
tions seems to be less sensitive to the coherence grade, highlighting the inherent difficulty of infer- ring these relations. This underscores the need for more advanced techniques capable of identify- ing implicit discourse relations, which could sig- nificantly enhance the overall performance of dis- course relation recognition models.
5.2
The Impact of Fine-grained Annotations on Discourse Coherence Grading
Constraints of manual annotation: Our dataset relies significantly on manual amno- tations by linguistic experts. Nonetheless, due to the labor-intensive and time-consuming na- ture of this process, there are inevitable limita- tions on the volume of annotated data. Further, the inherent subjectivity of manual annotation might lead potential inconsistencies and bias the annotated labels.
Model
P(%)
R1%)
F((%)
Acc(%)
CharGPT
34,13
30.26
31.15
55.00
CharGPTs-ate
41.37
42,40
41.49
54.00
CharGPT /inem grw.eedd ancorittues
34.73
53.40
Comparative performance of ChatGPT varia- tions in discourse coherence grading task.
Table 11:
As illustrated in Table 11, the inclusion of fine-grained annotations bolsters the ChatGPT model's performance in discourse coherence grad- ing. Specifically, this method improves recall sub- stantially (53.4%), indicating better detection of various error categories in essays.
Dependent on essay quality: Our dataset consists of essays authored by middle school students, which might contain language in- accuracies or logical inconsistencies absent in more formal, polished texts. These could affect the learning and generalization capabil- ities of the models, particularly when applied other types of text such academic for- mal writing.
Furthermore, this approach yields the highest accuracy (60%) among the compared models, vali- dating the value of fine-grained annotations. These findings reveal that even with infrequent eтror cate- gories, weakly supervised methods can utilize fine- grained annotations effectively for enhanced per- formance, underlining their utility in such context- dependent tasks.
A ccknowledgement
We appreciate the support from National Natural Science Foundation of China with the Main Re- search Project on Machine Behavior and Human- Machine Collaborated Decision Making Method- ology (72192820 & 72192824), Pudong New Area Science & Technology Development Fund (PKX2021-ROS) Science and Technology Com- mission of Shanghai Municipality (22DZ2229004), Shanghai Trusted Industry Internet Software Col- laborative Innovation Center and East China Nor- mal University International Conference Grant Pro- gramme.
6 Conclusion
In this paper, we introduced the Chinese Essay Discourse Coherence Corpus (CEDCC), a richly annotated, diverse dataset aiming to address lim- itations in current discourse coherence research. Our dataset integrates aspects of coherence grading, topical continuity, and discourse relations, thereby breaking away from the isolated approach of pre- vious studies. We further demonstrated the impor- tance of fine-grained annotations and the role of

---

References
Evaluation Conference, pages 2440-2445 Marseille. France. European Language Resources Association.
Hesam Amoualian, Wei Lu, Eric Gaussier, Georgios Ba- likas, Massih R. Amini, and Marianne Clausel. 2017. Topical coherence in L.DA-based models through in- duced segmentation. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1799- 1809, Vancouver, Canada. Association for Computa- tional Linguisties.
Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wen- biao Ding, and Minlie Huang. 2021. Long text gener- ation by modeling sentence-level and discourse-level coherence. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguis- tics and the 11th International Joint Conference on Natural Language Processing, ACL/JCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 6379-6393, Association for Computa- tional Linguistics.
Regina Barzilay and Mirella Lapata. 2005. Model- ing local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting of the As- sociation for Computational Linguistics (ACL '05), pages 141-148, Ann Arbor, Michigan. Association for Computational Linguistics.
Camille Guinaudeau and Michael Strube. 2013. Graph- based local coherence modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 93-103, Sofia, Bulgaria. Association for Com- putational Linguistics
Federico Bianchi. Silvia Terragni, and Dirk Hovy. 2021. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Natural Language Processing, ACL/JCCNPP 2021. (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 759-766. Association for Computational I Linguistics.
Andrew T. Jebb, Vincent Ng, and Louis Tay. 2021. A review of key likert scale development advances: 1995-2019. Frontiers in Psychology. 12.
Joint Conference
Gerhard Kremer, Katrin Erk, Sebastian Padó, and Stefan Thater. 2014. What substitutes tell us analysis of "all-words" lexical substitution corpus. In Proceed- ings of the 14th Conference of the European Chap- ter of the Association for Computational Linguisties. pages 540-549, Gothenburg, Sweden. Association for Computational Linguistics.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993-1022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedingy of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 4171-4186 Association for Computational Linguistics.
Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: dataset, evaluation and methods. Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214-223, Melbourne, Australia, Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 7871-7880, Online. Association for Computa- tional Linguistics.
Youmnaa Farag, Helen Yannakoudakis, and Ted Briscoe. 2018. Neural automated essay scoring and coher- ence modeling for adversarially crafted input. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HIL 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 263- 271. Association for Computational Linguistics.
Jiwei Li and Dan Jurafsky. 2017. Neural net models of open-domain discourse coherence. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 198-209, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 2014. The impact of deep hierarchical discourse structures in the evaluation of text coherence. In COLING 2014, 25th International Conference on Computational Lin- guistics, Proceedings of the Conference: Technical Papers. August 23-29, 2014, Dublin, Ireland, pages 940.9949.. ACL.
Ziheng Lin, Hwee Tou Ng, and Min- Yen Kan. 2011. Au- tomatically evaluating text coherence using discourse relations. In The 49th Annual Meeting of the Asso- ciation for Computational I Linguistics: Human Lan- guage Technologies, Proceedings of the Conference, 19-24 June, 2011. Portland, Oregon, USA, pages 997- 1006. The Association for Computer Linguistics.
Linea Flansmose Mikkelsen, Oliver Kinch, Anders Jess Pedersen, and Ophélie Lacroix. 2022. DDisCo: discourse coherence dataset for Danish. In Pro- ceedings of the Thirleenth Language Resources and

---

Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMML.P-JCNNLP 2019, Hong Kong, China, Novem- ber 3-7, 2019, pages 3728-3738. Association for Computational Linguistics.
Association for Computational Linguistics, 9:621 640.
Disha Shrivastava, Abhijit Mishra, and Karthik Sankara- narayanan. 2018. Modeling topical coherence in dis- course without supervision. CoRR, abs/1809.00410
Hongyi Wu, Hao Zhou, Man Lan, Yuanbin Wu, and Yadong Zhang. 2023. Connective prediction for im- plicit discourse relation recognition via knowledge distillation. In Proceedings of the 61 Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 5908-5925 Toronto, Canada. Association for Computational Linguistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 11692.
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional the- ory of text organization. Text- Text-interdisciilnnary Jour- for the Study of Discourse, 8(3):243-281.
Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang- 2019. Modeling coherence for discourse neural ma- chine translation. In The Thirty AAAI Con- ference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial In- telligence Conference, LAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- -eeruary 1, 2019, pages 7338-7345. AAAI Press.
WILLIAM C. MANN and SANDRA A. THOMPSON. 1988. Rhetorical structure theory: Toward func- tional theory of text organization. Text Interdisci- plinary Journal for the Study of Discourse, 8(3):443- 281.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in con- junction with ACL 2004, 25-26 July 2004, Barcelona, Spain, pages 404-411, ACI..
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances Neural In- formation Processing Systems 32: Anmual Confer- ence Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Elham Mohammadi, Timothe Beiko, and Leila Kosseim. 2020. On the creation of a corpus for coherence evaluation of discursive units. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence. pages 1067-1072, Marseille, France, European Language Resources Association.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008 The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)
Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt-based connective predic- tion method for fine-grained implicit discourse rela- tion recognition. arXiv preprint 10.00733...
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn, Res., 21:140: 10:1:::::667..
Hannah Rohde, Alexander Johnson, Nathan Schneider, and Bonnie L. Webber. 2018 Discourse coherence: Concurrent explicit and implicit relations. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2018, Mel- bourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2257-2267. Association for Computa- tional Linguistics.
Aili Shen, Meladel Mistica, Bahar Salehi, Hang Li, Tim- othy Baldwin, and Jianzhong Qi. 2021. Evaluating document coherence modeling. Transactions of the

---

A Annotation Scheme of Relations
Culerrnvv #19ESE
Cancesson
Reversal SE
Turndous 1L2
Our annotation scheme, inspired by Rhetorical Structure Theory (RST), Penn Discourse Tree- bank (PDIB), Chinese compound sentence theory, and Chinese sentence group theory (MANN and THOMPSON, 1988; Prasad et al., 2008), is intri- cately tailored to the nuances of Chinese discourse. It encompasses four tiers and thirteen labels, cap- turing a broad spectrum of logical semantic types in Chinese discourse. This versatility allows for unified analysis across diverse discourse units.
Co-sccurrence JEULGE
Progres
Bacspround JLSE
Oustrel 10 1i:266
Causal
Selvestive Iufozses
Reframmani 13
Speeitic Confitiona 6532 6022
Flaboration Wouttt
Hoperhetiool
12:
Figure 4: Hierarchical representation of the annotation scheme for coarse and fine-grained discourse relations.
While our scheme might not explicitly differ- entiate arguments in the manner of the PDTB, imbues directionality in discourse relations. Unlike the traditional undirected relations, our model de- lincates both parallel and subordinate relationships among discourse units:
Sequence (Co-occurrence): This label is used when there is a temporal, spatial, pro- cedural, or logical sequence of events within the discourse units of chapter. It includes both sequential and inverse orders but does not encompass simultaneous events.
These involve discourse units of equal standing, without any hierar- chical distinction. Examples include relations like Co-occurrence and Elaboration. In cases such as coherence or contrast, the or- der of discourse units can often be changed without affecting the overall meaning.
Parallel Relations:
Progression (Co-occurrence)) This label ap- plies when a subsequent discourse unit repre- sents an advance in quantity, quality, scope, or time from the preceding discourse unit, indi- cating increased depth. The order of the dis- course units is usually non non-interchangeable.
Contrast (Co-occurrence): This label repre- sents notable, direct, and relative difference or similarity between entities, concepts, ideas, actions, states, etc., appearing in the text.
Subordinate Relations: These denote hier- archy, where discourse unit takes prece- dence over the other. Relations like Reversal and Causal are illustrative of this category, where the primary discourse unit can be seen as the main clause, and the subsequent as the subordinate clause.
Concession (Reversal): This label is used when discourse unit presents hypotheti- cal scenario, and another unit describes a situ- ation that contradicts or opposes the hypothe- sis.
This distinction ensures our scheme neither over- simplifies nor overcomplicates the importance of each discourse relation. strikes an optimal bal- ance: intricate enough to convey the depth of Chi- nese discourse yet straightforward enough to mini- mize potential annotation challenges and subjectiv- ity.
Turnabout (Reversal): This label applies when one discourse unit presents an objec- tive fact, and another unit recounts a situation that opposes or contradicts it.
This label is used when subsequent discourse unit refines a prior unit, including examples, explanations, illustrations, additions, etc. Reversing the or- der of the units would transform the relation- ship into generalization.
Refinement (Elaboration):
Figure 4 visually illustrates schema, empha- sizing the interplay between coarse gransed and fine-grained discourse relation labels. Detailed def- initions of these fine-grained labels follow:
Coherence (Co-occurrence): This label de- scribes aspects of the same event, related events, contrasting situations that coexist, co-occur, or oppose in meaning. These as- pects can be reordered without altering the overall sentence meaning.
Generalization (Elaboration): This label ap- plies when a subsequent discourse unit gener- alizes, summarizes, or extends a prior unit. If the units were reversed, the relationship would be considered a refinement.

---

Background (Causal): This label describes the introduction of events, places, histories, etc., which often occurs chapters and forms a contextual relationship with the main body of the chapter.
The entire process spanned three months, during which total of 501 essays were annotated. This structured approach ensured streamlined annota- tion process, resulting in richly annotated corpus that can facilitate subsequent language model train- ing and research.
Objective Causal (Causal): This label is used when one discourse unit states cause. and another unit states the objective result that ensues from the cause.
C Inter-Annotator Agreement (IAA) Calculation
In this study, we adopted an Inter-Annolato Agree- ment (IAA) measure, computed as follows:
Subjective Inference (Causal): This label ap- plies when discourse unit states the factual basis, and another unit states the subjective conclusion inferred from it.
PA = T E(sher PAO
(1)
PA.j Σ
N
Specific Conditional (Causal): This label is used when one discourse unit presents a specific condition, and another unit states the result inferred from that condition.
In the above equations, PA denotes the overall Pairwise Agreement (PA) among the annotators, with P| representing the total number of annotator pairs. PAij represents the agreement for each pair of annotators and j, summed across all N text fragments. sf and sy denote the annotations made by annotator and j on text fragment k; respec- tively.
Hypothetical Conditional (Causal): This la- bel is used when one discourse unit presents hypothetical condition, and another unit de- scribes the outcome the condition were met or the measures needed to meet it.
Table 5 in the main text presents the IAA scores across different tasks performed during various data batch submissions. Each row corresponds to an individual data batch, with each entry indicat- ing the respective IAA score for the corresponding task. The rigorous calculation of TAA allows us evaluate the quality and reliability our annota- tion process, revealing areas of high agreement and highlighting areas for potential improvement.
B Detailed Annotation Process
Our annotation process was carried out by a team composed of four undergraduates, four postgradu- ates from language-related fields, and four expert reviewers with experience in Chinese teaching. The principle of minimal changes was followed in the process to retain the original language used by the secondary school students.
D ChatGPT Prompt Configuration
For all tasks, including Discourse Coherence Grading, Topic Sentence Extraction, and Discourse Relation Recognition we employ both zero-shot and few-shot learning strategies. Ad- ditionally, we discuss how the incorporation fine- grained annotations enhances the performance the ChatGPT model in discourse coherence grad- ing task. Please note that the original prompts were written in Chinese. We provide here their English translated versions.
Before the actual annotation process, the team underwent a training session to familiarize them- selves with the tasks. The tasks included grading the discourse coherence, identifying the topic sen- tences, and defining discourse relations.
To ensure efficiency and consistency, the data was divided into five groups for annotation. The ini- tial annotation was done by the undergraduate and postgraduate students, while the expert reviewers validated and corrected their work. This process was aimed at maintaining the quality and consis- tency of the annotations.
D.1 Discourse Coherence Grading
The prompts we use for this task are as follows: In the zero-shot setting, we ask the model:
Furthermore, we organized weekly online dis- cussions address any common issues that arose during the annotation process. The discussion also served as a platform to make necessary adjustments in the annotation process.
Given an essay, you are tasked with assessing its overall coherence and pro- viding a grade. The grade should be 0 for

---

incoherent, for average, and for excel- lent coherence. The provided essay will be in the format "essay_id essay_tiiee __itte _otatet__tttt,,,, where "tt' represents tab space, and different paragraphs in the essay content are separated by "In'. Please return the result in the format "id":[essay_idl" "Ll.""Logirrr [ggade]]' In the 3-shot setting, the prompt is modified as follows:
Your task is to extract the topic sentence from the following paragraph. The topic sentence should be a complete sentence that summarizes, narrates, and explains the theme of the paragraph. You should select as complete a sentence as possi- ble from the original text. If the para- graph does not have clear topic sen- tence, please answer "There is no topic sentence." Note that only few para- graphs lack a clear topic sentence. Here are three examples to help you under- stand the task: [S] Now, the paragraph is: [P]. Please provide the topic sentence.
Given an essay, you are tasked with assessing its overall coherence and pro- viding grade. The grade should for incoherent, for average, and for excel- lent coherence. The provided essay will be in the format 'essay id At essay essay_title It assay_content tab space, and different paragraphs in the essay content are separated by "In'. Please return the result in the format [id":[essyy_dd] MagiiGrraee:[graddee Here are three sample essays with their coherence grades for reference. The format of these samples is 'essay id It essay_titl title At essay_ _onttentnttttt \t coher- ence pnce_grade', where "\t" represents a tab space.[S][T]
Here, [S] represents the three provided samples, and [P] represents the content of the paragraph that needs to be processed.
where "\t" represents
D.3 Discourse Relation Recognition
For the task of discourse relation recognition, we present the model with different prompts in the zero-shot and 3-shot settings:
In the zero-shot setting, the model receives the following prompt:
Your task to recognize the discourse relation between two texts. You can find possible discourse relations and their def- initions in the following list: [D]. The input format is: ( ("Text1": "First text", "Text2": "Second text"). For example, if the input is: { ""Textl:: "I saw Núwa first killed giant turtle, propping up the sky with its legs.", "Text2": "Then, she killed a black dragon. "J, then you should output: 'Sequential relationship'. [T]
Here, [S] represents the three provided samples, and [T] is the associated information for the essay that needs to be assessed.
D.2 Topic Sentence Extraction
For the task of topic sentence extraction, we use the following prompts for both zero-shot and 3-shot learning settings:
In the zero-shot setting, we present the model with this prompt:
the 3-shot setting, we present the model with the following prompt:
Your task is to extract the topic sentence from the following paragraph. The topic sentence should be complete sentence that summarizes, narrates, and explains the theme of the paragraph. You should select as complete a sentence as possi- ble from the original text. If the para- graph does not have a clear topic sen- tence, please answer 'There is no topic sentence.' The current paragraph is: [P). Please provide the topic sentence.
Your task to recognize the discourse relation between two texts. You can find possible discourse relations and their def- initions in the following list: [D]. Now, let's look at some examples: [S]. For ex- ample, if the input is: ("Text1": : "I saw Núwa first killed giant turtle, propping up the sky with its legs.". "Text2": "Then, she killed black dragon."), then you should output: 'Sequential relationship'. [T]
In the 3-shot setting, the model is prompted with:

---

Here, [D] provides the detailed definitions of discourse relations (sce Appendix A for more de- tails), [S] represents the given three examples, and [T] represents the text formatted according to the norms.
XLNet at 15.80%, and BERT at 14.20%. In con- trast, ChatGPT and ChatGPT3-shot showed rel- atively poorer performance, scoring 9.48% and 7 7.00%, respectively. terms of overall accuracy (Acc), XLNet led with a score of 42.17%, closely followed by RoBERTa at 40.96%. However, even under the best circumstances, there remains sub- stantial room for improvement in these models. In implicit discourse relations, XI.Net again per- formed the best in terms of the overall F] score, with 19.97%, followed by RoBERTa and BERT with 16.13% and 16. 14%, respectively. However, the performance of ChatGPT and ChatGPT lagged again in this task, scoring 5.66% and 6.57% respectively. In terms of overall accuracy (Acc), XLNet led with score of 55.38%, while the per- formances of other models were not much different, and ChatGPT and ChatGPT3-shol were still under- performing.
D.4 Fine-Grained Annotation in Discourse Coherence Grading
The prompts we use for this task are as follows:
Your task is to read and understand a middle school essay, and then identify and count the fine-grained factors that impact the coherence of the essay. This includes the number of times the topic sentence deviates, the number of times connectives are improperly used, the number of times sentences are illogically broken, and the number of times the logical relation between contexts does not flow smoothly. The essay is provided in the format 'Ar- ticle IDItArticle TitleAteext Content', where "It' denotes tab space and 'In' indicates paragraph break. You should return your result in the format "'iti[AAticle ID], "OffTopii"::Numbii of Times Topic Sentence Devi- "MiisssedCoommctivssssNnnaa of Times Connectives Improperly Used], http://aappoppiaelaassssssss of Times Sentences Illogically Bro- "ogll""nchhreennLoiieee [Nuuberr of Times Logical Relation Between Con- texts Doesn't 't Flow SmoothlyJ'. It's important that you analyze every article on a word-for-word basis and mark all possible errors.
In Table 13, for discourse relations between sen- tences, the performance of the models is similar to the situation with inter-paragraph relations. XI- Net, RoBERTa, and BERT performed well in most cases, while ChatGPT and ChatGPT3-ad shoot per- formed poorly in most tasks.
In conclusion, although large pre-trained lan- guage models have achieved significant results in many NLP tasks, there is still room for im- provement their performance on fine-grained discourse relation recognition tasks, especially in the recognition of implicit discourse relations. No- tably, ChatGPT and ChatGPT generally per- formed poorly in this task, possibly because their pre-training process lacked training data specifi- cally targeting discourse relations. Therefore, in futurre work, we may need to pay more attention specific training techniques for this task to improve model performance.
Here is an example of how essay analysis and returned error count would look like to help you understand the [E][][E][]]]
Here, [E] represents the given example and [T] represents a well-formatted example of cssay anal- ysis and returned error count.
E Fine-grained Discourse Relation Recognition
As shown in Table 12, for explicit (inter-paragraph) discourse relations, RoBERTa performed the best with an overall F, score of 16.02%, followed by

---

Explict
Implict XLNet
BERT
RoBERTa
XLNet
CharGT
ChaiGiT:-
BERT
RoBERTa
CharGPT
ChatGPT:
Hypothetical Couditional
0
0
0
Progression
16.67
Sequence
61.73
Subjective Inference
0
0
0
Turnabout
11.11
30.00
43.75
37.04
0
2.74
Coherence
64.00
62.50
55.56
0
62.02
62.41
74.29
Refinement
46.15
16.67
0
38.15
42.31
40.52
Generalization
45.00
54,90
34,78
20.00
55.44
50.58
58.71
16.56
32.85
Concession
0
0
0
Background
0
0
0
16.00
8.33
Contrast
0
0
0
Objective Causal
Specific conditional
0
0
Total Fi
14.20
16,02
15.80
9:48
7.00
16. 14
16.13
19.97
5.66
6.57
Total Aoe
33.73
40.96
42.17
28.53
23.36
49.01
48.57
55,38
28.40
30.53
Table 12: Performance various models on the Fine- Fine-grained Discourse Relation Recognition between paragraphs. Displayed the F; scores (%) for Explicit and Implicit relations for each type the CEDCC dataset, and overall Fi scores and Ace for all Level-2 senses.
Explict
Implict
BERT
RoBERTa
CharGPT
CharGPTa abot
BERT
CharGPT
CharGi Ta Ttot
Hypothetical Couditional
79.49
30.19
20,00
Progression
59.32
55.86
60..0
4.55
8.11
18.18
15.19
18.18
22.50
4.26
Sequence
43.75
56.25
57.14
11.69
12.93
54.92
54.85
54.29
38.75
37.89
Subjective Inference
66.67
0
0
0
18.18
0
Turnabout
91.95
39.25
59.15
10.00
1.63
Cohereace
61.39
18.46
52.54
Relinement
71.26
70.59
70.45
0
0
66.57
66.58
70.69
0.56
Generalization
62.30
66.67
62,50
25.00
32,88
52.28
51.76
55.66
13.40
19.44
Concession
13.33
14.29
13.33
0
11.11
0
25.00
Background
26.79
Contrast
42.86
51.61
19.05
23.53
22.22
40,00
7.84
Objective Causal
87.88
86.15
91.34
0
16.90
28.57
21.28
23.26
Specific conditional
78.48
75.95
81.58
4.55
20.83
0
Total Ft
$8.48
51.16
55.48
11.75
20.29
27.34
21.25
29.31
7.50
8.92
Ace
21.80
36.06
56.75
Table 13: Performance of various models the Fine-grained Discourse Relation Recognition between sentences. Displayed are the Fj scores (%) for Explicit and Implicit relations for each type on the CEDCC dataset, and overall Fi scores and Ace for all Level-2 senses.


Reviews:
[{"review_id": "mzstwhJGsW", "reviewer": "Reviewer_cwRY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your invaluable feedback and thorough review. The paper introduces a novel multi-task benchmark that could be easily plugged in a framework designed to assess Large Language models so as to provide linguistically grounded intuitions on how such models understand text coherence. To address your concerns, I offer the following responses:\n1. The choice of three levels for coherence grading was made in reference to previous datasets such as GCDC and DDisCo. Considering the complexity and multifaceted nature of discourse coherence, we provided clear definitions and annotations for fine-grained factors under the guidance of linguistic professors. We recognize the merit of the Likert scale and will consider introducing more grading levels in future works.\n2. Thank you for pointing out the importance of providing more detailed statistics in section 3.4. To address your suggestion: Our dataset comprises 501 essays written by secondary school students. The essay with the maximum number of tokens contains 1,600 tokens, while the one with the minimum has 603 tokens. On average, each essay consists of approximately 713.18 tokens. We commit to providing more comprehensive dataset statistics in the next version of the paper.\n3. There was indeed an oversight on line 288. In reality, each essay contains an average of 7 topic sentences. Each secondary school essay consists of 7-8 paragraphs, with about 5 to 8 sentences each. Our annotation aim was to identify the topic sentence for each paragraph. We will add those necessary details to next version. \n4. The model we utilized was gpt-3.5-turbo. We will rectify this in the subsequent version.\n5. In lines 468-474, our intention was to highlight that, despite the appearance of randomness in peak performances, essays with higher coherence generally perform better in discourse relation recognition.\n6. We assure the authenticity of all experimental data. We've also observed the counterintuitive phenomenon you mentioned and discussed it in detail in the appendices E.\n7. The IAA formula you referenced comes from the paper \"What Substitutes Tell Us – Analysis of an “All-Words” Lexical Substitution Corpus\" (https://aclanthology.org/E14-1057.pdf). We will incorporate this citation in our next version.\n8. Thank you for your suggestion regarding the appendices. We plan to integrate its content into the main text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strength of this work is that the dataset is composed of real-world texts, even if the authors ackowledge that the use of texts produced in the context of middle school may suffer for the overall quality of the essays.\nAnother valuable aspect is that, for each subtask, authors test many models relying on different approaches and architectures, ranging from rule/entity-based models to  recent generative LLMs.", "reasons_to_reject": "I found many doubtful passages.\n\nThe annotation of coherence in 3 classes sounds like limiting the possibility to grade coherence, that is a complex and multifaceted phenomenon. A wider Likert scale would have been more suitable. \n\nIn 3.4 Data Statistics, some basic stats, such as size of the corpus/dataset in terms of tokens, average tokens per essay, and so on, may give the reader a more complete idea of the dataset.\n\nIn the same section, at 286-289, authors say that for each paragraph there is an average of 7 topic sentences. Earlier in the text, authors wrote that annotators had to identify topic sentences per paragraph, then select the main topic sentence. I wonder how long an average paragraph may be in secondary school essays? Considering the Schaffer method (first example I have in mind, maybe not the best), a well written paragraph in is composed of 8 sentences. In light of this, considering the majority of the sentences in a paragraph as topic sentences is an unnecessary step.\n\nat 341, it would be more correct to cite the correct/actual model used (e.g. code-davinci-002, text-davinci-002, text-davinci-003). ChatGPT is an impelemntation of a chatbot based on those mdoels.\n\n468-474: There’s not a trend in those results. Best performances appear randomly across the three levels\n\nTable 8: Are numbers correct? Majority of the tested models do perform better with implicit relations with respect to explicit. It sounds odd\n\nThe formula for the inter annotator agreement at 854 reminds me the Jaccard Index, but as a sort of variant. Please provide references for it.\n\nFinally, I would like to remind the authors that Appendices are not a way to overcome length limitation. As it is stated in the ACL's paper formatting guidelines:\n\"Appendices are material that can be read, and include lemmas, formulas, proofs, and tables that are not critical to the reading and understanding of the paper.\" \n(for reference: https://acl-org.github.io/ACLPUB/formatting.html#appendices)\n\nAppendices E and F are not critical, but they represent another experiment. The author have to integrate it into the main body of the paper or cut it out.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces the Chinese Essay Discourse Coherence Corpus (CEDCC), a comprehensive dataset designed for assessing discourse coherence in learners' essays through multiple tasks. Unlike previous research that focused on specific aspects of coherence, the CEDCC addresses this gap by including evaluations in three tasks: discourse coherence grading, topic sentence extraction , and discourse relation recognition.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xLT1F8VABU", "reviewer": "Reviewer_FdLx", "summary": "", "questions": "- Question A: According to the explanation in Appendix A, the discourse relations that were introduced in this work seem to be undirected, while the existing discourse relations are typically directed. Why did the authors adopt such undirected relations? I think that such undirected relations would be unsuitable for detailed discourse analysis.\n\n- Question B: I could not find any mention of data release, but do you plan to release the data you have created?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point.\n\nResponse to reasons for rejection:\n1. In our study, we indeed conducted an in-depth analysis of the correlations among various research topics affecting discourse coherence. Specifically, in the \"Discussion\" section, we elaborated on the impact of coherence on discourse relation recognition and the significance of fine-grained annotations for automated essay scoring (L465-L507). Our dataset is multi-task in nature, and for each sub-task, we tested a range of models based on different methodologies and architectures, from rule/entity-based models to the latest generative LLMs. The appendix further discusses the interrelation and dependencies among the sub-tasks. Although our exploration isn't exhaustive, we firmly believe that such a dataset is invaluable for a comprehensive assessment of discourse coherence. Furthermore, this dataset holds significant value for future research, such as delving deeper into the correlations between the sub-tasks.\n2. Traditional topic continuity modeling primarily relies on annotating topic words. However, our choice to annotate topic sentences is due to the richer information they encompass compared to individual topic words. Furthermore, our dataset can indeed be applied to topic continuity modeling tasks. As shown in Table 7, the representative topic continuity modeling method, LDA, achieves commendable performance on our dataset, second only to the oracle method.\n3. Thank you for raising the concern regarding the absence of a dev set in our initial submission. Due to constraints related to data volume and the paper submission deadline, we initially reported results on the test set. We recognize the importance of evaluating on a separate validation set for a more objective assessment of model performance. We’re pleased to inform you that we have now constructed a dev set and have conducted evaluations on it. The results from this validation set, along with any relevant updates, will be incorporated into the final version of the paper upon acceptance. We believe that this will provide a more comprehensive and reliable evaluation of our proposed methods.\n\nResponse to questions:\n\n1. Regarding your point about undirected relations, I'd like to clarify. In our study, there's actually a hierarchical relationship between coarse-grained and fine-grained discourse relations, which is indeed directed, as depicted in Table 3. When annotators label the fine-grained relations between argument pairs, they naturally infer the corresponding coarse-grained relations.\n2. Yes, with the aim of fostering research and further analyses, we plan to release the dataset we've created in the near future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The dataset created in this work might be useful for researchers who are interested in Chinese discourse analysis.", "reasons_to_reject": "- The research motivation of this work come from the background in which the three relevant research topics (i.e., discourse coherence grading, topic transition modeling and discourse relation recognition) have been studied independently, but unfortunately, this work also followed the task independence. That is, the created data was only used to evaluate a single research topic without reference to the labels annotated for the other research topics.\nIn the view of the research motivation, it is rather important to investigate, for example, correlation/dependence of the three types of manually annotated labels to emphasize the usefulness of the created data. However, no such analysis has been done at all in this work. The paper just shows the frequency (or ratio) of the annotated labels from a single coherence-related research topic.\n\n- There is no sufficient explanation as to why sentence-level topic annotation was employed instead of the existing topic continuity modeling. Since topic continuity modeling is different from the sentence selection annotation as described in Section 2.2, it seems unreasonable to annotate the selected topic sentence as a replacement for topic continuity (or transition).\n\n- The authors split the annotated data into the training and test set only. This means that there is no development set, which is normally used to determine hyper-parameters when fine-tuning a pre-trained language model such as BERT and RoBERTa. For this reason, it is difficult to believe that each evaluation was done in a reliable manner.", "questions_to_authors": "- Question A: According to the explanation in Appendix A, the discourse relations that were introduced in this work seem to be undirected, while the existing discourse relations are typically directed. Why did the authors adopt such undirected relations? I think that such undirected relations would be unsuitable for detailed discourse analysis.\n\n- Question B: I could not find any mention of data release, but do you plan to release the data you have created?\n", "paper_topic_and_main_contributions": "In this work, the authors took issue with the independence of three kinds of coherence-related research topics (i.e., discourse coherence grading, topic continuity modeling and discourse relation recognition), and they focused on studying those three types of topics simultaneously. More concretely, \nthe authors first collected a Chinese essay data that consist of 501 essays written by Chinese middle school students, and then conducted three types of coherence-related annotations (discourse coherence grading, topic sentence selection and discourse relation annotation) on the essay data. The authors employed 3-level discourse grading (excellent, average and poor) for annotating the labels of discourse coherence grading. In addition, as for annotating discourse relations, the authors introduced their own discourse relations that consist of four types of coarse-grained relations and thirteen types of fine-grained relations instead of the existing discourse labels such as RST and PDTB. The authors also conducted experiments of automatic discourse coherence grading, topic sentence selection, and discourse relation recognition using the annotated data. For example, as for the evaluation of discourse grading, the authors compared the traditional discourse grading methods using entity grid or entity graph with BERT/RoBERTa-based methods. The authors also utilized ChatGPT to assess the grade of an input essay by giving an appropriate prompt to ChatGPT. In the evaluations of sentence selection, the authors compared the language model-based methods with the traditional methods (e.g., lead-3 and textrank). Finally, in discourse relation recognition, language model-based methods (BERT/RoBERTa/XLNet) were compared with ChatGPT-based methods.\n", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- On the Creation of a Corpus for Coherence Evaluation of Discursive Units.\n  Elham Mohammadi, Timothe Beiko and Leila Kosseim.\n  Proceedings of the Twelfth Language Resources and Evaluation Conference, pp.1067-1072, 2020.\n"}}, {"review_id": "YDRu8sw2eh", "reviewer": "Reviewer_i1co", "summary": "", "questions": "1. Will the dataset be released? I could not find any related statement.\n2. Is the score depicted in Figure 2 provided by the teacher? If so it may be beneficial to make this explicit on paragraph 3.1.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to questions:\n\n1. Yes, our dataset will be released shortly.\n2. Indeed, the scores in Figure 2 were provided by the teachers. Thank you for pointing this out; we will make it explicit in the subsequent versions of the paper.\n\nFurthermore, thank you very much for all the suggestions you have provided for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is interesting, the paper is well-written and easy to follow. The paper is self-contained. The authors provide clear motivations underlying the work. The paper introduces a novel multi-task benchmark that could be easily plugged in a framework designed to assess Large Language models so as to provide linguistically grounded intuitions on how such models  understand text coherence.", "reasons_to_reject": "I feel no reason to reject the paper.", "questions_to_authors": "1. Will the dataset be released? I could not find any related statement.\n2. Is the score depicted in Figure 2 provided by the teacher? If so it may be beneficial to make this explicit on paragraph 3.1.", "paper_topic_and_main_contributions": "The paper presents a novel benchmark for assessing discourse coherence in human-generated texts in Chinese language. In particular, the dataset consists of 501 essays of at least 600 words each collected from secondary school students. The authors, together with the novel dataset, introduced a three-fold labelling schema for multi-task setting: discourse coherence grade, that is each essay is provided with a coherence score considering the text as a whole; topic sentences, that is annotators are requested to identify the sentence encapsulating the major claim of each paragraph and the discourse relation, where annotators are requested to identify the relation intervening between paragraph and sentence pairs.\nFinally, the authors assessed several models in the three tasks: the Discourse Coherence Grading, where models are requested to provide the global coherence of essays; the Topic Sentence Extraction task identifies the primary sentence in each paragraph, encapsulating the main idea; and the Discourse Relation Recognition, where models are required to detect the semantic and logical relations both within sentences or paragraphs.\nThe results point out that no single model excels across all tasks and categories, advocating for a hybrid approach that leverages individual model strengths or applying task/category-specific optimizations.\nInterestingly, ChatGPT performance is low across all assessed tasks. Additionally, explicit discourse relations in inter-paragraph tasks underperformed compared to their implicit counterparts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please find some minor comments below:\n- please consider anticipating the extension of the acronym NLP, it has been used on line 024 for the first time, but has been extended on line 172;\n- please consider rewriting Firstly on line 080 as firstly;\n- please consider rewriting four on line 225 as three;\n- please consider adding a blank space before round brackets when referring to the literature in case such as for lines 251 (Rhetorical Structure Theory(Mann and\n252 Thompson, 1988)) and 253 (Penn Discourse Tree-\n253 Bank(Prasad et al., 2008));"}}]

Author rebuttal:

Meta-review: The paper reports on a new resource of Chinese middle school essays annotated for three tasks: discourse coherence (global coherence score), topic sentence extraction (identification of the main sentence per paragraph), and, discourse relation recognition (logical relations between sentences). The paper also reports on several experiments validating the corpus using diverse approaches.

Unfortunately, resource papers are not particularly "exciting" as reflected in the lukewarm scores by the reviewers. There seems to be a consensus that the paper needs some re-wording, more in-depth description, and potentially better argumentation, all of which is feasible within the time frame for the final version. In addition, two of the reviewers took issue with what is reported about the corpus construction. Details concerning basic descriptive statistics of the resource are missing and information about how the data was obtained or preprocessed is also missing, e.g, were the essays anonymized? (this last concern is not coming from the reviewers, but from me after perusing the paper). Last, reviewer2's suggestion about highlighting the correlation between the tasks--although somewhat severely worded--would indeed make for a stronger paper. The authors do point to some correlations but structure their discussion per task which reads a bit fragmented. 

All in all, I believe this is a valuable resource for the community and that most of the issues can be fixed by rewriting (as opposed to running experiments), hence my recommendation.
Decision: Accept-Main