TOREE: Evaluating Topic Relevance of Student Essays for Chinese Primary and Middle School Education (2024)

Abstract: Topic relevance of an essay demands that the composition adheres to a clear theme and aligns well with the essay prompt requirements, a critical aspect of essay quality evaluation. However, existing research of Automatic Essay Scoring (AES) for Chinese essays has overlooked topic relevance and lacks detailed feedback, while Automatic Essay Comment Generation (AECG) faces much complexity and difficulty. Additionally, current Large Language Models, including GPT-4, often make incorrect judgments and provide overly impractical feedback when evaluating topic relevance. This paper introduces TOREE (Topic Relevance Evaluation), a comprehensive dataset developed to assess topic relevance in Chinese primary and middle school students’ essays, which is beneficial for AES, AECG and other applications. Moreover, our proposed two-step method utilizes TOREE through a combination of Supervised Fine-tuning and Preference Learning. Experimental results demonstrate that TOREE is of high quality, and our method significantly enhances models’ performance on two designed tasks for topic relevance evaluation, improving both automatic and human evaluations across four diverse LLMs. 1

text: Findings of the Association for Computational Linguistics ACL 2024, pages 5749–5765 August 11-16, 2024 ©2024 Association for Computational Linguistics
Topic relevance of an essay demands that the composition adheres to a clear theme and aligns well with the essay prompt requirements, a critical aspect of essay quality evaluation. However, existing research of Automatic Essay Scoring (AES) for Chinese essays has overlooked topic relevance and lacks detailed feedback, while Automatic Essay Comment Generation (AECG) faces much complexity and difficulty. Additionally, current Large Language Models, including GPT-4, often make incorrect judgments and provide overly impractical feedback when evaluating topic relevance. This paper introduces TOREE (Topic Relevance Evaluation), a comprehensive dataset developed to assess topic relevance in Chinese primary and middle school students’ essays, which is beneficial for AES, AECG and other applications. Moreover, our proposed two-step method utilizes TOREE through a combination of Supervised Fine-tuning and Preference Learning. Experimental results demonstrate that TOREE is of high quality, and our method significantly enhances models’ performance on two designed tasks for topic relevance evaluation, improving both automatic and human evaluations across four diverse LLMs. 1

heading: 1 Introduction
text: Topic relevance of an essay requires that the composition not only revolves around a clear theme but also its content aligns with the requirements of essay prompt, which is a critical aspect in evaluating the quality of an essay, as shown in Figure 1. Automating the assessment of topic relevance not only effectively reduces teachers’ workload but also efficiently enhances students’ writing skills, which holds significant educational implications.
*Corresponding author. 1Our code and dataset are released at https://github.
com/cubenlp/TOREE.
Essay prompt: Let's play a "Guess Who" game! Choose a classmate and write an essay about him or her without mentioning names, but make sure that others can guess who you're writing about after reading your description. Who did you choose? What are some of his or her unique traits?
Guess Who He Is
My friend and I play together every day. He is a boy who rides the school bus with me and plays games with me, and I really like him. We were in neighboring classes in first and second grade, but in third grade, our classes are far apart. So, I see him less often, but I still remember his appearance and personality clearly. He has a round, chubby face, short hair, and always wears a blue school uniform. He is tall and big for his age. He shares my interest in iPads and cars, and he is very kind to his sister, being generous and magnanimous to everyone. After hearing all this, you should be able to guess who he is, right?
An attractive title
Too focused on private interactions! Revise the content to better capture the characteristics of the person.
Nice description of his physical features and personality traits.
Figure 1: An explanation of topic relevance for a grade 3 essay. Texts highlighted in red indicate weakness and provide detailed suggestions, whereas texts highlighted in blue point out parts with accurate topic relevance.
Current research on automatic essay evaluation predominantly focuses on the development of Automatic Essay Scoring (AES), which aims to determine holistic and trait-specific rankings, or discrete scores to essays by analyzing a range of textual features, such as coherence, grammatical correctness, and expressive clarity (Jiang et al., 2023; Song et al., 2020; Farag et al., 2018). However, existing research in this field, while considering topic as an aspect of evaluation, has neglected the importance of topic relevance (He et al., 2022) (See Appendix A for detailed explanations of these two aspects in Chinese essay evaluation). Furthermore, despite the efficiency in scoring offered by AES systems, it is the detailed comment and feedback that is particularly beneficial for students to directly refine their essays and strengthen their writing abilities.
Automatic Essay Comment Generation (AECG) (Liu et al., 2023) seeks to provide students with constructive feedback tailored to improve particular areas of their writing as delineated by the strengths and weaknesses highlighted in the comments, which is commonly adopted by teachers (Liu et al., 2017). Nevertheless, the intersection of complexity and difficulty in AECG results in a
5749
这篇作文的中心与切题方面表现良好。作者 选取了一个具体的人物——尤唯，通过描写 她的品质和行为，深刻地展现了她的精神风 貌。尤唯是一个品学兼优、尊敬师长、文静 聪慧的女孩，她的友谊和帮助让作者感动。 作者通过描写自己与尤唯的互帮互助，表达 了对友谊的珍视和对未来的期许。整篇作文 紧扣主题，情感真挚，给人以启发和感动。
The essay excels in focus and relevance. The writer depicts Yu Wei, a character chosen for her vivid spirit shown through qualities and actions. Yu Wei is presented as academically and morally excellent, respectful, gentle, and intelligent. Her friendship and assistance touched the writer, who reflects on mutual support, valuing friendship, and hope for the future. The piece stays on point with sincere emotions, inspiring and moving readers.
这篇作文的中心与切题表现优秀。作者选择 了一位品学兼优、乐于助人的女孩作为写作 对象，通过描写她的品质和行为，表达了对 她的敬佩和感激之情。同时，文章也强调了 友谊的重要性，以及互帮互助的精神。从切 题的角度来看，这篇作文完全符合题目要求， 以一个具体的人物为例，深入挖掘其精神品 质和代表的风气，并表达了作者的情感倾向。 因此，我认为这篇作文的中心与切题方面都 做得很好。 The essay excels in focus and relevance. The author chose a girl who excels academically and is eager to help others as the subject for the essay, using descriptions of her qualities and actions to convey admiration and gratitude. The piece also highlights the significance of friendship and the spirit of mutual assistance. From the perspective of relevance, the essay fully meets the requirements, using a specific individual to delve into her moral fiber and the ethos she represents while expressing the author's emotional stance. Hence, I believe this essay excels in both focus and relevance.
这篇作文在中心和切题方面不合格。文章的 题目是《这样的人让我感动》，应该侧重于 写让作者感动的人的事迹，进而表现这个人 的性格、品质等特征。本文却从友谊说起， 讲二人之间是如何成为朋友，发展友情的， 与题目要求不符。建议作者在写作前仔细审 题，围绕中心组织事实材料。
The essay fails to meet the requirements in relevance and focus. The title is "Such People Touch Me" which should concentrate on the deeds of the person who moved the author, thereby showcasing their personality and qualities. However, the essay starts with friendship, discussing how two people become friends and develop their relationship, deviating from the topic. The author is advised to carefully review the question before writing and organize facts around the central theme.
EduChat
(a)
GPT-4
(b)
Teacher (c)
Figure 2: An example of topic relevance comment for one grade 7 student essay titled Such People Move Me based on EduChat, GPT-4 and a Chinese primary school teacher. More details of this example are shown in Appendix B. Texts highlighted in red indicate misinformation or incorrect advice, whereas texts highlighted in blue provide accurate feedback for students. Specifically, in part (a), EduChat incorrectly assesses the level as "excel" and fabricates a character named Yu Wei that does not exist; in part (b), GPT-4 inaccurately evaluates the essay’s level as "fully meets" and fails to identify the specific "author’s emotional stance"; in part (c), teacher-written comment correctly points out the topic derivation of this essay, explains why, and provides detailed suggestions for refinement.
paucity of research within Chinese essay evaluation. Zhang et al. (2022) represents a limited foray into this area, concentrating on comment generation for Chinese narrative essays at the sentence and paragraph levels for student cohorts in the fifth grade and beyond. However, it neglects the diverse requirements posed by various writing tasks and is confined to a singular literary genre. Recently, Large Language Models (LLMs) have been exhibiting considerable promise across various applications (Zhao et al., 2023). Nonetheless, these models, including GPT-4 2 and domain-specific iterations like EduChat (Dan et al., 2023), often make incorrect judgements regarding the topic relevance and generate overly generic or even misleading feedback when tasked with assessing topic relevance of essays, which is related to hallucination of LLMs, as shown in Figure 2.
To address these issues, we introduce a novel dataset TOREE (Topic Relevance Evaluation) fo-
2https://chat.openai.com
cusing on evaluating topic relevance of essays written by Chinese primary and middle school students. To the best of our knowledge, TOREE offers three primary advantages over existing works in this field. Firstly, TOREE is the first attempt to evaluate Chinese essays based on their adherence to topic relevance, which is more comprehensive, considering both the objectives of the writing assignment and the substance of the essay, rather than focusing solely on content. Secondly, TOREE encompasses a broad range of participants, specifically targeting students in the compulsory education stage of China, which includes grades 3 through 9. This span captures 13 distinct literary genres and covers 74 diverse essay prompts, thereby providing insights into student writing abilities across different styles and levels. Thirdly, TOREE is constructed as a comparison dataset that is apt for contrasting and evaluating models through both Supervised Fine-tuning (SFT) and Preference Learning (PL) (Rafailov et al., 2023) like Reinforcement Learning from Human Feedback (RLHF), making it highly
versatile for AES, AECG, and other various applications. Two tasks based on TOREE are designed for a comprehensive evaluation: Topic Relevance Recognition and Topic Relevance Comment Generation. Moreover, we propose a twostep method combining SFT and PL to elicit abilities of open-source LLMs to identify topic relevance level and generate specific and constructive comments. Experimental results suggest that our method effectively enhances models’ abilities on two tasks, improving both automatic and human evaluations across 4 diverse LLMs. To sum up, our contributions can be summarized as follows.
• We introduce TOREE, a novel comprehensive dataset for evaluating topic relevance of essays, which is beneficial for AES, AECG, and other applications.
• TOREE can be utilized through both SFT and PL. Open-source LLMs that have been trained with our method demonstrate a substantially improved ability to both identify topic relevance level and generate specific feedback.
• TOREE is a valuable resource for further exploration from the perspective of PL.

heading: 2 Related Work

heading: 2.1 Automatic Essay Scoring
text: The goal of AES is to predict holistic or traitspecific rankings, or numerical scores to essays by analyzing a range of textual features, such as coherence, grammatical correctness, and expressive clarity (Jiang et al., 2023; Song et al., 2020; Farag et al., 2018). Earlier research has utilized traditional Machine Learning (ML) methods like Support Vector Machine (SVM) (Jin et al., 2018). Deep Learning based methods include using CNN (Dong et al., 2017), LSTM (Tay et al., 2018) to predict scores, and more recently, focus has shifted to small language models (SLMs) like BERT (Yang et al., 2020; Mayfield and Black, 2020; Cao et al., 2020; Uto et al., 2020) and more complicated attentionbased design (He et al., 2022) for this purpose. Scoring certain essay traits seeks to attribute different scores to various characteristics of an essay such as clarity (Ke et al., 2019) and style (Mathias and Bhattacharyya, 2018). However, AES systems mainly provide convenience for educators but offer little to no feedback for students to improve their essays. In this study, our objective is not only
predicting scores, but also generating constructive topic relevance feedback with comprehensive advice on how to enhance essays to revolve around a certain them and more closely match the requirements of given essay prompts.

heading: 2.2 Automatic Essay Comment Generation
text: Despite the myriad of existing studies (Raheja et al., 2023; Dwivedi-Yu et al., 2022; Shi et al., 2022) and the availability of tools designed to aid writers, such as Grammarly 3 and current LLMs, these resources predominantly offer superficial assistance, which are adept at addressing grammatical inaccuracies or enhancing stylistic elements but fall short in enhancing the topic relevance of an essay. AECG targets at generating effective feedback for different students to purposefully revise their essays, thereby improving writing skills. To date, there has been a scarcity of research in this area, owing to its inherent complexity and the challenges involved. Zhang et al. (2022) leveraged a plan-based approach to generate feedback using the T5 model (Raffel et al., 2020), which was designed to evaluate Chinese narrative essays by pinpointing their strengths and weaknesses. Recent advancements in LLMs have led to the emergence of educationfocused LLMs which hold significant potential for educational applications, such as EduChat (Dan et al., 2023). However, the feedback provided by such models is often too general to aid students in specifically improving their essays or may even include flase information, potentially misleading students. Currently, ASAP 4 and TOEFL11 (Blanchard et al., 2013) are two most frequently used public datasets within the Automatic Essay Evaluation field. However, both datasets merely contain 8 essay prompts. In contrast, our research examines essay evaluations for a broader range of students, specifically those in the entire compulsory education stage, which includes grades 3 through 9, spans 13 different literary genres and covers 74 diverse essay prompts. Furthermore, we aim to provide more detailed feedback that can offer students clear, actionable advice on how to enhance their essays in a manner tailored to each individual.
3https://www.grammarly.com 4https://www.kaggle.com/c/asap-aes/data

heading: 3 The TOREE Dataset

heading: 3.1 Task Definition
text: We designed two tasks according to the levels of comprehension and generation. The objective of Task 1 is to identify the topic relevance level of an essay, while the goal of Task 2 is to generate specific and detailed comments for students.
Task 1: Topic Relevance Recognition Task 1 is designed as a 5-category classification task, mapping the topic relevance of an essay to one of five levels: Excellent, Good, Fair, Passable, and Failed. The specific criteria for these levels were designed and discussed by professional annotators with extensive Chinese teaching experience in our annotation team, with details provided in Appendix C.1.
Task 2: Topic Relevance Comment Generation The focus of Task 2 is pointing out the areas of the essay that effectively stay on topic, while also offering specific and detailed suggestions for revising essays to enhance relevance to the essay prompt, such as assistance with modifying the title, adjusting the narrative order, rewriting certain parts, etc.
Data Format Taking into account the use of SFT to foster the ability to generate comments, as well as the utilization of PL to better align with the preferences of educators, the structure of TOREE is articulated as a five-fold combination, comprising (instruction, essay, topic_relevance_level, chosen_comment, rejected_comment). Within the instruction field, the essay prompt and the grade of essay’s author are provided. The essay field includes both the title of the essay and its body content and the topic_relevance_level field is the annotated topic relevance level defined in Task 1. Furthermore, the chosen_comment and rejected_comment form a pair of preference data. More details about the preference are provided in Appendix C.2.

heading: 3.2 Dataset Construction
text: Essay Collection We gathered 730 original essays from the regular writing exercises of primary and middle school pupils and sorted by genre and grade. Through the implementation of rule-based filtering operations, we ensured the quality of essays to some extent. As a result, we ended up with a collection of 683 essays, with the distributions depicted in Figure 3. More details of our filtering rules are shown in Appendix C.3.
LLM Generation We selected Spark5 to generate topic relevance comments because of its relatively notable strengths in the field of Chinese education. In particular, we provided it with an essay prompt along with specific requirements for assessing topic relevance, which is shown in Appendix D. We chose Spark-v3 for generation and further details are provided in Section 5.3.2. Although Spark is adept at evaluating essays compared to other LLMs, the comments it produces still face hallucination problems and cannot be directly used for guiding students. Therefore, we put these comments in the rejected_comment field.
Human Annotation To start, we selected our potential annotators from a group of 20 individuals majoring in Chinese studies, ensuring they had at least an undergraduate degree and skills in reading and writing Chinese text. Moreover, following HaluEval (Li et al., 2023), these candidates were asked to annotate a few test samples for pre-annotation. We assessed their reliability using Fleiss’s Kappa κ for Task 1 and BERTscore (Zhang et al., 2019) for Task 2. We chose the top 6 performers as our final annotators. For Task 1, all annotators labeled the topic relevance level and we obtained κ = 0.812 (0.80 ≤ κ ≤ 1.00), showing a perfect agreement. For Task 2, within the annotation team, we divided the responsibilities, assigning three annotators to evaluate essays from primary school students in grades 3 to 6 and another three to assess essays from middle school students in grades 7 to 9. Consequently, we have
5https://xinghuo.xfyun.cn/
(a)Essay grade distribution.
(b)Essay genre distribution.
included the superior comments made by our annotators in the chosen comment field. More details of annotation are provided in Appendix C.4.

heading: 3.3 Dataset Statistics
text: As depicted in Figure 3, TOREE encompasses a comprehensive range of essay topics typically assigned to students within the seven grades of compulsory education stage in China, which includes 683 essays, spanning across 13 different literal genres and 74 diverse prompts. Notably, the curriculum for the first and second grades of primary school mainly focuses on learning characters and writing sentences, rather than writing essays. Detailed descriptions of genres and prompts are shown in Appendix C.5. Additionally, we divide TOREE into a training, validation, and testing set in a proportion of 8:1:1, with the details in Table 1.

heading: 3.4 Ethical Issues
text: We paid all the data annotators for their contributions. Additionally, we obtained clear consent from the essay authors and their legal guardians to use the essays for the purposes of labeling and sharing the work publicly. To protect the students’ privacy, we have removed any personal information such as names and addresses from the essays in our dataset.

heading: 4 Method
text: We propose a two-step method for recognizing topic relevance levels and generating corresponding topic relevance comments in a joint format.

heading: 4.1 Step 1: Supervised Fine-tuning
text: In the first step, we apply Supervised Fine-tuning for eliciting LLMs’ abilities to recognize levels and comment on the topic relevance aspect of an essay. Specifically, the training objective is to minimize the standard cross-entropy loss for auto-regressive generation, defined as:
LSFT = − 1
B
B∑
i=1
log pi(yi|xi) (1)
where B denotes the batch size, yi denotes the concatenation level ⊕ chosen_comment, and xi denotes the concatenation instruction⊕ essay.

heading: 4.2 Step 2: Preference Learning
text: We incorporate DPO (Rafailov et al., 2023) for further alignment with human comments. Specifically, we utilize LLMs’ intrinsic abilities to learn from preference comments, similar to RLHF. The training objective in this step is defined as:
LDPO = −E [ log σ ( β log πθ(y c i |xi)
πref(yci |xi)
−β log πθ(y r i |xi)
πref(yri |xi)
)] (2)
where E denotes the expectation over a batch, σ denotes Sigmoid function, πθ is the model undergoing training, πref is the frozen reference model (trained after SFT) and β is a hyperparameter controlling the deviation from the reference model.
Similar to Equation 1, xi is the concatenation instruction ⊕ essay, yci is the chosen comment concatenation level ⊕ chosen_comment, and yri is the rejected comment concatenation level ⊕ rejected_comment in the data point.

heading: 5 Experiment

heading: 5.1 Experimental Setup
text: Models We consider a broad range of methods in Automatic Essay Evaluation, including ML methods, SLMs, and LLMs. For ML methods, we selected RankSVM (Jin et al., 2018) and RandomForest for Task 1. For SLMs, we selected BERT (Mayfield and Black, 2020), XLNet (Yang et al., 2019), and RoBERTa (Liu et al., 2019) for Task 1. Regarding LLMs, we selected both open-source and closed-source LLMs that are currently supportive of Chinese language and are prevalently utilized in current research for both Task 1 and Task 2. For open-source models, we utilized Baichuan2-7B (Yang et al., 2023), ChatGLM3-6B (Du et al., 2022), Chinese-Alpaca2-7B (Cui et al., 2023), Qwen-7B (Bai et al., 2023), and EduChat (Dan et al., 2023). In terms of closed-source models, our selection comprised of ChatGPT (gpt-3.5turbo-0613), GPT-4 (gpt-4-turbo-preview), Claude (claude-3-sonnet-20240229) and Spark.
Evaluation Metrics For Task 1, the metrics utilized to evaluate performance are Precision, Recall, and Weighted-F1, while for Task 2, we use both automatic evaluation scores and human evaluation because of the complexities associated with this task. In terms of automatic evaluation scores, we select BLEU, ROUGE, and BERTscore (Zhang et al., 2019) for gauging the similarity in terms of both syntax and meaning between the generated comment and the reference comment (the chosen_comment field in a data point). For human evaluation, we incorporate the expertise of three annotators in our team to compare generated comments from the perspective of real-world educational applications. They evaluate pairs of comments for identical compositions, rendering a result of Win/Tie/Lose for each pair and the final result is obtained through majority vote.

heading: 5.2 Implementation Details
text: For testing, we compute Precision, Recall, and Weighted-F1 for ML and SLM methods under zeroshot setting on Task 1 and compute automatic evaluation scores for LLMs under few-shot setting on
Task 1 and Task 2. Notably, given that context length of LLMs is limited and each essay is relatively lengthy, we chose 2 shots for test. Regarding SFT and PL training, we adopted LoRA (Hu et al., 2021) with the LoRA rank of 8 and the dropout rate of 0.1 across all 3 epochs. All SFT runs were performed on a single NVIDIA RTX 3090 GPU, while DPO runs were executed on a single A100 GPU. Given the nature of the TOREE (shown in Table 1), we limited the maximum length to 1024 tokens. Training configurations include the learning rate of 5 × 10−5, the batch size of 2, and the employment of gradient accumulation with a step size of 2 across three epochs. In addition, we implemented a Cosine learning rate scheduler without the inclusion of warm-up steps and enable mixed precision training (fp16) to enhance efficiency.

heading: 5.3 Results and Analysis

heading: 5.3.1 Topic Relevance Recognition
text: The complete results are shown in Appendix E.
Poor Performance of Current LLMs As shown in left part of Table 2, the results reveal that both closed-source and open-source LLMs exhibit deficiencies in accurately assessing the topical relevance of essays, a performance lag evident when compared to SLMs and traditional ML methods. For example, the state-of-the-art GPT-4 achieves 36.29 and 43.07 Weighted-F1 under 2-shot and zero-shot settings, which is 4.36 lower than BERT. Furthermore, performance metrics for other opensource LLMs consistently fall below a Weighted-F1 score of 30. In contrast, our method markedly enhances the ability of LLMs to determine the topic relevance of essays. Specifically, the Weighted-F1 score for the Baichuan2-7B significantly improved from 20.31 to 42.86. This score not only demonstrates a substantial increase but also exceeds the GPT-4 under zero-shot scenario by 6.57 points.
Effect of Few-shot Prompting The effect of Few-shot Prompting on various models differs on Task 1. For closed-source LLMs except Sparkv2, few-shot prompting brings Weighted-F1 score gains up to 15.17 (Claude), while for open-source LLMs except EduChat, few-shot prompting brings huge performance drop on Weighted-F1 score up to 21.19 (Qwen-7B). As a common belief, fewshot prompting can enhance LLMs’ abilities on various downstream tasks by providing additional information that helps LLMs make decisions. We
hypothesize that closed-source LLMs and EduChat (which is specifically pre-trained on Chinese textbook corpus from education domain) have stronger Chinese comprehension capabilities, thereby enhancing Weighted-F1 score on Task 1.
Error Analysis To delve deeper into the predictions of LLMs and better understand our method, we present confusion matrices on Task 1 in Figure 4. Notably, GPT-4 exhibits a strong tendency to predict the test data into higher levels, with almost all its predictions concentrated in the top three categories. We hypothesize that this problem may be a form of sycophancy (Sharma et al., 2023) in the domain of essay assessment, a specific type of hallucination (Zhang et al., 2023), which can lead to an overly optimistic evaluation of student essays. Moreover, the performance of Baichuan27B is remarkably poor, failing to successfully predict any of the five categories in the vast majority of cases. This suggests that current vanilla open-
source LLMs may have significant limitations or deficiencies in handling task 1. Compared to GPT4, Baichuan2-7B-SFT shows a slight mitigation of sycophancy, but the improvement is not substantial. On the other hand, Baichuan2-7B-SFT-DPO effectively alleviates the sycophancy issue, resulting in a higher number of correct predictions and a more balanced distribution of predicted labels across categories, with fewer overly optimistic predictions towards higher grades. More explanations of error analysis on Task 1 are provided in Appendix F.

heading: 5.3.2 Topic Relevance Comment Generation
text: The right part of Table 2 presents automatic evaluation scores of evaluated LLMs on Task 2.
Automatic Evaluation Scores Drop of Few-shot Prompting Unlike Task 1, Few-shot prompting leads to a decline in all automatic evaluation scores on both open-source and closed-source LLMs across almost all models. Similarly, we surmise that models’ ability to process long texts and
comprehend essays is limited and Task 2 is too challenging. Consequently, these models struggle to differentiate between the essay to be evaluated and the demonstration essays provided, leading to some of the generated comments being influenced by the example essays and commenting on the wrong target. We believe that the example essays provided as in-context demonstrations introduced a certain degree of "noise" in this context.
In-Depth Analysis of DPO As shown in Equation 2 in Step 2, LLMs further learn to align with human through preference learning. We delved into the impact of varying preference data on the performance on Task 2. Specifically, in light of the results presented in Table 2, the capability of three versions of Spark to generate topic relevance comments has progressively improved. Furthermore, considering the cost associated with closed-source model APIs, we constructed three distinct preference datasets using these three models. It can
be inferred that the discrepancy between Spark-v3 comments and human annotations is the minimal, followed by Spark-v2, with Spark-v1 exhibiting the largest gap. The results are shown in Table 3. Our findings indicate that all models performed optimally in the scenario of h-v1, as evidenced by the highest scores of BLEU-4, ROUGE-L, and BERTscore. Furthermore, aggregating all preference data results in a decline in performance across all models, which may originate from the presence of multiple preferences for the same essay. Consequently, it can be inferred that the larger the disparity between the chosen comments and rejected comments within preference data, the more the model is able to learn, ultimately resulting in more effective comment. We believe this insight is helpful for both automatic essay evaluation and other domains involving preference learning.
Human Evaluation We conducted human evaluation experiments on comments generated by models after training with SFT and models after training with SFT and DPO, setting GPT-4 under 2-shot setting as baseline. As shown in Table 4, our method significantly enhances the model’s capability to generate topic relevance comments, with outcomes that are markedly superior to GPT-4 from the perspective of real-world applications. For instance, Baichuan2-7B-SFT-DPO achieved a win rate of 80.6% against GPT-4, and the win rates for other models did not fall below approximately 70%. Furthermore, the ablation results for all models (DPO vs SFT) also demonstrate the efficacy of our two-
step approach: the generated comments are more aligned with human-written feedback and possess greater practical value.

heading: 6 Case Study
text: As shown in Table 5, regarding this off-topic essay, GPT-4 mistakenly considers its central theme and relevance to be excellent. Although it provides a detailed suggestion describing the author’s specific work, the overall direction of the feedback is misleading to the student. In comparison, Baichuan27B-SFT offers recommendations instead of vague descriptions, while Baichuan2-7B-SFT-DPO identifies specific issues and provides executable suggestions ("review the essay prompt again thoroughly and ensure your essay focuses on the exploration of characters"), which is more closely resembling teacher comments.

heading: 7 Conclusion
text: In conclusion, this paper introduces TOREE, a novel comprehensive dataset for assessing the topic relevance of student essays, addressing a critical need in educational assessment. TOREE serves as a valuable resource for future research and development in the field of automatic essay evaluation, which can be further utilized through Supervised Fine-tuning or Preference Learning. Moreover, our two-step approach has demonstrated success in enhancing the performance of open-source Large Language Models in recognizing topic relevance and providing constructive feedback, presenting great value in real-world applications.
Limitations
The main limitation of the study is the relatively limited number of essays in the TOREE dataset, which is a consequence of the challenges associated with data acquisition. Obtaining a diverse and representative sample of student essays, particularly from the compulsory education stage, requires extensive collaboration with educational institutions and adherence to privacy regulations. Despite these difficulties, the dataset’s quality and the tasks designed around it provide a solid foundation for future research and model development in the field of automatic essay evaluation.

heading: Acknowledgment
text: We appreciate the support from National Natural Science Foundation of China with the Main Research Project on Machine Behavior and HumanMachine Collaborated Decision Making Methodology (72192820 & 72192824), Pudong New Area Science & Technology Development Fund (PKX2021-R05), Science and Technology Commission of Shanghai Municipality (22DZ2229004), Shanghai Trusted Industry Internet Software Collaborative Innovation Center and East China Normal University International Conference Grant Programme. We express sincere gratitude to Zhengyi Yu for his kind help to this work.

heading: A Topic and Topic Relevance
text: For an essay, the topic refers to whether the essay has a clear central focus, without considering the relevance of that focus to the prompt requirements. In contrast, the topic relevance of an essay refers to whether the essay’s central focus aligns with the prompt requirements, building upon having a central focus in the first place. In other words, previous work only focused on an essay’s central topic itself, whereas TOREE focuses on the alignment between an essay’s central topic and the prompt requirements (especially when the prompt constrains the genre of the essay). For example, an essay prompted to describe a scene would be considered off-topic/off-prompt even if it had excellent character descriptions. Given this, our work is the first to propose focusing on topic relevance when evaluating Chinese essays. Our primary motivation is to help students avoid writing essays that are irrelevant to the prompt, bridging the gap by evaluating topic relevance, aiming to improve the overall quality and responsiveness of student essays.

heading: B Details of Example
text: Full details of the example shown in Section 1 are shown in Figure 5.

heading: C More Details of TOREE
text: C.1 Topic Relevance Level
Details of topic relevance level in Task 1 are shown in Table 6. This division is discussed and determined by professional annotators for the following two primary reasons.
Correspondence with the Five-Level Evaluation Standard in Compulsory Education Stage This grading system aligns with the existing five-tier assessment criteria in the Chinese Language curriculum during the compulsory education phase. Teachers and students are more familiar with this structure, thereby enhancing their understanding and acceptance of the evaluation process.
Fine-Grained Assessment for Enhanced Feedback and Improvement The classification into five distinct levels offers a more nuanced approach to grading, allowing teachers to provide more detailed and accurate assessments of student work. In turn, students can utilize these specific grades to pinpoint areas for improvement and refine their
essays accordingly, leading to a clearer focus on topic relevance and overall writing quality.
C.2 Preference in Comments
Specifically, to meet the practical needs of Chinese language teachers in primary and middle schools and to help students improve their composition skills, we selected the preference in the TOREE that can be summarized as: whether the comment is specific and on-topic/on-prompt, whether it is targeted, and whether it provides feasible and executable suggestions for revision. For a pair of comments, we denote the one that better aligns with the above preference as the chosen_comment, and the one that deviates from the preference as the rejected_comment. Almost all comments generated by current state-of-the-art large language models, including those in the education domain such as Spark and EduChat, suffer from being too generic and vague, lacking specificity, and even exhibiting hallucinations, rendering them inapplicable for directly guiding students’ composition revisions. Therefore, we treat comments annotated by human experts as chosen_comment and model-generated comments as rejected_comment. A specific example is shown in Table 8.
C.3 Essay Filtering
We adopted a two-fold filtering strategy. Firstly, we removed all essays manifesting substantial discrepancies due to the OCR processing errors, which significantly differ from the original ones. Subsequently, we removed all compositions exceeding 700 words or consisting of fewer than 200 words, which fail to satisfy the fundamental prerequisites of essay assignments.
C.4 Annotation
Annotating the topic relevance level and comment is a very challenging task, which requires good Chinese reading comprehension skills and using encouraging words to identify both the strong and weak points of an essay, as well as to provide constructive feedback for enhancing topic relevance. A screenshot of the annotation system is shown in Figure 6.
Team Composition and Selection The annotation team consisted of 6 selected individuals with backgrounds in Chinese language studies. These annotators were chosen based on their performance in a pre-annotation test, ensuring they possessed
题目: 生活中我们会遇到各种各样的人，有的 让你尊敬，有的让你佩服，有的让你感动，有 的让你叹息。以《这样的人让我_____》为题， 写一篇作文。不少于500字。“这样的人”可 以是你熟悉的人，比如你的同学、邻居，也可 以是陌生的人，可以是某个具体的人，也可以 是某一类人。“这样的人”应该是具有某种精 神品质，或代表某种风气的人。要着眼于个性、 品质去描写，写出你的情感倾向。
这样的人让我感动
如果说友谊是一颗常青树那么，浇灌它的必定 是出自心田的清泉；如果说友谊是一朵开不败 的鲜花，那么，昭耀它的必定是心中升起的太 阳。初一新开学，几次摸底考，就有一个女孩 尤唯特殊，常常考试第一名，尊敬师长，一个 文静且聪慧的女孩，就是这样一个品学兼优的 人，却被我开朗的性格所打动，与我成为了好 朋友。我的成绩不太理想，算的上是班级的中 后好，时常就感到自卑，我向她吐露心声。也 许在那一刻，她决定帮助我。她不惜去除自己 课间时间，来为我巩固课上认知识，带我做回 家后整理一天中所错的题。后来我突然现，她 体育不好，但这是我的强项啊于把中午所有的 作业时间分割成她教我学习，监督她的体育， 我们互帮互助，取长补短，不美哉！在我们的 不懈努力之下，我们各自成绩都提高了不少。 因为，在我们互相帮助时，即帮助了他人，又 帮助了自己。希望我们的友谊能够天长地久。 学校的走廊，永远是我们吹着晚风，谈论未来 的小角落，长长的走廊，睹所有人的青春继续 向前走吧，要相信彼方上有荣光在！
这篇作文的中心与切题方面表现良好。 作者选取了一个具体的人物——尤唯， 通过描写她的品质和行为，深刻地展 现了她的精神风貌。尤唯是一个品学 兼优、尊敬师长、文静聪慧的女孩， 她的友谊和帮助让作者感动。作者通 过描写自己与尤唯的互帮互助，表达 了对友谊的珍视和对未来的期许。整 篇作文紧扣主题，情感真挚，给人以 启发和感动。
Prompt: In life, we meet all kinds of people: some command respect, others admiration, some touch our hearts, and some make us sigh. Write an essay titled "Such People Move Me," using at least 500 words. "Such people" may be someone you know, like a classmate or neighbor, or someone unfamiliar. It could be a specific person or a type of person embodying a particular spirit or representing a trend. Focus on their individuality and qualities to describe your emotional inclination.
Such People Touch Me
If friendship is an evergreen tree, then the sincere spring water from the heart must nourish it; if friendship is an everlasting flower, then it must be illuminated by the sun that rises in the heart. At the beginning of the first year, after several diagnostic tests, one girl stood out as special, consistently ranking at the top, respecting her teachers—a quiet and intelligent girl with both academic and moral excellence. This girl was moved by my outgoing personality and became good friends with me. My grades weren't great, and I often felt inferior, but I confided in her. Perhaps at that moment, she decided to help me. Sacrificing her break time, she helped me consolidate my knowledge and went through the questions I got wrong. Later, I realized that although she struggled with sports, it was my strength. So, we divided our lunch break into study time for her to tutor me and physical education time for me to coach her. Helping each other and learning from each other's strengths was a beautiful thing! With our unwavering efforts, both our grades improved significantly. When we helped each other, we not only assisted one another but also helped ourselves. I hope our friendship will last forever. The school corridor will forever be our little corner where we talk about the future in the evening breeze. As we all continue to move forward through our youth, let's believe there's glory ahead!
The essay excels in focus and relevance. The writer depicts Yu Wei, a character chosen for her vivid spirit shown through qualities and actions. Yu Wei is presented as academically and morally excellent, respectful, gentle, and intelligent. Her friendship and assistance touched the writer, who reflects on mutual support, valuing friendship, and hope for the future. The piece stays on point with sincere emotions, inspiring and moving readers.
作者选择了一位品学兼优、乐于助人 的女孩作为写作对象，通过描写她的 品质和行为，表达了对她的敬佩和感 激之情。同时，文章也强调了友谊的 重要性，以及互帮互助的精神。从切 题的角度来看，这篇作文完全符合题 目要求，以一个具体的人物为例，深 入挖掘其精神品质和代表的风气，并 表达了作者的情感倾向。因此，我认 为这篇作文的中心与切题方面都做得 很好。
The author chose a girl who excels academically and is eager to help others as the subject for the essay, using descriptions of her qualities and actions to convey admiration and gratitude. The piece also highlights the significance of friendship and the spirit of mutual assistance. From the perspective of relevance, the essay fully meets the requirements, using a specific individual to delve into her moral fiber and the ethos she represents while expressing the author's emotional stance. Hence, I believe this essay excels in both focus and relevance.
这篇作文在中心和切题方面不合格。 文章的题目是《这样的人让我感 动》，应该侧重于写让作者感动的 人的事迹，进而表现这个人的性格、 品质等特征。本文却从友谊说起， 讲二人之间是如何成为朋友，发展 友情的，与题目要求不符。建议作 者在写作前仔细审题，围绕中心组 织事实材料。 Human
The essay fails to meet the requirements in relevance and focus. The title is "Such People Touch Me" which should concentrate on the deeds of the person who moved the author, thereby showcasing their personality and qualities. However, the essay starts with friendship, discussing how two people become friends and develop their relationship, deviating from the topic. The author is advised to carefully review the question before writing and organize facts around the central theme.
EduChat
GPT-4
(a)
(b)
(c)
Figure 5: An example of generating topic relevance comment for grade 7 student essay titled Such People Move Me using EduChat and GPT-4. Human denotes human-written comment for preference. Text highlighted in bold are points requiring special attention for students. The right side consists of comments generated through three models. In each side, the right part is the translated English version for better understanding. Texts highlighted in red indicate misinformation or incorrect advice, whereas texts highlighted in blue provide accurate feedback or suggestions for students. Specifically, in part (a), EduChat incorrectly assesses the level as "excel" and fabricates a character named Yu Wei that does not exist; in part (b), GPT-4 inaccurately evaluates the essay’s level as "fully meets" and fails to identify the specific "author’s emotional stance"; in part (c), human-written comment correctly points out the topic derivation of this essay, explain why, and provides detailed suggestions on how to refine it.
strong Chinese reading and writing skills. The selection process involved evaluating their test samples with BERTscore to determine their ability to understand and annotate text accurately.
Training and Division of Responsibilities Before beginning the actual annotation work, the team underwent a training session to harmonize their understanding of the task’s requirements and the annotation guidelines. This training ensured consistency across annotations. The team was then divided based on the educational level of the essays they would annotate: Three annotators focused on essays from primary school students (grades 3 to 6) and the remaining three handled essays from middle school students (grades 7 to 9). For each essay, one annotator was primarily responsible for the annotation, another served as a reviewer to ensure accuracy, and a third annotator was involved in case of significant discrepancies between the first
two opinions.
Annotation Process Duration The duration of the annotation process can vary widely depending on the volume of text to be annotated, the complexity of the task, and the team’s efficiency. In our structured project, the entire process of annotation spanned a duration of one month.
Compensation Compensation was provided to each annotator based on the actual volume of annotation work completed in a fair manner.
C.5 Explanation of Essay Genres and Prompts
As shown in Table 7, an explanation of 13 distinct literary genres are listed. Moreover, a complete list of 74 diverse essay prompts is shown in Table 9.

heading: D Prompt Template
text: Our prompt for two tasks is shown in Figure 7.

heading: E Full Results
text: Full results of automatic evaluation scores are displayed in Table 10.

heading: F More Explanations on Error Analysis
text: As shown in Figure 4, we present the confusion matrices on Task 1.
• For GPT-4, its performance in predicting excellent is very good. However, for the remaining four levels, the predictions are generally overestimated, concentrated in the first column of the confusion matrix.
• For Baichuan2-7B, it can hardly make suc-
cessful predictions, with a large number of predicted labels concentrated in the sixth column.
• For Baichuan2-7B-SFT, its prediction performance is better than Baichuan2-7B.
• For Baichuan2-7B-SFT-DPO, the prediction results are significantly more concentrated around the diagonal, and the over-prediction of higher-level labels is alleviated to some extent.
Data field Content
instruction
There is a Chinese proverb, "Those who are close to vermillion are red, and those who are close to ink are black", which emphasizes the impact of environment on human growth. What do you think of it? Please make up your own mind, draw up your own topic, and write a refutation paper. No less than 600 words. Tips: 1. Sort out the way of writing the refutation, choose a suitable angle to refute, and at the same time show that your point of view is reasonable. 2. Choose materials that are consistent with your point of view. It is best to have factual arguments and reasonable arguments. 3. Make an outline based on your opinions and materials, communicate with classmates, supplement each other’s arguments, and complete the composition on this basis.
essay
Title: Influence Content: There is a Chinese proverb, "Those who are close to vermillion are red, and those who are close to ink are black." What does this sentence mean? Some people say that this sentence is about the impact of environment on human growth. But I don’t think so. My understanding is: if you play with good classmates or smart people, your grades will get better; if you play with bad classmates or people who are not smart, your grades will also get worse. That’s right to understand! Growing up, parents would say: "Don’t play with this person. He has poor grades. If you play with him, he will become worse and stupider. You should play with good classmates." When you think about it, What parents say is not unreasonable. I remember there was such a scene in a TV series: A female classmate who had very good academic performance and ranked first in the class met an unlearned and social friend outside. She knew how to drink and drink. smokes. ...Her friend will teach her things she didn’t dare to do before. Gradually, they become the closest friends. This female classmate slowly learns to skip classes, drink, dress loosely, speak dirty words, and her grades will improve. It has plummeted, and the former number one fell to the bottom. Once, she was playing outside and met a male classmate she liked. Seeing her like this, the male classmate advised her to study hard and not to affect other classmates. Maybe this sentence irritated her, and she felt very unhappy. My beloved The man said something bad about her, and she felt resentful and took out a knife and pointed it at him. The male classmate didn’t take it seriously and said she didn’t dare. The female classmate stabbed the male classmate in a fit of rage. The male classmate went to the hospital, and the girl also went to jail. Looking at it this way, there are not such cases in reality. What parents say is not wrong. Friends and partners also have an impact on people’s growth. Most people study or work in a day, and spend more than ten hours at school or in the company. In a week, you will work for three or four days, or even more. You will see classmates at school, and You will meet colleagues at the company, which is usually the case. The people you have the most contact with are not your parents but your classmates or colleagues. These are all friends or partners. Some of your behaviors may come from the influence of friends, and may even change dramatically. So can you say that friends and partners have no influence on a person’s growth? That shouldn’t be possible, the facts are before our eyes. Friends and partners do have an impact on people’s growth.
topic_relevance_level Failed
chosen_comment
This essay is unsatisfactory in terms of topic and focus. The requirement of the topic is to write a refutation paper, but the starting point of the author’s refutation is not an essential objection to the sentence "Those who are close to red are red and those who are close to ink are black" , it just presents its meaning in another way, which does not meet the requirements of the topic. It is recommended that the author carefully review the topic and choose an angle that touches the foundation of this sentence to avoid going off topic.
rejected_comment
This essay is an argumentative essay. The theme is to refute the theme of "those who are close to red are red and those who are close to ink are black", that is, the impact of the environment on human growth, that is, a negative opinion on this view. The author believes that, If you play with good classmates or smart people, your grades will get better; if you play with bad classmates or people who are not smart, your grades will get worse. In the article, the author illustrates through examples that friends and partners are really important for people’s growth. It has influence. However, the author did not fully prove his point of view. If he does not refute "Those who are close to red are red and those who are close to ink are black", it will be regarded as off topic.

heading: Future Work
text: The school corridor will forever be our little corner where we talk about the future in the evening breeze. As we all continue to move forward through our youth, let's believe there's glory ahead!
The essay excels in focus and relevance. The writer depicts Yu Wei, a character chosen for her vivid spirit shown through qualities and actions. Yu Wei is presented as academically and morally excellent, respectful, gentle, and intelligent. Her friendship and assistance touched the writer, who reflects on mutual support, valuing friendship, and hope for the future. The piece stays on point with sincere emotions, inspiring and moving readers. 作者选择了一位品学兼优、乐于助人 的女孩作为写作对象，通过描写她的 品质和行为，表达了对她的敬佩和感 激之情。同时，文章也强调了友谊的 重要性，以及互帮互助的精神。从切 题的角度来看，这篇作文完全符合题 目要求，以一个具体的人物为例，深 入挖掘其精神品质和代表的风气，并 表达了作者的情感倾向。因此，我认 为这篇作文的中心与切题方面都做得 很好。
The author chose a girl who excels academically and is eager to help others as the subject for the essay, using descriptions of her qualities and actions to convey admiration and gratitude. The piece also highlights the significance of friendship and the spirit of mutual assistance. From the perspective of relevance, the essay fully meets the requirements, using a specific individual to delve into her moral fiber and the ethos she represents while expressing the author's emotional stance. Hence, I believe this essay excels in both focus and relevance. 这篇作文在中心和切题方面不合格。 文章的题目是《这样的人让我感 动》，应该侧重于写让作者感动的 人的事迹，进而表现这个人的性格、 品质等特征。本文却从友谊说起， 讲二人之间是如何成为朋友，发展 友情的，与题目要求不符。建议作 者在写作前仔细审题，围绕中心组 织事实材料。 Human
The essay fails to meet the requirements in relevance and focus. The title is "Such People Touch Me" which should concentrate on the deeds of the person who moved the author, thereby showcasing their personality and qualities. However, the essay starts with friendship, discussing how two people become friends and develop their relationship, deviating from the topic. The author is advised to carefully review the question before writing and organize facts around the central theme. EduChat
GPT-4
(a)
(b)
(c)
Figure 5: An example of generating topic relevance comment for grade 7 student essay titled Such People Move Me using EduChat and GPT-4. Human denotes human-written comment for preference. Text highlighted in bold are points requiring special attention for students. The right side consists of comments generated through three models. In each side, the right part is the translated English version for better understanding. Texts highlighted in red indicate misinformation or incorrect advice, whereas texts highlighted in blue provide accurate feedback or suggestions for students. Specifically, in part (a), EduChat incorrectly assesses the level as "excel" and fabricates a character named Yu Wei that does not exist; in part (b), GPT-4 inaccurately evaluates the essay’s level as "fully meets" and fails to identify the specific "author’s emotional stance"; in part (c), human-written comment correctly points out the topic derivation of this essay, explain why, and provides detailed suggestions on how to refine it. strong Chinese reading and writing skills. The selection process involved evaluating their test samples with BERTscore to determine their ability to understand and annotate text accurately. Training and Division of Responsibilities Before beginning the actual annotation work, the team underwent a training session to harmonize their understanding of the task’s requirements and the annotation guidelines. This training ensured consistency across annotations. The team was then divided based on the educational level of the essays they would annotate: Three annotators focused on essays from primary school students (grades 3 to 6) and the remaining three handled essays from middle school students (grades 7 to 9). For each essay, one annotator was primarily responsible for the annotation, another served as a reviewer to ensure accuracy, and a third annotator was involved in case of significant discrepancies between the first
two opinions. Annotation Process Duration The duration of the annotation process can vary widely depending on the volume of text to be annotated, the complexity of the task, and the team’s efficiency. In our structured project, the entire process of annotation spanned a duration of one month. Compensation Compensation was provided to each annotator based on the actual volume of annotation work completed in a fair manner. C. 5 Explanation of Essay Genres and Prompts
As shown in Table 7, an explanation of 13 distinct literary genres are listed. Moreover, a complete list of 74 diverse essay prompts is shown in Table 9