RedHat: Towards Reducing Hallucination in Essay Critiques with Large Language Models

Authors: Andrew Zhuoer Feng, Cunxiang Wang, Bosi Wen, Boqun Kou, Haohan Chi, Yu Luo, Lin Fan, Xiaotao Gu, Jie Tang, Hongning Wang, Minlie Huang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: essay critique generation, large language model, hallucination

Abstract:
Essay critiques refer to the textual assessment of an essay, serving as the basis for the scoring of the essay, and are crucial for the improvements of the essay. Essay critique generation has received increasing attention after the blooming of large language models (LLMs), which show promising potential in writing and critiquing essays. Automatic critique generation can streamline both instructors and reviewers as well as spur LLM advancement in long context generation characterized by essay writing. However, current LLMs suffer from hallucinations when generating essay critiques, which are still under-explored in the community. To facilitate research in reliable essay critique generation, we first define this task with a unified input-output format as well as clear judging criteria. To minimize hallucinations in critique generation, we introduce RedHat, a novel approach that embeds the key information from essays directly into the generation process through document-level question-answering, ensuring critiques stay firmly anchored to the original text. We collected a large-scale, high-quality essay critique dataset called EssayC, annotated by human experts over multiple LLM-generated critiques, from a campus undergraduate essay writing course. We experimented RedHat backboned by commercial and open-sourced LLMs. Results showed that critiques generated by RedHat are preferred by human experts over baseline in 20% of cases on EssayC in detailedness and informativeness, with a decrement around 10% on hallucinations in our judging criteria.

Decision: reject
Reviewer scores: 3, 5, 8

Reviews and discussion:
<thread>
creation date:
1730210329039

thread_type:
official_review

<thread_object>
creator: reviewer_rgud

<content>
summary:
The background proposed in this paper is to use LLM to evaluate and criticize essay writing. It points out that the efficiency of using LLM is much higher than that of humans. However, LLM has some problems in performing this task, such as (1) Providing suggestions that do not match or are inappropriate to the essay content, and (2) Proposing logical errors that do not exist in the essay. These hallucination seriously affect the usability of LLMs in generating essay reviews.  This paper has two main contributions, a dataset and a new method to mitigate hallucinations. The results of automatic and manual evaluation demonstrate the effectiveness of the work proposed by the authors.

strengths:
1. The authors list definitions of various types of hallucinations in Table 1, which is helpful for clearly defined tasks.
2. The evaluation setting is solid which including both auto and human evaluation.

weaknesses:
1. From Figure 4, it seems that the evaluation results of GPT are not reliable. This is because people trust the results of artificial experts more. Although the results of untrained human annotators are also random.
2. Why consider ROUGE and BLUE as metrics, as they only consider word overlap. Maybe try BERTScore, BLEURT.
3.The writing structure seems a bit confusing. Chapters 2, 3, and 4 each contain a literature review, plus the work of this paper.
4. It is very good that the authors present a new dataset. But perhaps more annotation process and quality assessment should be provided?

questions:
Q1: From the perspective of the problem defined in Figure 1, if LLM fails to focus on the desired argument or is too focused on the conclusion, is it fit to be called a hallucination?
Q2: I am not sure whether such an open Q-A result is stable. Will it change with each run of model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730227853347

thread_type:
official_review

<thread_object>
creator: reviewer_ssm2

<content>
summary:
This paper addresses the essay critique generation task using large language models (LLMs) and proposes a technique to reduce hallucination in this task. The authors propose RedHat, a technique to reduce hallucination by incorporating document-level question-answering into the critique generation process. The authors compare the proposed method with others, including post-pretraining (PT) and supervised fine-tuning (SFT), and show that their proposed method does reduce hallucination.

strengths:
* The proposed method, RedHat, appears to be grounded in solid assessment practices, with critiques based on concrete practices in answering questions.
* They are working with a non-English (Chinese) dataset and task.
* They provide a clear, position-based analysis highlighting where improvements occurred.

weaknesses:
* The proposed method, RedHat, simply adds document-level questions and answers, which may come across as an incremental, somewhat trivial improvement.
* It’s unclear whether the issue the authors address is truly "hallucination" or simply improving task performance in critique generation. Are there types of essay critique generation errors that are not hallucination? A clear definition would be valuable here.
* The paper is quite vague; while the tasks and datasets used are in Chinese, this isn’t clearly stated. Working on non-English tasks and datasets is generally beneficial for diversity in the English-dominated LLM landscape. However, the paper lacks clarity on this and, more importantly, offers very limited insights on whether the findings apply to other datasets, languages, or base LLMs (e.g., GLM). For instance, it would be useful to know if similar techniques could improve essay critique generation in English using models like ChatGPT.

questions:
* There's very little information about EssayC. Including an overview of the dataset (e.g., types and size of data) would be helpful. Additionally, will you open-source the dataset?
* There’s limited information about the baseline methods (e.g., SFT and PT) in the main body of the paper, particularly regarding the training data used. It was challenging to navigate between the main paper and the appendix to find these details.
* You might consider a few-shot approach, where a few examples of essays and their critiques are provided in the prompt. This approach is often effective for tasks like essay scoring for non-native speakers.
* In Section 5.1, the metric "Detailedness" appears to be lower-is-better, while in the objective function (2) it is being maximized. Perhaps renaming it to something like "ambiguity" would clarify its intent.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730773950190

thread_type:
official_review

<thread_object>
creator: reviewer_oeu8

<content>
summary:
The paper introduces a benchmark and corresponding experiments for the task of automatically providing useful free-form feedback for human-written essays. The main objective is to reduce to what the paper refers to as 'hallucinations' - i.e. in this case irrelevant feedback.
To achieve this, the paper introduces a resource called EssayC - essays written in Chinese by students, which are enriched by LLM-generated critiques. The main contribution of the paper seems to be the use of an automatically derived questionnaire aimed at evaluating different aspects pertaining to the quality of the essay, which - together with automatically derived answers - is used to enrich the prompt of LLMs when generating the essay feedback, an approach which the papers calls "RedHat".

strengths:
Overall this work is intriguing and makes good use of prompt engineering in a clever way. The results seems to indicate some level of success.

weaknesses:
I have a few concerns with the paper:

- Firstly, the construction of the evaluation criteria seems rather ad-hoc, GPT4 is prompted for questions, which are then voted upon by human experts in form of GTAs. The results are in no way linked to or rooted in existing rich literature on evaluating essays. I think this is a missed opportuninty to investigate, how well GPT4's question lists align with existing criteria proposed in the literature. Similarly, the generated critiques are not compared to the actual human-written critiques which existed for the collected essays, in fact they were deliberatly removed and disregarded. This again presents a missed opportuninty to further understand the quality of the approach.

- Secondly, for all the prompt engineering, the results are not very impressive. This is down to a few points: Firstly, only a single LLM, chatGLM-9B is evaluated. Secondly, the overall scores in Table 2, pairwise comparisons in Figure 4 and Table 4 show only a small improvement over the chatGLM baseline. In fact given the rather small dataset size, I'm not convinced they are statistically significant. Thirdly, i am not at all convinced by section 5.3 - the Table and the corresponding explaining paragraph (l 428 ff) seem at odds with each other: The paragraph claims that QA-enhanced RedHat methodology increases the overlap (which in itself doesn't mean that the questions are useful), but the table shows the opposite. Indeed the baseline without QA seems to have a higher overlap than the RedHat method.

- Lastly, I am not convinced by the gravity and impact of the contribution - the task of essay feedback generation is rather niche and very application-focussed, only one language is studied and the presented approach amounts to prompt engineering. Generalising the findings would be difficult, as the human studies would need to be reproduced.

Minor concerns include the wording - for example, the decision to use the word "hallucinations" is not very spot-on, "hallucinations" refer to content that is made up in summaries, i don't think what the paper describes as "hallucinations" meets this broad definition. Furthermore, the vocabulary used to outline the contributions is too strong given the actual results - for example l 103 claims to "prove" generalisability, yet I have not found a formal proof. Similarly, l 107 states that human annotators prefer automatically refined essays based on RedHat's feedback, while the majority of the annotators is indifferent to the changes (60% vote tie in Table 4).


Overall, given the concerns outlined below, I would gravitate towards rejection.

My recommendations to improve the paper would be to (a) root the approach more firmly in existing literature e.g. by comparing the question lists with existing literature; (b) increase the scope, by e.g. comparing more models or comparing model results to human-written critiques. The collection of human ratings would also enable the creation of preference data, which could be used to more efficiently align the models (e.g. by RLHF methods) rather than just using SFT. It should also enable to learn a essay quality ranking method, e.g. by regressing the overall/hallu/detail/info scores in table 2. This should greatly increase the impact of the work, as this learned metrics could be used to predict the quality of essays _without human annotations_, thus enabling to generalise the method and reproduce the findings without reproducing the human annotations.

questions:
Please try to address my points raised in weaknesses, specifically (a) lack of rooting in literature and comparison between human-written and generated feedback, (b) the discrepancies between claims and empirical evidence of efficacy of your results and (c) whether you could reproduce the results using more LLMs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553284933

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 1: Supplemental Experiment of RedHat on other LLMs and other languages/datasets (1/2)

comment:
> Regarding whether RedHat can be generalized to other LLMs, languages, datasets

We are grateful for all reviewers' suggestions on consolidating the experiment scope. 

First, we have followed reviewer ssM2's suggestion, changing the naming of one of our evaluation dimensions "**Detailedness**" into "**Ambiguity**", since it is lower-is-better in the results. We also revised the sign in the objective function.

In order to address Reviewer oeU8 and ssM2' concern about the findings can be applied to other languages or LLMs, **we conduct two** **additional** **experiments**: **Qwen2-7b-Instruct on EssayC** and **ChatGPT-4o on English** publicly available artificial intelligence conference papers (e.g. ICLR, Neurips, ACL). Conclusions are consistent with GLM-4-9b on EssayC that RedHat reduces hallucinations and improves the overall credibility in generated critiques. Details are as follows.

**Qwen2-7b-Instruct on EssayC**

For the experiment on Qwen2, RedHat outperforms the Baseline Qwen2 by 0.136/3.8%/6.585%/6.83% (Overall, Hallu, Ambiguity, Info), and surpasses SFT by 0.652/12.4%/20.00%/34.94%. We also evaluated polished human instructor's critiques under the same standards and revealed 0.200/15.19%/3.03%/18.73% improvement to baseline Qwen2. Additionally, the QA accuracy of Qwen reaches 92.3%, which indicates that LLM is more understood in the evaluated essay, thus contributing to the reduction in hallucination, ambiguity and increments in informative and overall scores. 

|                   | Overall ($\uparrow$) | Hallucination % ($\downarrow$) | Ambiguity % ($\downarrow$) | Informative % ($\uparrow$) |
| ----------------- | -------------------- | ------------------------------ | --------------------------- | --------------------------- |
| Human             | 3.387                | 47.34                          | 11.65                       | 30.63                       |
| Qwen2             | 3.187                | 62.53                          | 14.68                       | 11.90                       |
| 5-shots           | 3.178                | 61.01                          | 11.14                       | 12.66                       |
| Qwen2-redhat      | 3.267                | 62.03                          | 7.59                        | 15.70                       |
| Qwen2-redhat-weak | 3.323                | 58.73                          | 8.10                        | 18.73                       |
| Qwen2-sft         | 2.615                | 74.43                          | 27.59                       | -19.24                      |
| Qwen2-sft-redhat  | 2.636                | 77.72                          | 22.28                       | -10.63                      |
| Qwen2-PT          | 2.777                | 71.65                          | 24.30                       | -9.11                       |

In pairwise comparison, RedHat wins over Baseline Qwen2 by 7.22% and 10.36% with GPT-generated answers and self-generated answers. 

|                   | RedHat better | Qwen better | Both are good | Both are bad |
| ----------------- | ------------- | ----------- | ------------- | ------------ |
| Qwen2-redhat      | 33.26         | 26.04       | 7.38          | 33.32        |
| Qwen2-redhat-weak | 36.30         | 25.94       | 7.18          | 30.57        |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553589155

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 2: Information and description of EssayC

comment:
> Regarding Information about EssayC annotation quality and statistics

Reviewer ssM2 and rgUd have expressed their interests in the detailed information about EssayC. Here we sincerely introduce the construction process and statistical overview.

- **Essay Selection**: EssayC randomly collects undergraduate essays whose topics cover Environment Science, Biological Science, Software Engineering, Game Industry, Earth, Social Science, Journalism and Communications, Economics, Humanities, Literature comments, and so on. Most science, engineering and humanity and social science are covered. 
- **Polish**: Human comments may be incomplete in grammar and organization. We used GLM4-130B to refine and complete their grammar and structure based on the human comments. 
- **Testset split**: 36 essays are randomly picked out of the above process under each field topics. The leftovers are beneficial as training data for supervised fine-tuning.
- **Filter**: For the testset, we asked the annotators to read through the teacher's critiques in the paragraph and filter out unqualified ones(like with only punctuation marks or subjective comments with mere expressions of feeling). Then we devised a raw critique-quality classifier on GLM-4-9B to auto-filter the leftovers in the train data section. Critique numbers drop from 675 to 395 in the test set, and from 51238 to 31694 in the training set after filtration.
- **Annotation and quality assessment**: During the evaluation phase, each generated comment is annotated by two graduate teaching assistants. In case of discrepancies, a third graduate teaching assistant makes the final decision. Our overall Inter Annotator Agreement is 0.71 in GLM-4 and Qwen-2 as a whole, ensuring annotation consistency and reducing random interference.
- **English dataset**: To consolidate the gravity of our work, we followed [1] and picked a subset of ICLR Conference paper and arxiv papers with the number 10. We picked those papers containing less formulas and illustrations, and more importantly, ensuring the paper authors' are accessible so that they could judge the quality over the generated critiques.

We have updated the information about EssayC in Section 2.

|                                 | EssayC | English |
| ------------------------------- | ------ | ------- |
| Essay                           | 36     | 10      |
| Avg len                         | 5204.7 | 42087.3 |
| Critiqued paragraphs            | 395    | 100     |
| Avg paragraph  length           | 278.2  | 1278.4  |
| Avg teacher critique length     | 76.78  | /       |
| Pointwise Annotations           | 5530   | 200     |
| Pointwise Annotation Dimensions | 4      | 4       |
| Pairwise Annotations            | 1580   | 100     |
| Avg Critique len                | 98.53  | 89.65   |

[1] Ryan Liu, Nihar B. Shah. ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing. Arxiv. 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732630842504

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 3: Manuscript Update

comment:
We sincerely thank all reviewers for their constructive suggestions, which have greatly helped improve our manuscript. In response to each reviewer’s feedback, we have made revisions to enhance clarity, precision, and depth. Below is a summary of the major changes made in the revised version:
- Section 1 - Introduction: We have revised the wording in Line 101, changing "prove" to "suggest" to better reflect the tone of our argument.
- Section 2 - Task & Dataset Definition: We have added Table 1, which links our work to existing literature. Additionally, we revised Lines 126-140 to align with these changes, and added Lines 142-156 to introduce the construction and statistics of the EssayC dataset, with the addition of Table 2.
- Section 3 - Hallucinations: We modified Lines 187-196 to better connect our definition of hallucination to related work in other tasks. In Lines 177 and 179, we replaced "Detailedness" with "Ambiguity" and updated the notation "A(c)" and its corresponding sign in the objective function.
- Section 4 - Methodology: We clarified the statement of RedHat’s contribution in Lines 245-253. Additionally, in Line 291, we replaced "Detailedness" with "Ambiguity" to ensure consistency.
- Section 5 - Experiments:
  - In Section 5.1 (Lines 319-323), we provided a more detailed description of the datasets used in our experiments.
  - In Line 358, we included a description of the few-shot method employed.
  - In Line 365, we again replaced "Detailedness" with "Ambiguity."
  - In Lines 374-376, we added the annotation for Inter-Annotator Agreement and included a discussion of quality control in the annotation process.
  - We updated Table 4 and Figure 4 to include results for Qwen2 and ChatGPT4.
  - In Section 5.3, we introduced additional evaluation metrics, including BLEURT and BERTScore, and included Qwen2 results for both questions and answers.
- Section 6 - Conclusion: The discussion of limitations has been moved to Appendix B for clarity and space.
- Appendix:
  - We added the criteria for critiquing English conference papers to Appendix C.2.
  - We included the questions for English conference paper critiquing in Appendix E.2.
  - We provided detailed descriptions of the data preparation for SFT and PT in Appendices F.1.1 and F.2.1.
  - Finally, we added the few-shot prompt to Appendix F.3.

We believe these revisions address the reviewers’ concerns and significantly improve the manuscript. Thank you once again for your valuable feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposing a prompting technique to generate grounded critiques for essays; the main goal is to mitigate "hallucinations" in the critiques that refer to phenomenon not seen in the essay. The approach shows some improvement over baselines like SFT, etc. based on human evaluation. Reviewers raise the issue of this human evaluation criterion not being grounded on existing literature in argumentation. The paper can be augmented with more discussion/analysis of the critiques to better outline how meaningfully different the generated critiques are (score differences of <0.5 points on a likert scale are not informative in this regard).