Sanitizing LLMs: Retrospective Learning for Self-Correction of Inconsistent Samples via User Preferences

Authors: Cheng Chen, Ivor Tsang
Primary area: foundation or frontier models, including LLMs
Keywords: Large language Model, Prompt Based Task, New Downstream Task with Unsupervised Data, Unsupervised Data annotation with User Preference

Abstract:
With the advent of large language models (LLMs), using LLMs in conjunction with prompt-based tasks has demonstrated the ability to reduce the high cost and inefficiency of human annotations. Nonetheless, in unsupervised new downstream tasks that require user preferences to align data annotations with expectations, existing evaluation methods for prompt-based tasks become ineffective, especially when ground truth annotations are insufficient or missing. To fill this gap, we propose the novel Consistent and Inconsistent (CAI) Ratio, inspired by our experimental observation that LLMs underperform when the number of inconsistent samples—those with inconsistent predictions across LLMs and the student model—exceeds the number of consistent samples. By estimating the CAI ratio and identifying consistent and inconsistent samples with our proposed CAI identification approach, we aim to minimize inconsistency and enhance the accuracy of LLM-generated annotations for unsupervised data. To achieve this, we introduce Retrospective Learning (RetroL) with user preference, a data-centric approach that collaborates with the student model and LLMs, using a small number of human annotations as user preferences to resolve inconsistencies in the identified samples. Applied to eight domain-specific NLP datasets, our Retrospective Learning approach, leveraging CAI identification, significantly improved the accuracy of LLM-generated responses, with the CAI ratio increasing as the accuracy improved.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730575899355

thread_type:
official_review

<thread_object>
creator: reviewer_vkpa

<content>
summary:
Authors propose a metric called Consistent and Inconsistent (CAI) Ratio where they identify the ratio of consistant and inconsistent data points. Then they use this metric and propose a method called Retrospective Learning (RL) to make more inconsistent data points consistant which can help boost the accuracy of the generated data.

strengths:
The approach proposed in the paper is straightforward, easy to understand and easy to implement which can make people to easily use it.

weaknesses:
While the approach proposed in the paper is easy to understand, the paper is not written clearly! Specially the beginning few sections of the paper including the introduction. The reason is that there were many undefined terms in the beginning of the paper that would make it hard to understand the paper in the beginning. The reader might understand the paper only after going through the paper multiple times and decoding the terms. The paper might need significant revision to make it clear. In addition, there are confusing terminology used in the paper like using RL for Retrospective Learning. I would advise changing these acronyms to something that is not widely used to avoid confusion. In addition, the motivation of the work is not super clear to me either. All these points resulted in the paper to not be fully clear and made it hard for me to assess the paper completely. 

Another concern is that authors use the same metric as an evaluation metric (CAI) based on which the mitigation approach is proposed which might not be fair. Although authors use accuracy as a metric as well, it would be good to use other metrics to make the results more robust. Can authors do some human experiments or perhaps use other metrics as well?

questions:
Mentioned the question in the weaknesses section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703484925

thread_type:
official_review

<thread_object>
creator: reviewer_d96a

<content>
summary:
For ensure the model generated labels/responses are aligned with end-user needs, this paper first proposes a consistent and inconsistent ratio for evaluating the performance of LLMs or student generated annotations. The main motivation is the consistent samples have significantly higher accuracy than inconsistent ones. This ratio is used to identify consistent and inconsistent samples. For resolving the self-correction with limited user preference data, this paper also introduces a new method retrospective learning to identify and correct inconsistent samples for minimizing inconsistency and enhancing the accuracy of LLM annotations. The proposed method achieves a better classification accuracy consistently cross datasets with a higher consistent and inconsistent ratio.

strengths:
Strengths: 

1.	For assessing the annotations generated by LLMs or student model, this paper proposes a consistent and inconsistent sample identification and ratio. It can identify the consistent and inconsistent samples. It utilized the predictions from teacher model and student model. If the predictions from these two models are same, then it is a consistent sample. Vise versa.

2.	The retrospective learning method is proposed for self-correction. It includes two modules: divide-and-conquer self-correction and majority voting. With this method, the prediction accuracy is improved because inconsistent samples impact the model performance.

weaknesses:
Weaknesses:

1.	Although this paper has shown some results on various datasets, it still lacks the comparison with other methods that can handle the similar problem. 

2.	More experiments and explanations about the reliability of the CAI score is needed. And this score is highly depended on the selected teacher model and student model. More explorations and comparisons are needed here.

questions:
Please check the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730774361421

thread_type:
official_review

<thread_object>
creator: reviewer_4bru

<content>
summary:
This paper introduces the Consistent and Inconsistent (CAI) Ratio which serves as an effective evaluation metric for unsupervised data with user preferences and Retrospective Learning(RL) to self-correct the identified inconsistent samples using consistent samples.

strengths:
- The results are diverse and demonstrate the effectiveness of the proposed approach.
- The idea behind the proposed ratio and data-centric approach is intuitive.

weaknesses:
- Figure 1 lacks clarity and should present each circle in sequential order. Additionally, the notation in the text does not correspond with the notation in the figure.
- In fact, my primary question throughout the entire paper is why a student model is necessary at all; this aspect does not seem to be addressed in the introduction. The motivation seems to lack clarity and justification. 
- There is a lack of detail regarding the clustering operations.

questions:
- At line 116, referencing a work from 1999 cannot be considered "recently." 
- Additionally, careful attention should be given to the appropriate use of \citet and \citep for citations, as well as ensuring consistent and correct application of \ref and \eqref.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732706929150

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
**Response to Reviewer Feedback**

We greatly appreciate the constructive comments provided by the reviewers, which have significantly enhanced our manuscript. Below, we summarize the major revisions and responses to the key points raised:

**1. Expanded Experiments and New Baselines**
- Added three more NLP datasets for a total of eight, covering diverse domains.
- Evaluated one additional model: Added Meta-Llama 8B Instruct (open-source) alongside two existing closed-source LLMs, enhancing our comparative analysis.
- Introduced two new baselines, Self-Refine and Reflexion, demonstrating the superior performance of Retrospective Learning (RetroL).

**2. Validation of the CAI Ratio**
- Performed Pearson correlation analyses, confirming strong positive correlations between CAI ratios and accuracy, with significant p-values.
- Reported on CAI ratio disparities across models, underlining the lower accuracy of Meta-Llama 8B Instruct compared to others.

**3. Improvements in Presentation and Motivation**
- Clarified terminology, replacing "RL" with "RetroL" to prevent confusion with Reinforcement Learning.
- Enhanced the introduction to better articulate the challenges and necessity of a student model in scenarios where teacher model competency is low.
- Updated figures for clearer visual representation and alignment with the text.

**4. New Results and Observations**
- Presented new findings that show RetroL's significant improvements in annotation accuracy and CAI ratios across all datasets and models.
- Noted the framework's adaptability to both open-source and closed-source LLMs.

**5. Addressing Reviewer Comments**
- Provided further justifications for the student model, emphasizing its role in managing noisy teacher outputs.
- Clarified clustering operations and their relevance to our methodology, while outlining potential future work involving human experiments.

**Conclusion**
The rebuttal addresses all concerns raised about the reliability of the CAI ratio, the need for the student model, and comparisons with related methods. Our updates underscore the robustness and adaptability of the Retrospective Learning framework across a variety of datasets and LLM configurations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors introduce "Retrospective Learning" to increase the accuracy of data generated for downstream tasks, e.g., to be used for evaluation. As far I as understand it, the problem being addressed here is basically trying to ensure "teacher competence" for new tasks which lack manual annotations. To this end, the authors introduce the notion of Consistent and Inconsistent (CAI) examples. 

As pointed out by multiple reviewers (vKpA, 4bru), the motivation and framing is rather muddled in this paper. I read the abstract and introduction (even after being revised for clarity) and agree with this. The author response in OpenReview was actually more cogent, in my view. In any case, I think the work would need substantial revision from its current form to clarify the contributions on offer.