Improving Language Model Self-Correction with Meta-Feedback
Authors: Xinnuo Li, Yunxiang Zhang, Lu Wang


Reviewer paper summaries: This paper proposes Feedback-on-Feedback, an idea to improve self-feedback of LLMs by generating meta-feedback on their own feedback. Specifically, their method generate multiple (initial) feedback and generate meta-feedback when the initial feedback have disagreement. Their experiments evaluate FoF of GPT-3.5 and Llama 3 (8B) on multiple benchmarks in math reasoning, machine translation, and programming problem. They observe improvements over Self-Refine on all datasets they evaluate.
Summary of strengths: The idea of generating feedback on LLM feedback has not been sufficiently explored. This paper shows that feedback-on-feedback improves self-correction performance from self-refine.
Summary of weaknesses: ### Novelty

The idea of improving feedback is relatively unexplored, but their method (FoF) seems a variant of well-known approaches of self-consistency and sample-and-rank.

It is not surprising that well-known approaches for improving general responses from LLMs can improve feedback from LLMs. I think this paper does not provide observations that cannot be straightforwardly expected from prior work.

I recommend providing more detailed analysis on the behaviors of meta-feedback (Section 5.3) or hyperparameter selection (Section 4.5, 4.6). This paper also should compare their methods to more simple methods of self-consistency and sample-and-rank.

### Missing Citations

Some related papers are missing. I understand some of them are not cited because they were published recently, but I recommend citing them in future versions of this paper (possibly as parallel works).

Sample-and-rank (FoF is a variant of sample-and-rank):

* Training Verifiers to Solve Math Word Problems https://arxiv.org/abs/2110.14168
* Large Language Models are Better Reasoners with Self-Verification https://arxiv.org/pdf/2212.09561

Improving LLM-based feedback:

* Evaluating Large Language Models at Evaluating Instruction Following https://openreview.net/forum?id=tr0KidwPLc
  * Prompting strategies for improving LLM feedback
* Evaluating LLMs at Detecting Errors in LLM Responses https://arxiv.org/abs/2404.03602
  * This paper evaluates self-consistency for LLM feedback

### Statistical Test

Given a small improvement by FoF (e.g., Table 1), I would like to see the results of statistical tests on the performance differences.

### Hyperparameters

* Section 4.4: "This temperature value ensures that the generated feedback samples are diverse"
  * I wonder whether a temperature of 0.7 "ensures" the diversity
* I recommend providing more analysis in Section 4.6
Comments, suggestions and typos: ### Other Related Papers

There are some recently publised related papers.

Self-correction:

* When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs https://arxiv.org/abs/2406.01297
  * A survey of self-correction, which provides an analysis in intrinsic self-correction.

Analysis in the error detection capabilities of LLMs on LLM responses:

* LLMs cannot find reasoning errors, but can correct them! https://arxiv.org/abs/2311.08516

### Minor Suggestions

* Section 5.3.1: No need to use sub-subsection because there is only one sub-subsection

### Typos

* Line 011: "1While"
* Line 069: Use \citet
* Line 251: "samples feedback with temperature of 0" (I guess temperature of 0.7?)
* Line 252: "(see Figure ??)"
* Table 1: "s" in the second row
Overall assessment scores: 2