Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-correction method which trains a corrector to iteratively correct imperfect generation results. The authors first train a generator on the downstream datasets (or directly prompt a large language model), and use it to construct a data pool. Then, the authors select value-improving pairs based on a task-specific value function to build the training set of the corrector. Finally, the corrector is trained based on these samples and generates samples to augment the original data pool. Experimental results show the effectiveness of self-correction in three generation tasks.
strength_and_weaknesses: Strengths:

1) This paper is well organized and easy to follow.
2) The proposed method can be applied to a wide range of text generation tasks. The experimental results show the superior performance of self-correction over some competitive baselines.

Weaknesses:

1) The name of the method “self-correction” is confusing for me, because the authors train a separate corrector to improve the base generator. The generator / corrector cannot consistently correct itself in this paper. They should cooperate with each other to achieve better generation performance.

2) From the perspective of correctors, the proposed method seems to train a text editing model (corrector) via pseudo-labeled data generated by a pre-trained model (generator). Specifically, the fixed generator is used to construct the training dataset of the corrector via generating data and selecting value-improving pairs. Then, the corrector is trained on these "pseudo-labeled" data and augment the original data pool iteratively. Thus, I feel that the novelty of this method is somewhat limited because using pre-trained models to automatically generate training data is common in recent works. I don’t find any specific design when training the corrector.

3) The feedback has been mentioned for many times in this paper. But this part is individual compared with the whole design. I don’t find any specific module to properly incorporate the feedback signal into the corrector.

4) The experimental setting may be unfair because the corrector has a relatively large amount of model parameters. Thus, the total number of parameters in self-correction (including the generator and the corrector) is significantly larger than that of other baselines.
clarity,_quality,_novelty_and_reproducibility: The authors should further clarify the method design and the experimental settings. The overall quality of this paper is OK. But in my view, the novelty of the proposed method is somewhat limited from the perspective of correctors. The reproducibility of this paper is degraded due to the lack of codes.
summary_of_the_review: The proposed method can adapt to various text generation tasks and achieve good performance. However, as mentioned in my review, the authors should solve the concerns about the novelty, the method design, and the experimental settings to make this paper ready for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper suggests a new method to generate sequences with language models. Instead of sampling directly from the model, the process first generates a candidate and then revises the candidate using a "self-correction" model (possibly in multiple rounds). The core of the paper is an elegant method to generate training data for self-correcting models. Starting with a generative language model, the method generates candidate answers. The idea is to select the candidate answer that is "wrong" (i.e. of low quality) but most similar to the correct answer (=ground truth, which is assumed to be available for the training set). This forms a new training example that consists of the original input, the flawed candidate and the improved candidate (=ground truth).

The similarity between the selected candidate answer and the correct answer is important to make the correction related to the originally generated sequence, which the paper confirms in an ablation.
strength_and_weaknesses: Pros: Great idea, generally great set of experiments.

Cons: I was not able to understand the baselines. Since this paper assumes a dataset with ground truth answers. So, an obvious baseline for this approach would be a model that is fine-tuned on the ground truth. I was not able to understand which of the baselines were fine-tuned and which of them relied on few-shot prompting. I kindly ask the authors to clarify this point (or provide the additional baseline). I will raise my score significantly, if this can be clarified.
clarity,_quality,_novelty_and_reproducibility: I consider the idea how to generate data for training self-correcting models the main novelty. This is a significant and clearly expressed idea that might help a wide range of applications.

The paper is written very well and the authors promise to release the source code upon acceptance of the paper.

The authors might find that this paper is related: https://arxiv.org/pdf/2003.10555.pdf.
summary_of_the_review: Probably a great paper, but with a serious question around the baselines. I will increase my score if this concern can be addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents self-correction approach for sequence generation. Specifically, given a sequence decoded by the base generator, they train a corrector to generate another sequence, with the goal of achieving a better score than the input sequence. They design the self-corrective learning algorithm to train the corrector, where they select sequence pairs for training that: (1) the target sequence improves the score; and (2) the target sequence stays relatively similar to the input sequence. They also evaluate a setting where the corrector can leverage additional natural language feedback for correction. They evaluate their approach on 3 tasks: mathematical program synthesis, lexically-constrained generation, and toxicity control. The results demonstrate that adding a corrector improves the results over the base generator.
strength_and_weaknesses: Strengths:

1. Iteratively decoding is a promising direction to improve the solution quality for a wide range of tasks, including sequence generation.

2. The evaluation covers different domains and shows good empirical results.

Weaknesses:

1. Learning input correction is not a novel approach. This work completely ignores a long line of research on learning to repair in the code domain. For example, there are existing works that learn a neural debugger to repair the prediction of a base program synthesizer [e.g., 1, 2, 3, 4], and propose neural networks for stand-alone program repair tasks (among others, [5, 6] are closely related in terms of the approach design). The authors should provide a proper discussion of related works in this space.

2. The evaluation setup is not convincing. The base generators utilize a small beam size, and sometimes with greedy decoding. A fair comparison is to increase the number of samples from the base generator, and see whether the corrector improves the performance with the same number of samples in total.

3. Also, it is unclear whether the approach improves over the SOTA results. For example, on GSM dataset and some other math benchmarks, the SOTA approach is self-consistency [7], where the best result on GSM is 78%. The authors should evaluate their self-correction method upon better base generators and compare the performance.

4. In Equation 4, it is unclear how the similarity function is defined.

[1] Gupta et al., Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis, NeurIPS 2020.
[2] Balog et al., Neural Program Synthesis with a Differentiable Fixer.
[3] Fu et al., Coda: An End-to-End Neural Program Decompiler, NeurIPS 2019.
[4] Le, Wang et al., CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning, NeurIPS 2022.
[5] Yasunaga and Liang, Graph-based, Self-Supervised Program Repair from Diagnostic Feedback, ICML 2020.
[6] Yasunaga and Liang,  Break-It-Fix-It: Unsupervised Learning for Program Repair, ICML 2021.
[7] Wang et al., Self-consistency improves chain of thought reasoning in language models.
clarity,_quality,_novelty_and_reproducibility: The writing is generally clear, except that the similar function design is unclear.

The presented self-correction approach is sound and empirically effective. However, it is not a novel approach and has been well-studied in neural program synthesis domain, and the authors did not properly discuss the related work.

The authors promise to release the source code upon paper acceptance.
summary_of_the_review: Iteratively decoding is a promising direction. However, this approach is not novel and has been well-studied in neural program synthesis domain, and the authors did not properly discuss the related work. Meanwhile, the evaluation setting is not convincing and misses some important comparisons and ablation studies. Therefore, I recommend rejecting this submission.


--------
I thank the authors for their response, and I updated my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a method to correct the outputs of language models using
another language model in order to satisfy semantic constraints. A function
$v(y)$ is needed to measure the quality of the output hypotheses $y$. The
corrector is trained to improve this quality while staying close to the original
hypothesis. The training is done on value-improving pairs: pairs of outputs $(y,
y')$ from the data pool which belong to the same input and where there is an
improvement in quality ($v(y) < v(y')$). The data pool is initialized by the
base generator but is continously expanded with the corrected outputs, so the
corrector can learn to correct its own output even further in the case of
multiple iterations. It is also possible to provide feedback to the corrector
which improves the results.

The authors study the model and obtain state-of-the-art results on three diverse
tasks with different constraints: a mathematical program synthesis task, a text
generation task given lexical constraints (which words should be present in the
text), and a text generation task where the generated text should not be toxic.


strength_and_weaknesses: ### Strengths

The paper is built upon a good idea with practical applicability: the method allows to
correct large language models using relatively small computational resources.

The evaluations are also comprehensive with different tasks, baselines, and
ablation studies, and the obtained results are excellent.

### Weaknesses

My first concern is about self-correction. I believe that the model corrects its
mistakes and not itself: it does not reprogram itself to work more efficiently.
It is not different in this respect from a genetic algorithm which improves
subsequent generations, especially if we combine it with a neural network. I
don't see what the presented method adds in comparison to warrant naming
it "self-correction". Also, naming the method "self-correction" is too general.

My second concern is about the explanation of the algorithm in 2.1, it could be
made clearer. In the first paragraph, "(e.g., a classifier)" doesn't help to
understand what $v(y)$ is. The second paragraph states that "the algorithm
collects a pool of generations, groups them and selects pairs of generation that
increase in value and are nearby". I think this could be expanded to be more
helpful and precise. The authors could state what the generations are (and that
the first generation is from the outputs of the base generator). The algorithm
doesn't really group the generations: as far as I understand it works on subsets
of the data pool, where each subset corresponds to an input example and contains
(among other things) the outputs of the base generator and the iteratively applied corrector for
that input. Lastly, the algorithm selects pairs of outputs for the same input
which could be from the same generation.

The organization of 2.1. could also be improved somewhat, as the ordering of the
phases does not correspond to the order in which they happen in the algorithm.
Particularly, I would move Exploration after Initialization and before Pairing.

Minor concerns:
- I think it's a stretch that self-corrector paired with NeuroLogic outperforms
  NeuroLogic-A* in Table 2.
- The last line of page 8 should say that Figure 6 is in the Appendix.
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written, has high quality, is novel and practical. The
authors will make the code publicly available upon acceptance, making it easier
to reproduce the results.
summary_of_the_review: I really liked the paper and I think it could have high impact. I have concerns
about the concept of "self-correction" and about the clarity of the exposition
of the algorithm. If those are addressed I'm going to significantly raise my
score.

-------------------------------------------------------------------------------

Update:

My concerns were paritally addressed, so I'm raising my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents the idea of training a corrector for sequence generation, which can be used to improve samples from a base model.  The authors designed a simple way of creating training data for the corrector by sampling from the base model and creating pairs in the direction of improvement.  Extensive experiments are done for a variety of domains that show the promise of this approach.

Overall the reviewers and myself liked the idea, but there are some questions that are worth addressing a bit more.  One is related work - following the suggestions by the reviewers the authors can consider putting the work better into the context, in particular given the extensive literature on the topic of editing programs in the program generation / synthesis literature.  The other is the setup for comparisons, the reviewers raised a few important points, that the base + corrector setup (1) uses an additional model and therefore the total parameter count can be significantly higher, and (2) potentially uses more evaluations, which creates an unfair advantage.  The authors acknowledged these in the comments to the reviewers and have made some changes to the paper, but in general these two points should be made very clear and followed consistently in the comparisons.

I'm supporting an acceptance given the proposed idea's simplicity and good empirical results, but do urge the authors to improve the paper and address the issues mentioned above.
justification_for_why_not_higher_score: A couple issues are worth addressing a bit more for this work to get a higher score, in particular on related work and the setup of the experiments, regarding the base + corrector setup potentially using more model parameters and more evaluation budget.
justification_for_why_not_lower_score: Overall the idea proposed in this paper is valuable, and experiment results back that up (though with some room for improvement).
Decision: Accept: poster