Generating Sequences by Learning to Self-Correct
Authors: Sean Welleck
Ximing Lu
Peter West
Faeze Brahman
Tianxiao Shen
Daniel Khashabi
Yejin Choi
Venue: ICLR 2023 poster (2023)
Keywords: Language models
text generation

Abstract: Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content. Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs. Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation. We present Self-Correction, an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations. To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations. We show that  Self-Correction improves upon the base generator in three diverse generation tasks - mathematical program synthesis, lexically-constrained generation, and toxicity control - even when the corrector is much smaller than the base generator. 


Published as a conference paper at ICLR 2023
GENERATING SEQUENCES BY LEARNING TO "SELF-JCORRECT
Sean Welleck 1.3
Ximing Lu
Peter West
Faeze Brahman 1.3,7
Tianxiao Shen' Daniel Khashabi² Yejin Choil
I Allen Institute for Artificial Intelligence
Center for Language and Speech Processing, Johns Hopkins University *Paul G. Allen School of Computer Science & Engineering, University of Washington
ABSTRACT
Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, avoiding undesir- able content. Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack mechanism it- eratively revise their outputs. Moreover, some powerful language models are of extreme scale inaccessible, making inefficient, if not infeasible. to update their parameters for task-specific adaptation. We present SELF-CORRECTION, an approach that decouples an imperfect base generator (an off-the-shelf lan- guage model or supervised tor that learns to iteratively correct imperfect generations. To train the correc- tor, we propose online training procedure that can use either scalar natural language feedback on intermediate imperfect generations. We show that SELF- CORRECTION improves upon the base generator in three diverse generation tasks- mathematical program synthesis, lexically-consse constrained generation, and toxicity control- even when the corrector is much smaller than the base generator.
gruuence-to-sequene model) from separate correc-
INTRODUCTION
The standard practice for natural language generation tasks is inherently single-pass: applying a decoding procedure either a few-shot prompted language model one tuned for given task, then considering the generation as "finished" (e.g. Radford et al. (2019); [0019):roownn Brown et al. (2020); Chen let al. (202T)) Powerful generation models often meet most of the task requirements, yet miss few (e.g., omitting subset of keywords), or generate incorrect hypotheses that nevertheless provide useful structure (c.g., a correct problem solving strategy with a missing step). However, after gener- ating even a slightly sub-optimal sequence, the single-pass paradigm requires models to "start from scratch", effectively discarding work already done. A more natural, intuitive approach is leveraging the generation as useful starting point to refine into a higher quality output.
To formalize this intuition, we introduce Self-Correction for Sequence Generation. Figure demonstrates its central principle: a generation model is re-framed as base generator, which pro- duces a reasonable initial hypothesis but does not need to solve the task in one pass, and a second modulc-the perrector-tranne to make up the difference between the hypothesis and optimal so- lution. Neither the generator nor the corrector must solve the full task in one pass, and the corrector can be applied multiple times to iteratively improve the output (3.6). We propose simple, general procedure for training the corrector (Figure 2) by pairing generator outputs with carefully selected targets. The result system which self-corrects, producing outputs through multiple generation passes and breaking the task into steps that can be solved by dedicated and efficient sub-systems.
Self-Correction builds past work for correction in the code and text (e.g. Yasunaga et al. (222) Faltings et al. (02I)) domains, but provides a unified formalism with minimal assumptions about
First authors, contributed equally. {Sccond authors, contributed equally.

---

Published as a conference paper at ICLR 2023
Self- Caff-Coreector
Constrained generation
Toxicity continuation There's 5 no noed nuke shreats if...
Math
Write with
apples Carlos had How many for Sam?
Generator
bow, prepare, shoot, srrww target
I1| shoot arrow.ot target
Bow
you know won't ta that you
missing Carlox
Proparn
X
85
Corrector
shoot en artzo the target with my bow
Proparc X
you know we won't take that a@1
X- 6
operator
I'll prapace my baw and shoot
you know we won't accept that kind of behavior
target
0.00
good!
Figure 1: SELF-CORRECTORS decompose generation into a base generator that proposes an initial hypothesis, and a corrector that iteratively improves its quality.
data and feedback, which applies generally to diverse tasks. A corrector model improves the base generator on 3 such tasks in our experiments: mathematical program synthesis ($3.1, lexically con- strained generation (3.22, and toxicity reduction ($3.3). The trained corrector model even transfers larger generator with similar performance to training from scratch ($3.4). Finally, we explore introducing a third module to the Self-Correction system -xpplexxl using natural language feedback to guide corrections with promising results. Self- Ceff-Correction an exciting path build on the generations of strong models, with efficient, effective, and transferable corrector networks.
2 SELF-CORRECTING SEQUENCE GENERATORS
typical autoregressive text generator (e.g. GPT-3 (Brown et al. 2020 maps an input prompt to distribution over outputs using single parameterized module (e.g. a large transformer), Po(y)ar). We explore an alternative that decomposes into two modules, a base generator, and a corrector,
plylx) =
Σ Po(Bola) (ollynn (3lBo..)
(1)
generator
where the generator provides initial hypothesis that is refined by the corrector. In practice, the cor- rector can be applied multiple times, x)) = E,, Ey, ... -EEr Po x>) II P9 Since a model of this form can both generate and correct its generations, we call it Self-Corroctor.
Self-corrector, have several unique properties compared to typical generators. First, self-corrector decouples generation and correction, allowing us to freely parameterize each module - for instance, by prompting single language model using different language models. In this paper, develop framework to train a separate corrector model ($2.1) We find that the resulting self- corrector improves upon the generator alone ($3), even when the corrector is much smaller ($3.4).
Second, since the generator and the corrector are separated, we can keep the generator as a general- purpose language model and train the corrector with different objectives for different task require- ments. In $2.1I we propose training algorithm for the corrector that is dedicated to improving generations, where the improvement can be any aspect, measured by scalar values.
Third, the corrector can receive explicit about intermediate generations to guide subsequent generations. Formally, p(y(ær) Σy, (v)o//))ool (yl yo, x. (3o)). where f is the feedback. The feedback can be of many forms, e.g. a sentence, a compiler trace, etc. In contrast, typical generator that generates in single pass does not leverage feedback on its own generation. In this paper, we show that the corrector can learn to exploit explicit natural language feedback achieve better performance ( 3.5) Next, we describe our training framework of the corrector.
2.1 LEARNING A CORRECTOR
Our goal is to have the generator generate an initial hypothesis, then improve the hypothesis with the corrector (Eq. IT. We train the corrector improve the quality of hypothesis, while staying as close as possible to the original hypothesis. Here, quality is measured with a scalar value function v(y) which is accessible training time (e.g. 0/1 indicator of program correctness, toxicity score).
2

---

Published as a conference paper at ICLR 2023
Exploration
Pairing Pair hypotheses with value-improving corrections
Over-generate hypotheses and corrections
doa
banch...
Simlatty
King, park, banetj
part
A dog sis park on btechh
Simi
An emol park bunch.
My deg sum banch a
Sivilaitty 066
Corrector
dog alte
a pork.
Learning Train on similar, va value-improving corrections
Get value and (optional) feedback
(dea sundi
Corrector
A dog oht in park
0.66
rncldde the word "band""
A dog Include the ward "bench"
dog sits park on bench
dog park bench
1.00
Connect
Figure 2: SELF-CORRECTIVE LEARNING iteratively trains a corrector by generating hypotheses and corrections, forming value-improving pairs, and selecting those with high similarity for learning.
Algorithm Self-corrective learning
input Generator corrector po. prompts X, value v(-), feedback f())
Initialize datapool D by sampling from po for iteration E .... do
Initialization: Eq.
Form value-impooving pairs from for step in 1,2..... .. .M do
Pairing: Eq
Sample a batch of valuc-improving pairs from P using Eq.
Compute the loss and update 9 using gradient descent for X do
Learning
Sample hypotheses y from datapool
Generate corrections y' o -- x. (w)) Add all (x. D/, v(y') ((y'), (w/)) to the datapool D
Exploration: Eq
Since direct supervision on how to improve hypotheses not available, we design a new algorithm train the corrector, which we refer as self-corrective learning. The algorithm collects pool of generations, pairs them and selects pairs of generation that increase in value and are nearby, then updates the corrector on these pairs. As training progresses, more generations are added to the pool using the current corrector. Algorithm summarizes self-cone cof-ccoreettve learning, detailed below.
Initialization. Self-corrective learning begins with generator Po(No]=),, a corrector Po [y, x) set of training prompts X, and a value function v y - R. Optionally, we can use additional feedback f y F and learn po (y/l) d/ly,, x. fy)), where F is arbitrary.
The algorithm initializes a datapool of (input, output, value, feedback) examples by using the gen- erator to generate multiple outputs for each input. Formally,
D, (ax y. v(y). f(v)) for all y E 2 q(Po(-1=))) D D..
where INN denotes N outputs generated with decoding algorithm g (e.g. temperature sampling). When available, (I. e(y), (v)) examples from another source (e.g. dataset) can also be added.
Pairing. Next, self-corrective learning forms value-improving pairs: examples of mapping hy- pothesis to a higher-valued correction. We use the datapool D to form set of (input, hypothesis, correction) pairs. pair formed when an output has a higher value than another
Ps = (xx V. v(y) < v(y/) for all y.v E Do X Dz), P U zEX
(3)
Next, self-corrective learning selects (input, hypothesis, correction) pairs to update the corrector with. We sample an input, u(X), then sample (ær, y, v') pair proportional to its
Learning.
We also the value and feedhack for y and along with (z, y. v/), which omit reduce clutter.
3

---

Published as a conference paper at ICLR 2023
improvement in value well the proximity between the hypothesis and the correction y':. (v())) - - + 3 s(v, /Z(y).
[((2, 3/) [ar] x exp
(4)
imppoveement
pouximity
where s(,, 3/) is a similarity function and (v) normalizes over the available corrections for y in Ps. Increasing the hyperparameter a E R20 puts more weight on targets that add more value, while increasing 3 E Rz0 retains more similar targets. We update the corrector using the cross-entropy loss L(0)= = - log pe (y'1p y. z; f()) on batches sampled in this way.
Exploration. During exploration, self-corrective learning adds new generations to the datapool by generating from the current corrector:
D'_ {xx v/ u v(g'), f()))) for all y' v' q(po(-1y. D' D'= (5)
and updating the datapool D + DUD'. The hypotheses to correct can come from any source, e.g. newly sampled from the base generator, or from the datapool: we use the latter our experiments.
Inference. We use the trained corrector along with a generator to generate trajectory yT, and consider yr the final output. Since marginalizing over the intermediate gen- erations in Eq. iis intractable, we approximate each summation with a single sequence generated with a decoding algorithm q-)). That is, we decode from the generator, then repeatedly from the corrector:
yo: 1l, ....... .
Generation: Ye ((oo(y (z))
Correction: 1++1 ~ ((Po(Ut+1 111. ((x))) t= 0,
The stopping time T is either fixed, or when a target value is obtained (if v(y) available).
EXPERIMENTS
We evaluate SELF-CORRECTION which generations are strictly correct or incorrect, and generators typically have low performance; lexically-constrained generation, which allows for partial credit, and generators usually give partially-correct solutions (e.g. matching out of constraints); and toxicity control, where 'cor- rectness' is more loosely defined, and the output space is much more open-ended. Our experiments are organized to study three settings:
diversity of tasks: mathematical program synthesis, in
1. Using self-correctors improve upon generators 3.2 3.3)
2. Correcting generators that are much larger than the corrector ($3.4).
Leveraging explicit feedback during training and inference ($3.5).
Next, we describe the self-correction setup and baselines for each task, along with their results.
3.1 MATHEMATICAL PROGRAM SYNTHESIS
First, we consider mathematical program synthesis (Austin al. 202 Mishra al. 2022). Given natural language problem specification x, the task is to generate a program that upon execution returns the correct answer .r. The task is challenging as it draws on language understanding, multiple-step mathematical problem solving (e.g. identifying solution strategy, decomposing problem), and leveraging symbolic tools (e.g. built-in operations, variables). Furthermore, the task demands high level of precision, e.g. single misplaced operation makes the program incorrect.
Experimental setup. As the corrector we use GPT-Neo 1.3B (Black al. (02T), an open-soure source autoregressive language model. GPT-Neo is pre-trained on language and code (Gao et al.) 2021
and hence is widely used for code-related generation (e.g. [Chen et al. [202]); Ni et al. (0222) ;
Mishra et al. (2022).. We consider two settings for the initial generator: (1) separate tuned
instance of Geo 1.3B, and (2) few-shot prompted GPT-3 (Brown et al.; (200). For GPT-3, we
evaluate the davinci and text-davinei-002 engines, representative of large ( 175 generators that
are state-of-ther of-tt-rrrt in related tasks (Wei et al. 2022 See the Appendix for additional details.
"Code will be available at www.github.ccom/welleee /seff correction
" "Estimated size of davinct httppa::/bblg..lleutttt. ai attptt.///ppp.. madel sizes).
Further details available.

---

Published as a conference paper at ICLR 2023
Dataset
Model
Correct
Dataset
Model
Params
Correct
Multiarith
GPT-NEO 1.3B
60.00
GSM
OpenAl 3B
3B
15.50
OpenAl 6B
6B
20.00
CORRECT
98.33
GPT-NEO 34
2.7B
18,80
+SELF-CORRECT,
99.17
NEO FCP+PCP 34
2.7B
19,50
Multitask
GPT-NEO 1.3B
49.02
GPT-NEO
1.3B
8.57
SELI-CORRECT 73.53
CORRECC
1.3B
21.26
+SELF-CORRECT 78.24
1.3B
24.22
Table 1: Evaluation results of mathematical program synthesis experiments. GPT-NEO (1.3B) is the initial generator for SELF-CORRECT SELF-CORRECT. means only applying the corrector to incorrect outputs. Italicized: original non-program version of GSM.
Prohlem:
Problem:
It takes Jennifer 20 minutes groom each of her 2 long hair dachschunds. If she grooms dogs every day, how many does spend grooming her dogs in 30 days?
Mrs. Wilshorough saved $500 to buy concert tickets for her family. She bought 2 VIP tick- ets $100 each and regular tickets at S50 each. How much of her savings does Mrs. Wilsborough have after she buys the tickets?
Generator:
Corrector:
Generator:
Corrector:
a=20+2
a=20+2
a=2+100
a=2*100
b=a*30
b=a*30
b=3+50
b=3+50
answer=b
c=b/60 #fix
c=a+b
c=500-a-b #fix
(aantea)wwer
answer=c
answer
answer=c
print (answer)
print ( (answer)
print (answer)
Figure 3: Grade-school-math (GSM) self-corrections. On the left, the corrector fixes the units (from minutes to hours) in the generator's solution. the right, the corrector revises the logic so that the program computes the total savings instead of the spent on tickets. We add #fix here to indicate the change. See Figure 7 and Figure 8 for additional examples.
Self-correction setup. As the value function we use correctness, which is 1 when the program
executes and outputs the ground-truth answer and 0 otherwise. Our main experiments do not use explicit feedback, i.e. f(y) = O. At inference time, we study two settings for the corrector: (1) applying corrections and selecting the final generation, (2) oracle setting that only corrects draft if the draft is incorrect. We use greedy decoding for the generator and corrector, and 1.
Datasets. We evaluate on problems from 5 problem solving datasets: MultiArith (Roy al. 2015 AddSub (Hosseini et al. 2014). SingleOp (Roy et al.] 2015), SVAMP (Patel et al. 202 and GSM8k (Cobbe al. 202T). As in prior work (Austin al., 2021; Ni al. 2022 Mishra al. 2022), we frame these as program synthesis by converting their solutions to Python programs. We separate our experiments into three increasingly difficult settings:
1. MultiArith, using problems from the MultiArith arithmetic word problem dataset.
Multitask, using problems from arithmetic datasets (MultiArith, AddSub, SingleOp, SVAMP).
3. GSM, using problems from the challenging GSM8k dataset.
For the MultiArith and Multitask settings. we make train/valid/test splits using 60/20/20% of the respective datasets. Similar to Ni et al. (2022). for the GSM setting we use the official GSM8k test split, and create a validation split using 20% the training set. Note that the problems and answers in all datasets are the same as those from the original non-program datasets.
Baselines. We compare SELF-CORRECT with its fine-tuned baseline generator (GPT-Neo 1.3B) in all three settings. For the GSM setting, we compare with existing work that uses models within the same magnitude of scale, including NEO FCP+PCP (Ni et al. (022). which tunes GPT-NEO 2.7B with additional self-sampled programs, and their fine-tuned GPT-NEO 2.7B baseline. We also report 3B and 6B fine-tuned GPT3-like language models from Cobbe et al. (202T), which were trained on the non-program version of GSM8k. We evaluate larger models later in 3.4

---

Published as a conference paper at ICLR 2023
Method
Runtime
CIDER
Constraints
Method
Fluency
Constraints
NouroLogic 28
2.04s
14.70
97.70
Prefix-Tuming
2.96
91.16
NeuroLogic-A
19.24s
15.20
97.80
NouroLogic (28
2.80
96.91
NeuroLogic-A
30
2.85
96.97
GPT-2
0.20s
14.97
91.38
SELF-CORRECT
0.80s
15.30
94.58
GPT-2
2.94
91.50
+NeuroLogic 2.24s
15.28
97.80
SELF-CORRECT
2,98
98.77
Table 2:
generation. By training conrector to optimize constraint satis- faction, SELE-CORRECT improves constraints while maintaining fluency, without modifying the underlying generator. Due space, show CIDER for COMMONGEN and human judgement for E2E as measures of fluency. Other metrics show similar trends and can be found in the Appendix.
Results. As seen in Table the self-corrector improves upon the generator in all three settings, using either inference strategy: always correcting (SEL.F-ORRECC), or only correcting incorrect solutions (SELF-CORRECT.) The self-corrector's performance on Multiarith is very high after correction (98- 99%), a 38 point improvement over the generator, with a similar gain the Multitask arithmetic setting. On the challenging GSM dataset, the self-corrector achieves 21%, and 24% with only correcting incorrect solutions, up from 8.57% for the generator. Notably, this is higher than the larger 2.7B GPT-Neo (also larger than senerator+coorrcttor).. larger models tuned the language version of GSM. The results show that self-corrective learning can improve task performance via training corrector. Qualitatively, the self-corrector can correct values a correctly structured solution, fix the order of operations within multistep solution, adjust unit conversions, and make larger multipart revisions (see Figures 8). Notably, these learned automatically.
3.2 LEXICALLY CONSTRAINED GENERATION
Next, we consider lexically constrained generation. Given a set of constraint words x, the task is to generate a sentence y that includes all the given constraints. Faithful constraint satisfaction is crucial for many downstream tasks, e.g., those that require converting information to text (McKeown) 1985).
Datasets and Metrics. We experiment on COMMONGEN (Lin al., 2020) and E2E (Novikoval et al. 2017). COMMONGEN is benchmark for generative commonsense reasoning where the task is to generate a coherent sentence given a set of words (e.g., dog, catch). E2E involves convert- ing structured inputs into natural language. For both tasks, we report standard metrics including human/automatic measures of fluency (BLEU, CIDER, etc.) as well as constraint coverage. We collect human measures of fluency Amazon Mechanical Turk; the Appendix for details.
Setup. We parameterize the base generator with GPT-2 Radford et al. (2019) (large-size for COM- MONGEN and medium-size for E2E). We fine-tuned the generator for each task. As the value func- tion for self-corrective learning we use coverage, i.e. the percentage of constraints that are present in the output. For inference, use beam search with the generator, then do up corrections using beam search, stopping early if all constraints are met. See the Appendix for additional details.
Results. Table shows the evaluation results. The self-corrector substantially improves constraint coverage over its GPT-2 generator for both tasks, while maintaining or improving its language qual- ity. On the COMMONGEN benchmark, the self-corrector paired with the NeuroLogic constrained decoding algorithm (Lu et al. 202I) achieves the best results, outperforming the more sophisti- cated Meurol-ogic-- A decoding algorithm, while being an order of magnitude faster. Notably, on E2F, self-correction outperforms Neurologic-A" decoding, despite only using standard beam search. This suggests that a corrector can be viewed as an alternative to using a more sophisticated decoding procedure (A*) for improving performance without modifying the underlying model. See Figure
3.3 TOXICITY REDUCTION
Next, we consider the task of toxicity reduction GGehman et al. 2020 Liu et al. 2021). ). Given a
prompt z, the task is to generate fluent continuation y while avoiding offensive content. This task important for ensuring safe language model deployment, yet challenging: due to misaligned pretraining objectives (i.e. modeling internet text vs. non-toxic text), language models are suscepti-

---

Published as a conference paper at ICLR 2023
Toxicity Avg. Max.
Fluency Perplexity
Diversity
Prob.
dist-2
dist-3
0.22
GPT-2
0.527
0.520
11.31
0.85
0.85
Toxicity
0.20
PPLM 7
0.520
0.518
32.58
0.86
0.86
GeDi 17
0.363
0.217
43.44
0.84
0.83
Self-Correct
DExpert
20
0.314
0.128
25.21
0.84
0.84
0.18
Querk
PPO
DAPT 15
0.428
0.360
31.22
0,84
0.84
3
PPO 22
0.218
0.044
14.27
0.79
0.82
Correction iteration
Quark 29
0.196
0.035
12.47
0.80
0.84
SELF- COLF-CORRECT
0.171
0.026
11.81
0.80
0.83
Figure 4: Applying multiple cor- rections reduces toxicity.
Table 3: Toxicity reduction. GPT-2 is the base generator.
ble to generating toxic completions, even when prompted with seemingly innocuous text (Giehman et al. 2020). Along with its practical importance, the task tests whether jeoff-ocorrectors be an effective mechanism for controlling the outputs of language models in an open-ended setting.
Datasets and Metrics. We the REALTOXICITY PROMPTS benchmark (Gehman al. 2020) which contains 100k prompts designed to elicit toxic generations. Following the experimental setup Liu al. (202I), during training we use 85K prompts from the training set, and for evaluation we use the same 1OK non-toxic prompts from test set as Liu et al. (202T). We use Perspective API to measure maximum toxicity, defined as the average maximum toxicity over 25 sampled generations, and the (empirical) toxicity probability of at least 1 out of 25 generations being toxic.
Baselines. We compare SELF- CORRECT with its generator (GPT-2) and previously reported base- lines from Lu et al. (2022a). including PPLM (Dathathri et al. 2020), GeDi (Krause et al.) [021).
DExpert (Liu al. 2020), DAPT (Gururangan et salj|220) PPO (Lu et al. 2022a). and Quark (Lu et al. 2022a). The latter two Proximal Policy Optimization (PPO) and Quantized Reward Kondi- tioning (Quark) represent strong, state-of-the art approaches based on reinforcement learning.
Setup. We use the off-the-shelf GPT-2 Large the generator, and finetune another GPT-2 Large as the corrector. During inference, we use nucleus sampling with p = 0.9 to generate 25 samples for all baselines. As the value function, we use the Perspective API score, vyy) [0. 1], which measures the toxicity of the completed sequence. We do up to three corrections with the corrector model.
reduces the rate of toxic generations substantially, while also maintaining fluency and diversity. SELF-CORRECT outperforms all baselines. This in- cludes inference-time algorithms (PPLM, GeDi, DExpert), which do not modify the generator but degrade fluency and yield higher toxicity compared to SELF-CORRECT, as well as reinforcement learning methods (PPO, Quark) that adjust the generator using toxicity as (negative) reward. The strong baselines use equal or more parameters: PPO and Quark use 3 and 2 model copies. The results show that SELF-CORRECE is effective for detoxification, without modifying the generator.
Results. Table 3 shows that
3.4 CHANGING MODULES CORRECTING GPT-3
Next, we show that a self-corrector can improve the outputs of a generator that much larger than the corrector. We consider two cases: (1) training with small generator, then swapping in the larger generator test time; (2) training with the larger generator, i.e. using the large generator to initialize the datapool for self-corrective learning, then using the large generator at test time.
Toxicity. We evaluate case (1) for reducing the toxicity of a large generator (GPT-2 XI., GPT-3). We generate an initial sequence using the large generator, then refine it with corrector trained in the previous experiments ($3.3). Table 4]shows that the resulting self-corrector (large generator corrector) has substantially reduced toxicity compared to the large generator. This shows the promise of using (self- coorrectrss for controlling the outputs of large language models.
Math program synthesis. Table 4 shows results for math. Analogous to toxicity, the corrector is able to correct larger generators swapped at test-time. For instance, the GPT-3 Instruct generator has quite high performance (84.90 Multitask, 36.80 GSM), which improves 90.90 and 45.00,

---

Published as a conference paper at ICLR 2023
Task
Dataset
Generator (train)
Generator (test)
Generator
Self-corrector
Neo 1.3B
GPT-3
6.96
24.30
Math Synthesis f
GSM
Neo 1.3B
GPT-3 Instruct
36.80
45.00
GPT-3 Instruct
GPT-3 Instruct
36.80
45.92
GPT2-1
GPT2-XI.
0.383
0.027
Detoxification
RTPrompts
GPT2-L
GPT-3
0.182
0.025
GPT2-L
GPT-3 Instruct
0.275
0.023
Table 4: Modularity (program synthesis and detoxification). Self-correctors can correct very large generators, either by swapping in the generator at test-time, or training with the generator. For math synthesis, the corrector is GPT-Neo 1.3B, and here we only correct incorrect outputs. For detoxification, the correction GPT2-L, and we correct all the outputs.
Toxicity +
Constrained Gen.
Math t
Avg. Max.
Prob.
Fluency
Fluency
Constraints
Correct
Correct.
Generator SELF-CORRECT
0.527
0.520
11.31
14.97
91.38
49.02
49.02
0.171
0.026
11.81
15.30
94.58
74.31
79.80
FEEDBACK
0.156
0.020
11.86
15.24
95.88
81.76
82.35
Table 5: Explicit natural language feedback. Correct, means only correcting incorrect outputs.
respectively, by adding corrector. The self-corrector (large generator corrector) improves further by training with the GPT-3 Instruct generator, to 92.75 and 45.92, respectively.
3.5 LEVERAGING EXPLICIT FEEDBACK
Next, we demonstrate SELF-CORRECT'S capacity to incorporate explicit natural language feedback. This amounts defining a feedback function f. then using the same self-corrective learning and inference algorithms 2.1 in preceding experiments (in those experiments, f returned 0). We show that correctors learn to use the feedback, as evidenced by higher performance.
Toxicity. We use additional fine-grained information from the toxicity API as natural language feedback. Specifically, besides the overall toxicity score, Perspective API also provides scores for fine-gratee fine-grained attributes of toxicity (e.g. identity attack, profanity, flirtation, etc.). At training time, compare the attribute scores from a hypothesis and its selected correction, and use the attribute with the largest decrease natural language feedhack (e.g. "decrease toxicity in profanity"). At inference time, we call the API on the current hypothesis and use the attribute with the highest score.
Lexical constraints. In training time, generate natural language feedback for every example pair (I,, y/3/) by elaborating the extra lexical constraints satisfied by y' but not y. e.g. "adding constraint word: read". At inference time, elaborate all missing constraints in the current hypothesis.
Math program synthesis. Math program synthesis contains variety of problem types and errors, without an automated means for identifying the errors (e.g. an API). We explore obtaining natural language feedback about the current program by prompting a large language model. We prompt the model with a problem, hypothesis program, gold solution, and few-shot demonstrations that show feedback on one part of the program; e.g. In the initial guess, 3 should be subtracted. When the program is correct, the feedback Correct. At inference time, we also use feedback from the language model. We allow the feedback model access to gold solution, which we expect makes the feedback higher quality, with the risk of solution leakage at inference- Our results this task are thus used only to study the fcasibility of explicit feedback for math program synthesis.
Setup. For toxicity, lexical constraints, and math we use REALTOXIIITYPROMPPE
COMMONGEN, and the MULTITASK arithmetic setting, respectively. We follow the setup of each task's previous experiments ($3.3) (3.I), except for math we use 5 correction iterations (previously 1). For math, use GPT (text- daxt--avincc-0022 with 6 demonstrations the feedback model.
Results. Table 5 shows that explicit natural language feedback improves performance all three tasks. For toxicity, this means that providing fine-grained attributes (e.g. identity attack, profanity,
8

---

Published as a conference paper at ICLR 2023
Ablation
Math
COMMONGEN
SELF-CORRECT
78.24
94.55
proportional sampling
77.25
93,49
Correct
80
value pairing
62.35
91.76
79
78
Table 6: Effect of pairing and proportional sampling.
77-
feedbbeck
76
Feedback
Exploration Multiarith
Multitask
GSM8k
75
2
89.20
73.49
17.60
Correction iteration
99.17
78.24
23.96
Figure 5: Math: multiple corrections.
Table 7: Effect of exploration program synthesis.
etc.) during learning and inference improves upon using only the scalar toxicity score. Intuitively,
feedback may help the model to focus on a useful correction; e.g., see Figure 6
3.6 ADDITIONAL ABLATIONS AND ANALYSIS
Effect of multiple corrections. Previously, 4 showed that multiple corrections led to better toxicity reduction. On math (Multitask setting). Figure 5 shows that performance improves with more than one correction, and that multiple corrections are more beneficial with feedback. Intu- itively, in this math task, after 2-3 corrections the model needs additional guidance.
Effect of pairing and proportional sampling. Self-corrective learning (i) samples pairs for learn- ing proportional to Equation 4| (ii) only pairs sequences that improve value. We ablate these features by training on Multitakk using a data pool that samples a pair for learning uniformly (rather than Equation 4), and data pool without value pairing. Table shows that both improve performance.
Effeet of exploration. To ablate the effect of exploration, we train a baseline only on correction pairs induced from the base generator. Table 7 shows results on the three math datasets, indicating that exploration improves performance.
RELATED WORK
Celf-Correction relates to work modeling text edits including supervised Wikipedia edits (Reid &
Neubig 2022 Faltings al. 2021 Schickk al..
(022), unsupervised perturbations (Miao et al., 2019] Liu et al. (020), training on human-wriiii written critiques (Saunders et al.) 2022), or refining con-
tinuous variables (Lee al., 2020: Li et al. 2022 Qin (022).. In contrast, Self-Correction learns a text corrector online to improve a quality measure without supervised edits or critiques. Recently, Scheurer et al. (0022) use natural language feedback improve generations. Denoising sequences is common pretraining objective (Devlin et al.) 2019 Lewis et al.) [020 Raffel et al.) 2020), while self-correction 'denoises' generations to improve sealar quality measure. Reinforce- ment learning (RL. is often used to improve scalar measures in generator (Ziegler et 20,19 Sti- ennon al., 2020 Lu al, 2022a), yet infeasible for many models (e.g. those accessed by API), and uses only scalar feedback. Moreover, RL-tuned generators can be used within Self- Seff-Correction. Self-Correction decomposes generation into multiple steps, similar to methods that generate ratio- nales (Wei et al.) 2022 Dohan al. 2022), but Self-Corree Correction produces intermediate steps of the same form as the output, allowing iterative application. Self-Corrcction relates to work on program synthesis (Fu et al. 2019 Balog al. 2020 Gupta al. 2020 Le et al. 2022 and repair (Gupta Yasunaga & Liang (202)) is closest in methodology, but
let al. 2020) Yasunaga & Liang 2020).
Self- Correctnnn uses domain-agnose aggnoti formulation; see the Appendix for discussion.
5 CONCLUSION
We introduced self-correctors, class of models that decompose generation into initial generation and correction steps. We study self-correctors with a fixed base generator along with a corrector trained improve outputs according to a scalar measure of quality. We presented simple, general procedure for training the corrector, and find that self- ceffecorreetion is applicable and effective for im- proving performance, and controlling the outputs of both small and large generators. Moreover, found that celf-correction along with our learning framework provides a promising mechanism for using natural language feedback to improve generation, opening many avenues for future work.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton, Program synthesis with large language models. ArXiv, abs/2108.07732 2021.
Matej Balog, Rishabh Singh, Petros Maniatis, and Charles Sutton. Neural program synthesis with differentiable program fixer, 2020. URL
wwtp:///gggaabaaa/aa/ooo// 10924
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autore- gressive Language Modeling with Besh-Teensorflow March 2021. URI. /tt.i.oirrrrroooo If you use this software, please cite it using these metadata.
10. 5081/281///2 wwwp/oenoo.... 5297715
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL htttps://arxiv org/ bbgg/abs/ae 2210011011 14168
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. ArXiv, abs/1912.02164, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18633/11/111-1422 1423. URL https: //aclanthologyy org/11991123
David Dohan, Winnie Xu, Aitor Lewkowyez, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Narain Sohl-Dickstein, Kevin Murphy, and Charles Sutton. Language model cascades. ArXiv, abs/2207.10342 2022.
Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and Bill Dolan. Text editing by command. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 5259-5274, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.188....0 18653/v1/ 2021 n02lln maiel--miin44444.. URL https: hwtp//ooonnntooooc.ooooooooooooa .rg/
/ 202
414
Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushanfar, and Jishen Zhao. Coda: An end-to-end neural program decompiler. In Advances Neural Information Processing Systems, 2019.
10

---

Published as a conference paper at ICLR 2023
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2021. hRLpttss///xxii... abs /2s./2101..o 00027
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxici-
tyPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Asso-
ciation for Computational Linguistics: EMNLP 2020, pp. 3356-3369, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v 1/2020. hit.:////////00000.iinnn--mmmmoo findings-emnlp.3011 URL httpsss/accLarbhooooyo.rr/////// wwwg://222oooo www.::/..iniiinnseemmmmp...opp
Kavi Gupta, Peter Ebert Christensen, Xinyun Chen, and Dawn Song. Synthesize, Execute and Debug: Learning Repair for Neural Program Synthesis. In Larochell, M Ranzato, R Hadsell, M F Balcan, and H Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17685- 17695. Curran Associates. Inc., 2020. URL /ttphttss:///roeeeeeee wtwpeeeeddinnss.nnuiiiieee neurips. pdf
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy. Doug Downey, and Noah A. Smith. Don't 't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acI- main. 740. URL https: htte//aclaaiee www.anoonnoooooooooooooooooooooa org. acl-main. 740
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 523-533 Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/11D1441100 1058. URL
the
https: hwt..aantoooooy.oro//122...
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 202/, pp. 4929-4952 Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 11211166///222111 findingss-emlll..... URL findings-eemnlp 424
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. arXiv preprint arXiv:2207.01770, 2022.
Jason Lee, Raphael Shu, and Kyunghyun Cho. Iterative refinement in the continuous space for non-autoregressive neural machine translation. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 1006-1015, Online, November URL.
2020. Association for Computational Linguistics. doi: 11216665/111222220mmmmmmmmmmm httipss/acllantooooy..ooo org/2222 emnlp-main.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871 1-71-, On- line, July 2020. Association for Computational Linguistics. doi: 10. 12023//12202..cmmmama URL
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Computational I Linguistics. doi: 10. 18658/ v1/2021 la3--ln
U BtRl/tttts://ccaatthtt org
aal-lloonn 353
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsumori Hashimoto. Diffusion- Im improves controllable text generation. ArXiv, ahs/2205.14217, 2022.
11

---

Published as a conference paper at ICLR 2023
Jared Lichtarge, Christopher Alberti, Shankar Kumar, Noam M. Shazeer, Niki Parmar, and Simon Tong. Corpora generation for grammatical error correction. ArXiv, abs/1904.05780, 2019.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGien: A constrained text generation challenge for generative com- monsense reasoning. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 1823-1840, Online, November 2020. Association for Computational Linguistics. doi:
URL pattp::/aallantoologyyy org.:/00...g//000....//
findings-emn_p
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti- experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6691-6706, Online, August 2021. Association for Computational Linguis- acl-long.522
tics. doi: 10. 10.18653/v1/2021. aoo. acl acl-long.522
URL ntttps://aclantaiin
org/2021.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannanch Hajishirzi. Generated knowledge prompting for commonsense reasoning. Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pp. 1154-3169, 2022.
Xianggen Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song. Unsupervised para- phrasing by simulated annealing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 302-312, Online, July 2020. Association for Computational Lin- URL
guistics. doi: 120055//w/0000..cmmmmmma acl-main. 28
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. NouroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4288-4299, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/0211 http:///222..aammmiii... maacc-- URL //c http://aaaaantooooogoooo wrtplloooyoor////// 2021 www.alaaaaminn.mm maw.p//.......2222mmaaaamm...... 339
Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. CoRR, abs/2205. 13636, 2022a. doi: 20.48555//rXX1.220....33 10. 47.44855/axxiv 2205. 13636.
httpptttt://oo// org/
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. NeuroLogic a*esque decoding: Constrained text generation with lookahead heuristics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 780-799, Seattle, United States, July 2022b. As- sociation for Computational Linguistics. doi: 10.1 (0.18653/v1 00/65//1//200 hrw.laattoolgy..rrg////o
URL https:
paaacc-maing 57
Kathleen McKeown. Text Generation. Studies in Natural Language Processing, Cambridge Univer- sity Press, 1985. doi: 10. 111017//B009780511222777
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. Cgmh: Constrained sentence generation by metropolis-hhattings sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 33 (01):6834-6842, Jul 2019, doi: 10. 1609/aaaa va3iaaiii.333311 v33001.33111161133 URL |tLLhhtts:///oooo aaai. httg://neex..ppp/BA///rrrtaa...e
4659
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: unified benchmark for mathematical reasoning. ArXiv, 2022.
12

---

Published as a conference paper at ICLR 2023
Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Oleksandr Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. Learning from sampled correct and partially-correct programs, 2022. BtLLht:ss///axiit org/ abs 2tbbs/2205.13131 14318
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Di- alogue, pp. 201-206, Saarbrücken, Germany, August 2017. Association for Computational Lin- ort.t//s///cccaccooooooooooo////
guistics. doi: 10. 10.11663/11/117-55255
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080- 2094, Online, June 2021. Association for Computational Linguistics. doi: 10.1 8653/v1/20211 naacl-main. 168. URL
org. 2021. naacl-main 168
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-basee con- strained text generation with langevin dynamics. arXiv preprint arXiv:2202.11705. 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
wttp://rr/ppppp////a/aeeeee //jmlr. Somml..orge
html
Machel Reid and Graham Neubig. Learning to model editing processes, 2022. URL https: 7/openreview.net//Eeeeee dit.uummidddibbbaafff/fata
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: httts://arxiv.
Stgg/bbs1908 10084
Subhro Roy, Tim Vieira, and Dan Roth. Reasoning about quantities in natural language. Transac- fions of the Association for Computational Linguisties, 3:1-13, 2015.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022. URL httpps///arxiv.
org/abs 2206 05802
Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback, 2022. abs 2tt//2204.11444
Timo Schick, Jane Dwivedi- Yu. Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave and Sebastian Riedel. Peer: A collabora- tive language model, 2022. URL https: / arxiiv org/ abs/ 220022000222 11663
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 4615-4629, Online, November URL
2020. Association for Computational Linguistics. doi: 10. http:///122202m///// htttps://aclanthol..s org.ooggyorg//2200 emnlp-main. 373
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Asso- ciates, Inc., 2020. URL URL.httpss//pproceeiiingg meurseeepdddnnnnnneeepeeepeeep wttppppppe///2//iee.... http://cceeooooo.ooooooooo/ppp/p (E595544552220990daa888882222223
13

---

Published as a conference paper at ICLR 2023
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 2022.
Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In 37th International Conference on Machine Learning, ICML 2020, 2020. ISBN 9781713821120.
Michihiro Yasunaga and Percy Liang. Break-it-fixitt, Unsupervised learning for program repair. In International Conference Machine Learning (ICML), 2021.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. LM-critic: Language models for unsuper- vised grammatical error correction. In Proceedings the 202/ Conference on Empirical Methods in Natural Language Processing, pp- 7752-7763, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: (0.18653/v1/202 1er2022...mlpmmii... org 2021 emnlp-main. 611
611. URL https::::: / htt.hhttss///aaaatttllioaoo
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: arXiv:199.008590 08593, 2019. https:: aariiviiiv... org abs 1909 08593
14

---

Published as a conference paper at ICLR 2023
APPENDIX
A RELATED WORK
Self-correction provides flexible framework for improving the performance of off-the-shelf and fine-tuned language models on a wide range of tasks by decomposing generation into a base gen- erator and corrector. Our framework's minimal assumptions the form of the corrector, value function, and data used to train the corrector, as well as its wide applicability differ from prior work.
Learning to fix code. Our work relates two streams of research in the code domain. One stream deals with program synthesis, in which corrector model corrects code from base synthesizer until meets a given specification (Fu al. 2019 Balog et al. 2020 Gupta al. 2020 Le al. 2022), while another stream deals with program repair: correcting code that is provided as input Gupta al., 2020. Yasunaga Liang, 2020 2021). Recently, Le et al. (2022) developed a modular pro- gram synthesis approach that involves a correction module trained on ground-tree truth outputs. In con- trast, self-corrective learning supports cases without ground-truth outputs, e.g. toxicity.
Closest to our methodology is Yasunaga & Liang (202T) Unlike Yasunaga & Liang (202T) self- correction does not assume mechanism for generating synthetic negatives, dataset negatives, or separate model that generates negatives. This is important because engineering these compo- nents for each new task can be prohibitive. Second, Yasunaga Liang (202T) assume a 0/1 value function, while self-correction supports general scalar value functions. This important for tasks such as toxicity that do not have a strict notion of correctness. Finally, we propose new pairing and proportional sampling mechanisms found to be important (Table 6).
Iterative text edits. Self-correction relates to recent works on editing text, including modeling
Wikipedia edits (Reid & Neubig, 2022 Faltings al. 2021] Schick et al. [022], which relies
supervised edits, unsupervised methods (Miao et al. 2019; Liu et al.) (020) that perturb sequences
Liu
with simple operations (e.g. insertion, deletion), editing with models trained human-written
critiques (Saunders et al. 2022). or iteratively updating continuous variables (Lee et al., 2020; Li
al., 2022] Qin et al.) [2022). In contrast to these, self-correction learns an expressive text-to-text corrector that is trained online to improve quality measure, without requiring supervised dataset of edits or critiques. Recently, [Scheurer et al. (2022) incorporate human feedback by fine-tuning on refinements that are similar to the feedback, rather than through an iterative corrector module. Finally, correcting text is inherent to the task of grammatical error correction (c.g. Lichtarge et al. 20119 Yasunaga et al. (2021); our work differs in that correct module within a generation system, and provide a framework for addressing a variety tasks.
Denoising and reinforcement learning. Separately, denoising grouhd-truth sequences is a com- mon pretraining objective (Devlin et al.) [2019] Lewis et al. 2020) Raffel et al.] 2020), while self- correction 'denoises' generations to improve scalar quality measure. Scalar measures
often improved with reinforcement learning (RI.) on a base generator (Ziegler et al.) 2019; Stteennn et al.) 2020 Lu al. 2022a), which is infeasible for improving many language models (e.g. those cessed through an API), and uses only scalar feedback. Moreover, self-correction learns a delta between a generation and solution, and is complementary used within a celf-corrector. Finally, RL can be used as an alternative learning algorithm for training a corrector, which an interesting direction for future work.
RL-tuned generators, which can be
Modular generation. Self-correction decomposes generation into multiple steps, and is thus part of the general class of methods that decompose generation into 'cascade' of modules (Dohan et al. 2022). Examples include using separate knowledge generation modules (Shwartz et al. 2000 Liu et al. 2022), or generating rationales before response (Wei al. (022). Self-correction also produces chain of intermediate steps, but each step is of the same form as the output, allowing for re-using previous generations.
15

---

Published as a conference paper at ICLR 2023
B ADDITIONAL EXPERIMENTAL DETAILS
B.1
DETAILS
In all of our experiments we use off- embedddig similarity function from Tente formers (Reimers & Gurevych 2019): sentence-transforeee wttprrmmeeaaaaaaaiaaaaaaaa oot.rooreertaaaa-iiinni-------
B.2 MATHEMATICAL PROGRAM SYNTHESIS
We fine-tune separate instance of GPT-Neo 1.3B as an initial generator, using the Hug- gingface library with default hyperparameters, except for evaluation steps, which we set to small number to ensure strong checkpoint is selected for each dataset. We use the fine- tuned initial generator initialization for the corrector, and tune the corrector on sequences [SC]x[CURR] C lyj [END), , where x is a problem, Y: and Yj form a residual pair, and [ are special tokens. The loss is on tokens after [START].
Feedback. We write demonstrations using training problems and generations from our GPT- Neo base generator, and GPT-3 ((ext-davinci-002) same training procedure and hyperparameters, except that the sequences now include feedback, [SC]x[ CURR Jyi [FEEDBACK]] y [START jyj [END], where is problem, yı and yj form residual pair, and F Faz, y:) is feedback. We include loss on tokens after [FEEDBACK].
feedback model. We the
B.3 LENICALLY-CONSTRAINEE GENERATION
Hyper-parameters. Table 8 and Table show hyperparameters for CommonGen and E2E.
Human Evaluation. We evaluate fluency of generations in E2E task using human annotators
on Amazon Mechanical Turk (AMT). We randomly sampled 100 instances, along with generations of different baselines and self-corrections. For each instance, we ask 3 annotators evaluate the fluency of generations on a 3-point Likert scale. We aggregate annotations from annotators using majority vote. We restricted the pool of annotators those who are located in US or CA, and had 98% approval rate for at least 5,000 previous annotations.
Hyperparameter
Assignment
Hyperparameter
Assignment
Predictor
GPT-2
Predictor steps batch size optimizer learning rate decoding alg.
GPT-2Meedeum
steps batch size optimizer learning rate
6000
10000
128
100
Adam
Adam
L.c
Lec
decoding alg.
beam search (k=5)
beam search (k==)
Table 8: Hyperparameters for COMMONGEN.
Table 9: Hyperpurameters for E2E,
C ADDITIONAL RESULTS
Toxicity Avg. Max.
Fluency
Diversity
Prob.
Perplexity
dist-2
dist-3
GPT2-L SERRECORRECT
0.527
0.520
11.31
0.85
0.85
0.171
0.026
11.81
0.80
0.83
SELF-CORRECT + FEEDBACK
0.156
0.020
11.86
0.80
0.83
Table 10: Evaluation results of toxicity reduction experiments with natural language feedback.
D QUALITATIVE EXAMPLES
16

---

Published as a conference paper at ICLR 2023
Task
Dataset
Generator (train)
Generator (test)
Generator
Self-corrector
1.3B
GPT-3
46.70
80.00
Multitask
1.3B
GPT-3 Instruct
84.90
90.90
GPT-3 Instruct
GPT-3 Instruct
84.90
92.75
Math Synthesis
1.3B
GPT-3
6.96
24.30
GSM
1.3B
GPT-3 Instruct
36.80
45.00
GPT-3 Instruct
GPT-3 Instruct
36.80
45.92
GPT2-L
GPT2-XL
0.383
0.027
Detoxification
RTPrompts
GPT2-L
GPT-3
0.182
0.025
GPT2-L
GPT-3 Instruct
0.275
0.023
Modularity (program synthesis and detoxification). Self-correctors correct very large generators, either by swapping in the generator at test-time, or training with the generator. For math synthesis, the corrector is GPT-Neo 1.3B, and here only correct incorrect outputs. For detoxification, the correction GPT2-L, and we correct all the outputs.
Table 11:
Bleu-4
CIDER
Coverage
Runtime
NouroLogic 28 NcuroLogiee-A Aesque
26.70
14.70
97.70
2.04s/sent
3D
28.20
15.20
97.80
19. 19.24s/sent
GPT-2
27.90
14.97
91.38
0.2s/sent
SELF-CORRECT
27.98
15.30
94.58
0.8s/sent
SELF-CORRES + feedback
27.82
15.24
95.88
0.8s/sent
CLLL-CCRRECT-NuuoLoggi
28.17
15.28
97.80
2. 2.24s/sent
Table Evaluation rresults of lexically-connsraannedd generratooooooo on COMMONGEN.
Coverage
BLEU-4
NIST
R-L
METEOR CIDER
PREFX-TUNING (Li Liang 2021
91.16
70.30
8.82
72.10
46.30
2.46
GPT-2
91.50
67.12
8.67
70.25
45.58
2.33
SELF-CORRECT
98.77
68.81
8.78
68.60
45.11
2.38
Table 13: Evaluation results of lexically-constrained generation on E2E.
Problem:
Problem:
Melanic had 19 dimes in her bank. Her gave her 39 dimes and her mother gave her 25 dimes. How many dimes does Melanie have now?
Lana's favorite hand was holding a concert where tickets were 6 dollars each. Lana bought 8 tickets for herself and her friends and 2 extra tickets in anyone else wanted go. How much did she spend?
Generator (GPT-Neo):
Generator (GPT-Neo):
answer 19 print (answer)
answer= 8+8. 0) print (answer)
Feedback (GPT-3);
Feedback (GPT-3))
the initial guess, 39 is not included.
In the initial guess, $ 2 tickets are not included.
Corrector (GPT-Neo):
Corrector (GPT-Neo):
answer
39
print ( (answer)
answer= (6. 0* (8. 00..8.++2..0 0+2.0)) print (answer)
Figure 6: Self-correction with natural language feedback.
17

---

Published as a conference paper at ICLR 2023
Problem:
Problem:
Ralph watches TV for 4 hours a day from Monday to Friday, and 6 hours day Satur- day and Sunday. How many hours does Ralph spend watching TV in one week?
The pirates plan to explore 4 islands. Two is- lands require walking 20 miles per day while the other islands require 25 miles per day. How many miles will they takes 1.5 days to explore each island?
walk if
Generator:
Corrector:
Generator:
Corrector:
a=4+7
a=4*5
b=6*7 c=a+b
b=6*2
a=20#2
a=20+2
c=a+b
b-25*2
b=25+2
answer=c
answer=C
c=atb
c=a+b
(annwee))wer)
print(answery
d=c*1. 1.5
d=o*1.5
e=d+b
answer=d
answer=e
print (answer)
print (answer)
Figure 7: Grade school math (GSM) self-corrections. Left: the structure of the generator's solu- tion is valid, but it incorrectly uses the total number of days in week for both and b; the corrector fixes the program to correctly account for the 5 weekdays and weekend days. Right: the genera- tor's solution contains incorrect addition at the end; the corrector removes this line, resulting in correct program.
Problem:
Problem:
spiral notebook costs and personal planner costs $10. How much would cost total to buy 4 spiral notebooks and 8 personal planners at 20% discount?
Julia has $40. She spends half of her money to buy a new game for her phone, She spends a quarter of what she has left on in-game pur- chases. How much money does she have left?
Generator:
Corrector:
Generator:
Corrector:
a=4+15
a=40/2
a=40/2
a=4+15
b=8+10
b=40-a
b=a/4
b-8+10
e=atb
c=b/4 4
c=a-b
c=a+b
d=c*20/100
d-b-c
answer=
answer=C
e=c-d
e=d+40
print (answer)
print (nsswrr)
answer=e
answer=e
print
print (answer)
Grade school math (GSM) seff-corrections. Left: the generator's is program doesn't 't include the discount; the corrector appends the discount to the program. Right: a more sophisticated multipart correction. The generator's assignment of (line 2), and addition (line 5) incorrect. The corrector removes these lines and adjusts the variable names accordingly.
Figure 8:
18

---

Published as a conference paper at ICLR 2023
Constraints:
'iable', "puper'. "read'
Generator:
A man is reading book on a table.
Corrector:
A man is reading book table and writing a piece of paper.
Constraints:
'stand' "wait" 'look'
Generator:
He waits for someone to look at him.
Corrector:
Be waits for someone to stand and look him in the eye.
Constraints:
"walk', 'park", "bench', 'dog,, 'sit'
Generator:
à dog aitting on bench in a park.
Corrector:
A dog sitting on bench and man walking dag in park.
Constraints:
Thee Mill lType: restaurant food English price: high customer rating average area: riverside family Iriendly
near Cafe Rouge
Generator:
The Mill is an expensive, three atar reataurant located near Cafe Rouge.
The Mill is children friendly English restaurant in the riverside area near Cafe Rouge. It has an average customer rating and high price range.
Constraints:
Blue Spice! Type: restaurant food Chinese riverside family friendly near Rainhow Vegetarian Cafe
Generator:
Spice restaurant providing
located riverside.
It is near Rainbow Vegetarian Cafe.
Corrector:
Blue Spice is family friendly Chinese restaurant in the riverside Rainbow Vegetarian Cafe.
Figure 9: Constrained generation self-corrections. Top: COMMONGEN, and Bottom: E2E. Words in green indicate constraints that were previously missing.
Prompi:
Waters then went on to call the Trump
Generator:
administration "bunch of idiots" and "insalent clowns [who are] running"
Corrector:
administration support for the Trans-Pacitic Partnership trade "a total non-event.
Figure 10: Toxicity control self-corrections. Words in red indicate toxic language.
19


Reviews:
[{"review_id": "vDc360uMqu", "reviewer": "Reviewer_E4Ei", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method to correct the outputs of language models using\nanother language model in order to satisfy semantic constraints. A function\n$v(y)$ is needed to measure the quality of the output hypotheses $y$. The\ncorrector is trained to improve this quality while staying close to the original\nhypothesis. The training is done on value-improving pairs: pairs of outputs $(y,\ny')$ from the data pool which belong to the same input and where there is an\nimprovement in quality ($v(y) < v(y')$). The data pool is initialized by the\nbase generator but is continously expanded with the corrected outputs, so the\ncorrector can learn to correct its own output even further in the case of\nmultiple iterations. It is also possible to provide feedback to the corrector\nwhich improves the results.\n\nThe authors study the model and obtain state-of-the-art results on three diverse\ntasks with different constraints: a mathematical program synthesis task, a text\ngeneration task given lexical constraints (which words should be present in the\ntext), and a text generation task where the generated text should not be toxic.\n\n", "summary_of_the_review": "I really liked the paper and I think it could have high impact. I have concerns\nabout the concept of \"self-correction\" and about the clarity of the exposition\nof the algorithm. If those are addressed I'm going to significantly raise my\nscore.\n\n-------------------------------------------------------------------------------\n\nUpdate:\n\nMy concerns were paritally addressed, so I'm raising my score.", "strength_and_weaknesses": "### Strengths\n\nThe paper is built upon a good idea with practical applicability: the method allows to\ncorrect large language models using relatively small computational resources.\n\nThe evaluations are also comprehensive with different tasks, baselines, and\nablation studies, and the obtained results are excellent.\n\n### Weaknesses\n\nMy first concern is about self-correction. I believe that the model corrects its\nmistakes and not itself: it does not reprogram itself to work more efficiently.\nIt is not different in this respect from a genetic algorithm which improves\nsubsequent generations, especially if we combine it with a neural network. I\ndon't see what the presented method adds in comparison to warrant naming\nit \"self-correction\". Also, naming the method \"self-correction\" is too general.\n\nMy second concern is about the explanation of the algorithm in 2.1, it could be\nmade clearer. In the first paragraph, \"(e.g., a classifier)\" doesn't help to\nunderstand what $v(y)$ is. The second paragraph states that \"the algorithm\ncollects a pool of generations, groups them and selects pairs of generation that\nincrease in value and are nearby\". I think this could be expanded to be more\nhelpful and precise. The authors could state what the generations are (and that\nthe first generation is from the outputs of the base generator). The algorithm\ndoesn't really group the generations: as far as I understand it works on subsets\nof the data pool, where each subset corresponds to an input example and contains\n(among other things) the outputs of the base generator and the iteratively applied corrector for\nthat input. Lastly, the algorithm selects pairs of outputs for the same input\nwhich could be from the same generation.\n\nThe organization of 2.1. could also be improved somewhat, as the ordering of the\nphases does not correspond to the order in which they happen in the algorithm.\nParticularly, I would move Exploration after Initialization and before Pairing.\n\nMinor concerns:\n- I think it's a stretch that self-corrector paired with NeuroLogic outperforms\n  NeuroLogic-A* in Table 2.\n- The last line of page 8 should say that Figure 6 is in the Appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O6Tqk-o3RvI", "reviewer": "Reviewer_iywo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggests a new method to generate sequences with language models. Instead of sampling directly from the model, the process first generates a candidate and then revises the candidate using a \"self-correction\" model (possibly in multiple rounds). The core of the paper is an elegant method to generate training data for self-correcting models. Starting with a generative language model, the method generates candidate answers. The idea is to select the candidate answer that is \"wrong\" (i.e. of low quality) but most similar to the correct answer (=ground truth, which is assumed to be available for the training set). This forms a new training example that consists of the original input, the flawed candidate and the improved candidate (=ground truth).\n\nThe similarity between the selected candidate answer and the correct answer is important to make the correction related to the originally generated sequence, which the paper confirms in an ablation.", "summary_of_the_review": "Probably a great paper, but with a serious question around the baselines. I will increase my score if this concern can be addressed.", "strength_and_weaknesses": "Pros: Great idea, generally great set of experiments.\n\nCons: I was not able to understand the baselines. Since this paper assumes a dataset with ground truth answers. So, an obvious baseline for this approach would be a model that is fine-tuned on the ground truth. I was not able to understand which of the baselines were fine-tuned and which of them relied on few-shot prompting. I kindly ask the authors to clarify this point (or provide the additional baseline). I will raise my score significantly, if this can be clarified.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "KB0umI72Ox", "reviewer": "Reviewer_qE6M", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-correction method which trains a corrector to iteratively correct imperfect generation results. The authors first train a generator on the downstream datasets (or directly prompt a large language model), and use it to construct a data pool. Then, the authors select value-improving pairs based on a task-specific value function to build the training set of the corrector. Finally, the corrector is trained based on these samples and generates samples to augment the original data pool. Experimental results show the effectiveness of self-correction in three generation tasks.", "summary_of_the_review": "The proposed method can adapt to various text generation tasks and achieve good performance. However, as mentioned in my review, the authors should solve the concerns about the novelty, the method design, and the experimental settings to make this paper ready for publication.", "strength_and_weaknesses": "Strengths:\n\n1) This paper is well organized and easy to follow.\n2) The proposed method can be applied to a wide range of text generation tasks. The experimental results show the superior performance of self-correction over some competitive baselines.\n\nWeaknesses:\n\n1) The name of the method “self-correction” is confusing for me, because the authors train a separate corrector to improve the base generator. The generator / corrector cannot consistently correct itself in this paper. They should cooperate with each other to achieve better generation performance.\n\n2) From the perspective of correctors, the proposed method seems to train a text editing model (corrector) via pseudo-labeled data generated by a pre-trained model (generator). Specifically, the fixed generator is used to construct the training dataset of the corrector via generating data and selecting value-improving pairs. Then, the corrector is trained on these \"pseudo-labeled\" data and augment the original data pool iteratively. Thus, I feel that the novelty of this method is somewhat limited because using pre-trained models to automatically generate training data is common in recent works. I don’t find any specific design when training the corrector.\n\n3) The feedback has been mentioned for many times in this paper. But this part is individual compared with the whole design. I don’t find any specific module to properly incorporate the feedback signal into the corrector.\n\n4) The experimental setting may be unfair because the corrector has a relatively large amount of model parameters. Thus, the total number of parameters in self-correction (including the generator and the corrector) is significantly larger than that of other baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "73FlF0dtQAl", "reviewer": "Reviewer_Udn3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents self-correction approach for sequence generation. Specifically, given a sequence decoded by the base generator, they train a corrector to generate another sequence, with the goal of achieving a better score than the input sequence. They design the self-corrective learning algorithm to train the corrector, where they select sequence pairs for training that: (1) the target sequence improves the score; and (2) the target sequence stays relatively similar to the input sequence. They also evaluate a setting where the corrector can leverage additional natural language feedback for correction. They evaluate their approach on 3 tasks: mathematical program synthesis, lexically-constrained generation, and toxicity control. The results demonstrate that adding a corrector improves the results over the base generator.", "summary_of_the_review": "Iteratively decoding is a promising direction. However, this approach is not novel and has been well-studied in neural program synthesis domain, and the authors did not properly discuss the related work. Meanwhile, the evaluation setting is not convincing and misses some important comparisons and ablation studies. Therefore, I recommend rejecting this submission.\n\n\n--------\nI thank the authors for their response, and I updated my score.", "strength_and_weaknesses": "Strengths:\n\n1. Iteratively decoding is a promising direction to improve the solution quality for a wide range of tasks, including sequence generation.\n\n2. The evaluation covers different domains and shows good empirical results.\n\nWeaknesses:\n\n1. Learning input correction is not a novel approach. This work completely ignores a long line of research on learning to repair in the code domain. For example, there are existing works that learn a neural debugger to repair the prediction of a base program synthesizer [e.g., 1, 2, 3, 4], and propose neural networks for stand-alone program repair tasks (among others, [5, 6] are closely related in terms of the approach design). The authors should provide a proper discussion of related works in this space.\n\n2. The evaluation setup is not convincing. The base generators utilize a small beam size, and sometimes with greedy decoding. A fair comparison is to increase the number of samples from the base generator, and see whether the corrector improves the performance with the same number of samples in total.\n\n3. Also, it is unclear whether the approach improves over the SOTA results. For example, on GSM dataset and some other math benchmarks, the SOTA approach is self-consistency [7], where the best result on GSM is 78%. The authors should evaluate their self-correction method upon better base generators and compare the performance.\n\n4. In Equation 4, it is unclear how the similarity function is defined.\n\n[1] Gupta et al., Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis, NeurIPS 2020.\n[2] Balog et al., Neural Program Synthesis with a Differentiable Fixer.\n[3] Fu et al., Coda: An End-to-End Neural Program Decompiler, NeurIPS 2019.\n[4] Le, Wang et al., CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning, NeurIPS 2022.\n[5] Yasunaga and Liang, Graph-based, Self-Supervised Program Repair from Diagnostic Feedback, ICML 2020.\n[6] Yasunaga and Liang,  Break-It-Fix-It: Unsupervised Learning for Program Repair, ICML 2021.\n[7] Wang et al., Self-consistency improves chain of thought reasoning in language models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster