Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper builds on work on in-context learning and rationale generation. 

The proposed approach uses a small seed set of query-answer pairs along with their rationales to prompt a large language model (GPT-J) to produce rationales along with the answer.

This prompted LM is then used to generate rationales for the remaining examples in the training set. When the predicted answer is correct, the generated rationale is added back into the training set to fine-tune the next version of the LM.

To prevent saturation where no new reasoning patterns are discovered, authors propose "rationalization". If the LM predicts the wrong answer, the correct answer is hinted to the model to elicit a new rationale and answer. If again, the answer is incorrect, the example is ignored. However, when given the hint, if the model predicts the right answer, then the associated rationale is added to the training set.

This iterative procedure of collecting more rationales and fine-tuning the LM on the larger dataset is repeated till saturation.

Authors show this procedure is useful to learn better performing models for simple arithmetic, common sense reasoning, and complex math problems; the model outperforms much larger models with fewer boot-strapped training examples. The generated rationales are analyzed for correctness and bias. An ablation is conducted to test the impact of "rationalization".
strengths_and_weaknesses: Strengths
---
[+] The paper is clearly motivated, clearly written, and properly positioned with respect to past work.

[+] The method is intuitively explained along with possible connections to reinforcement learning.

[+] The rationalization procedure is ablated and described along with relevant caveats and warnings.

[+] Generated rationales are analyzed for correctness and bias.

Weakness
---
[-] Training stability is not discussed for all datasets. Only the instability in arithmetic reasoning is discussed.

[-] The number of cases of "right for the wrong reasons" is not estimated. There is a very good discussion on the types of errors, however, to paint a complete picture, there needs to be some estimation of what fraction of correct predictions were paired with incorrect rationales.
questions: Can authors provide concrete results backing claims made in Lines 354 and 371?
limitations: Appendix G contains a necessary discussion on the limitations of rationale generation systems and large language models more broadly. However, attention needs to be drawn to the section in the main body of the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel and interesting method for training pretrained large language models (LLMs) on reasoning tasks. 
Specifically, an LLM is trained to output a rationale along with the answer. 
If the predicted answer is correct, then the generated (question, rationale, answer) tuple is added to the training set, on which the model is further trained.
For questions that the model gets wrong, this paper proposes an additional “rationalization” step that trains the model to generate a new rationale by providing the correct answer.
This training approach allows the model to use its current reasoning capabilities to improve itself.
The model was applied to three diverse tasks - arithmetic, commonsense QA, and math-word problems. 
Starting from only a few-shot dataset with rationales, the model self-generates a large training dataset with rationales and also performs few-shot models as well as models that are directly fine-tuned on the dataset (without rationales)
strengths_and_weaknesses: Originality:

1. The paper presents a novel method for improving the performance of LLMs by using its current reasoning capabilities. It also leads to the generation of large scale rationale dataset, which is a bonus. While there are no novel technical ideas put forth by the paper, the general idea is novel and attractive and also generates interesting results

Quality:

2. The paper is generally of high quality with interesting ideas and well-done experiments on diverse benchmarks. Although, I think the paper should clearly report the current state-of-the-art numbers on CommonSenseQA.

Clarity:

3. The paper is very clearly written and was very easy to follow.

Significance:

4. Making LLMs generate its own rationale and use it for reasoning is a new research area with a lot of interest. I believe this paper brings in interesting results which will lead to further work in this area.

Weakness:

1. The biggest weakness of this approach is that there is no *guarantee* that the rationales generated by the model is correct even though it generates a right answer. Now this could easily be exacerbated as the model feeds these possibly flawed explanations to itself. The paper notes this limitation in a few places. But it is important to explicitly note this in a separate limitation section.
questions: It was not clear to me how you provide the hint to the model during rationalization. For example, for CommonSenseQA, what does the prompt look like with the hint? Is there a special marker on the correct answer?

In Figure 4a, the fact that the performance of (n+1)th digit starts to improve only after nth digit has been learned is quite clean and intuitive. However, with the model with rationalization, this behavior goes away, can you explain why?

Also, just to be clear in Fig 4a and b, the results on the validation sets are reported after each iteration, right?

What is the test accuracy on CommonsenseQA? Also please also report the state-of-the-art numbers on CommonSenseQA as presented in [29] (the same paper from where the GPT-3 numbers are taken from)

Minor: Line 178 - outer loop → inner loop
limitations: I believe it is important to explicitly include a section on the adverse effects of models generating flawed rationales while giving the right answer.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work introduces a boostrapping strategy for rationale generation to promote reasoning in question answering. Unlike existing generation methods, the boostrapping framework is able to iteratively improve rationale generations by filtering irrelevant rationales that lead to incorrect answers, and with another step of rationalization to revise the wrong rationales.
strengths_and_weaknesses: Strength:

The idea of bootstrapping rationale generation is interesting, and is specifically beneficial considering there is no annotated ground-truth rationales to learn from. Then it makes much sense to automatically filter those rationales that lead to incorrect answers and further augment the refined dataset with newly-generated rationales with another step of rationalization taking the correct answer as a hint. The authors conduct extensive experiments on three different domains to demonstrate the effect of bootstrapping along the iterations and the clear advantage in terms of performances over other baselines.

Weakness:

(1) The details for prompt design and format are not quite clear. Specifically, from Figure 2, it shows the prompt contains all the candidate choices, whereas there seems to be no choices in the prompt for rationale generation in section 3.1. How exactly does the format differ for rationale generation with and without rationalization? Does rationalization process also provides a number of demonstrations in its prompt?

(2) More experiments could be added: -- The current setting samples one rationale for each example. Since only one rationale is needed, I wonder how does greedy decoding perform (which brings less randomness). -- The authors mention that continually training the model may overfit and thus they re-train from $M_0$ for each iteration. It is better to empirically provide the evidence on how does continual training perform. -- Table 1 gives results on dev set of CQA. How about test set? -- Similar to Figure 4, do you evaluate the results on CQA along increasing iterations? Does the pattern align with Figure 4? -- For case study, it is better to provide some concrete examples to demonstrate the two points.
questions: (1) Comparing the rationale generation with and without rationalization, I start to wonder whether the authors have considered only using rationalization with hint to produce the rationales for fine-tuning. How does the performance differ from the proposed strategy? The reason for this question is that when filtering rationales leading to incorrect answers, it is possible that a good rationale is filtered, or a bad rationale is kept. Hence, it should be more reliable if providing the correct answer as hint to generate rationales.
limitations: yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method for "bootstrapping reasoning" -- starting with a few examples of reasoning in a domain and producing supervised explanations for that domain. This uses a combination of two methods: generating an explanation and an answer and keeping the explanation if the answer is right, plus post-hoc rationalizing correct answers for cases where the model makes the wrong prediction. That is, the correct answer is used to "prime" the language model to generate an explanation geared towards that answer. Experiments are conducted on an arithmetic dataset, on GSM8k, and on CommonsenseQA.  Rationalization leads to faster learning for arithmetic, though not necessarily better "final" performance. On CommonsenseQA, the rationalized model does perform better and is able to use more of the training data. Human annotation shows that these generated explanations are of good quality.
strengths_and_weaknesses: STRENGTHS

- This is an exciting and very timely research direction. The core idea seems like a very good one and is likely to garner interest as interpretation generation in the context of these reasoning tasks becomes more widely studied.

- It's nice to see a diversity of problems tackled here: fully "rationalizable" (arithmetic) all the way to CommonsenseQA which has a very loose notion of what an explanation is.

- The experiments with GPT-J will be reproducible by broad swaths of the community with access to closed-off models, which is nice to see in this space.

- The discussion of the experiments in Section 5 is generally quite clear and helpful to shed light on the results

WEAKNESSES

- There's a pretty significant focus here on arithmetic, which in my opinion is not that interesting a problem to address with language models. It largely doesn't play to their strengths and is very synthetic in nature, though I admit it can be a useful test bed.

- Missing citations: While the application of this methodology to natural language reasoning / chain-of-thought is novel to my knowledge, this basic technique follows a long line of work on using weak supervision for problems like semantic parsing and knowledge base question answering. In these cases, the model does latent reasoning over possible derivations leading to the right denotation, which is essentially the same idea and has been described in the same policy gradient formulation before.

While the key idea dates back as far as Zettlemoyer and Collins's work, here are a few more recent citations:

Kelvin Guu et al. "From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood" https://arxiv.org/pdf/1704.07926.pdf

Omer Goldman et al. "Weakly Supervised Semantic Parsing with Abstract Examples" https://aclanthology.org/P18-1168.pdf

As a result, I don't think this paper is all that novel methodologically -- it's more applying an old idea to a new setting. This is fine as a contribution but more limited than what the paper claims.
questions: None
limitations: The limitations section was okay, although I feel like there are a few other chief limitations:

(1) Error propagation: if the model is bad at rationalizing, it is self-training on bad rationalizations. Even if they may somehow yield the right answer, there is a "right for the wrong reasons" concern here

(2) Potential non-generalizability of the results to larger models -- while possibly impractical for the authors to experiment with, the recent "emergent properties" paper (as well as discussion in GPT-3 and PaLM) shows that the fact of 10-100x in LM size makes a huge difference in results for tasks like these.
review_title: null

Meta-review: metareview: All reviewers found this paper to be interesting and timely on a topic of much current interest -- prompting LLMs with rationales to improve their accuracy. While methodologically the paper follows existing work in not so novel ways, the proposed procedure leads to strong enough empirical gains to be of wide interest to the community. The authors are encouraged to revise the paper incorporating the reviewer's suggestions.
confidence: Certain
Decision: Accept