Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a pipelined method for reasoning tasks with language models. The proposed pipeline has two parts: a rational generation module and a reasoning module. The proposed method has two major novelties: 1) the rationale generation uses a frozen LM with few-shot prompts. 2) The reasoning module uses regularizations to improve the faithfulness of utilizing the rationales.

The paper compares the proposed method with several baseline methods and shows that the proposed model achieves better performance on 4 QA tasks which require external knowledge. They also test the out-of-distribution performance and robustness of noisy rationales of their proposed method.
strength_and_weaknesses: Strength:
- The paper proposes a novel LM pipeline that rationalizes with prompt-based learning and reasons via counterfactual regularization.
- The empirical results show the advantage of the design choice: 1) pipeline-based model over unified model and 2) regularization method for reasoning module
- The paper is well-written

Weakness:
- The evaluation is only about the proposed method and its own variants (baselines). But the performance is much lower than the state-of-the-art results. For instance, the best performing CSQA system has accuracy around 80% (https://www.tau-nlp.sites.tau.ac.il/csqa-leaderboard2), and the proposed method only achieves 60%. Of course, the best systems use additional knowledge sources with careful task-specific tuning. But the gap makes me wonder if the proposed method could help upon the state-of-the-art system.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is well-written, and I can understand technical details.
- Quality: The paper is solid with clear motivation and detailed empirical results.
- Novelty: The paper is novel for its regularization on reasoning module.
- Reproducibility: Some hyperparameters are missing but one should be able to reproduce with some effort.
summary_of_the_review: The paper proposes an interesting method combining LMs with different capacities for open-domain QA tasks. The first LM generates answer-specific rationales for the second LM to reason about. Two regularization objectives are proposed to improve faithfulness. The ablation study on 4 widely used benchmark shows promising performance of the proposed method. However, the performance is far away from the state-of-the-art system for the corresponding tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an LM pipeline that first generate free-text rationales and use the generated rationales to fine-tune a reasoning module such that it makes the prediction that relies on the rationale as much as possible. A regularization scheme is proposed to mitigate issue where rationales might be ignored.

The method is evaluated on in-domain and out-of-domain setups using accuracy and faithfulness as metrics.
strength_and_weaknesses: Strengths:

- The method is simple, sound, and makes intuitive sense. To my knowledge, this is the first work to
- The general evaluation setup is largely appropriate such as choice of datasets, ID vs OOD evaluation and measuring faithulness

Weaknesses:

- The current formulation only allows the method to be applied to multi-choice classification problem or where the classification target space is not too large
- The baselines are not on the same parameter scale especially the w/o Rationalization baseline. It is unclear whether the performance gain is coming from using a strong rationale LM or that this framework in-general is better than the other options. (This might explain why ID even increases.)
- One way control for the issue above might be using rationale modules of different sizes, and use a small LM (<1B) to fine-tune on the rationales generated by these different modules, and eventually stack it with the proposed PINTO training procedure. At the very least it might be a better comparison to the Fine-tuned self-rationalization baseline.
- It might be a stretch to draw any conclusion from Figure 4 as mentioned in section 5.2 due to the high variance and lack of trend
- The issue of answer leakage is mentioned in section 3, I am not sure how the proposed framework circumvent such an issue. With PINTO, the rationale can still contain answer information right? (Please correct me if I misunderstood.)
- Would be nice to have variance statistics
clarity,_quality,_novelty_and_reproducibility: - The paper is well written, appropriately organized, and easy to follow
- The method is novel as far as I am aware
- The method is straightforward and practical
- The general quality is good, but some limitations remain (will increase the scores if the authors can address them)
summary_of_the_review: The paper presents a practical method to solve multi-choice classification problem using rationales. The paper is well-written and provides a new way of incorporating rationales to improve OOD performance. Despite obvious limitations and some issues in how the baselines are constructed, there is no major flaws.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors, presents PINTO a two stage pipeline to improve rationale-based language reasoning. The first step is to prompt a language model (20B in this case) to generate rationale given a question, while the second step is to fine tune a smaller (<1B) LM with the pair (Question, generated rationale) -> Answer.

The authors experimented PINTO on the CSR benchmarks (CommonsenseQA, StrategyQA, OpenBookQA, QASC). The results show that PINTO works better across the four dataset when compared to other baseline settings. The authors also reports interesting ablations and case studies. 
strength_and_weaknesses: Weaknesses
It's hard to grasp the novelty and the contribution of this paper. For instance, the main message, or at least if I understood this paper correctly, is that mid size LM (20B) can be use to generate rationale (which per se is not novel) and the by fine tuning on it a smaller model the performance are "close" to simple baselines (which are not SOTA nor this is novel per se). Some detailed points:
- Prompted Self-Rationalization uses larger model to solve the task directly, however the model is very large. Then why not use those generation to train a smaller model as in [1,2] directly? 
- Based on the provided analysis that  more faithful rationales can improve the performance, what is the quality of the generated rationales? is there a small scale human eval to quantify this. 


[1] https://arxiv.org/abs/2110.07178
[2] https://arxiv.org/pdf/2104.08826.pdf
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is not very clear on the contribution and something misses baseline in the main table (e.g. Table 2, some results are in the caption of the table). More over the name of the baselines (e.g., Standard or Dropout) are very hard to follow. 
Quality: the paper has some merit. 
Novelty: not very novel (see weakness). 
Reproducibility: with some effort, but it's reproducible.
summary_of_the_review: The paper lack of a major contribution and major focus. After the discussion between the reviewers, I decide to keep the increase my score (post-rebuttal), to 5.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an algorithm to leverage LLM's rational through prompt, to improve final downstream models' performance by taking the computation into account. Specifically, to save human efforts, the authors propose to use LLM to generate rationals. It then train a QA model based by giving the generated rationals. To alleviate the shortcut, they further propose a counterfactual regularization to encourage the model to learn to use the rational instead of learning the shortcut. 
strength_and_weaknesses: * The main strength of the paper is paying attention to the computational budget, which is one motivation behind the algorithm. I believe it will become an important direction as the pretrained models are getting larger and larger. 

* The algorithm design is simple. On the other hand, the non-trivial part is the potential shortcut issue, and the authors propose a counterfactual regularization to resolve it. 

### Questions

* Although I understand one argument of the paper is computation efficiency, I'm still curious about what is the performance of the fine-tuned LLM on the benchmarks. The used LLMs are still "relatively" small compared with GPT-3 and PaLM. It would be nice to have these numbers to serve an upper bound.

* For the OOD results, the proposed algorithm leverage the rational from the LLM. However, it has been demonstrated that LLM has strong zero-shot performance as well as great generalization capability.  Therefore, regardless the training data it used, when testing on the different dataset, as long as the rational provided by LLM is faithful, isn't it expected that any model built on top of it can generalize well?  In this case, is it fair to argue the proposed model has good generalization capability? or it's  actually  benefitted by LLM's good generalization capability?

* Question about robustness of the perturbed data.  When the rational is wrong, the model can still make right decision. Although the authors try to argue it's not a contradictory results, it does sound contradictory to me that, if the model is encouraged to take the rational into account when making decision, how can it not be misled if we feed a wrong rational? if it can be robust, isn't it not leveraging rationals when making decisions?  Could the author comment more on it? 

* The rational qualities are also known sensitive to the prompt design. How does it affect the final algorithm performance?
clarity,_quality,_novelty_and_reproducibility: * Novelty: see bullet 1 above
* Quality: the paper is well written and the the experiments are thorough.
* Reproducibility: the code is provided
summary_of_the_review: The paper focus on a computational efficient algorithm by using LLM, which seems important to me. The overall quality is good as aforementioned. There are some questions not clear to me, but overall I think it's a good paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method (PINTO) for question answering via generating free-form text rationales using language models. The key difference with prior work that has also explored similar setups is the fact that this paper uses LM prompting to generate rationales for each answer choice (without any finetuning) and then fine-tunes a separate reasoning model that produces plausibility scores for all the answers along with their generated rationales. Experiments are performed on four QA datasets and although the proposed method does not outperform all baselines (e.g. GPT-3 w/ self-prompting) and is worse than SOTA (e.g. self-consistency for chain of thought, Wang et al., 2022), it uses less parameters overall and the authors also conduct several analyses that provide interesting findings. The authors also responded to several reviewer concerns around novelty and added new baselines to the paper based on reviewer suggestions. The reviewer discussion also raised some points around the clarity of the messaging and the scope of the proposed method.
If accepted, I encourage the authors to 1) more clearly specify that the key contribution/focus is on model efficiency (in terms of number of parameters), since other methods like self-prompting also do not require human-labeled rationales, and 2) more explicitly state the scope (in abstract and intro) to be choice-based question answering tasks.
justification_for_why_not_higher_score: See reviewer discussion above
justification_for_why_not_lower_score: The benefits of the paper seem to slightly outweigh the weaknesses.
Decision: Accept: poster