Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an interesting approach to align the language models (GPT-3 in this case) with their users' intent. This is very important because often these language models generate outputs that contain untruthful and toxic information that is not helpful to the user. The paper proposes a three step framework to improve the model’s alignment with its users: (1) supervised finetune of the GPT-3 models with a dataset of labeler demonstrations. (2) train a reward/ranking model using the fine-tuned model and a dataset of labeler annotations comparing model outputs. (3) further fine-tune the GPT-3 model with reinforcement learning using the trained reward model.  This paper has thorough human evaluations on the resulting fine-tuned model (called as InstructGPT). The paper shows that InstructGPT improves on various aspects including truthfulness and reduces toxic output generation in comparison to the baseline GPT-3. 
strengths_and_weaknesses: Strengths: 
1)  The proposed approach is very interesting and novel. A similar idea has been applied previously for improving summarization but the current paper addresses a much broader distribution of tasks. 
2) The human data collection process is well documented (in the Appendix). 
3) The experimental results are very thorough with a lot of human evaluations. Human evaluations make the most sense to show the effectiveness of proposed model to cover a broad distribution of tasks

Weaknesses:
1) The paper provided only minimal information on the deduplication process of prompts, that too in the supplementary. I think it is important to clearly mention the full details of this process. It is not clear how the paraphrased prompts (or prompts with the same example or task but with different templates) are handled. 
2) Also, it is unclear whether there is any overlap on the training data and NLP datasets used in the paper for some ablations (e.g., RealToxicityPrompts and TruthfulQA). It would be useful to share some analysis on the data distributions. 
3) Sorry if I missed it somehow, but I did not find any “empirical” evidence for the claim: “1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3”

questions: 1) Can you provide some empirical evidence on by how much margin 1.3B InstructGPT is better than 175B GPT-3?
2) Can you provide evidence that there is no contamination of training data and evaluation datasets (NLP datasets) used in the results section? 
3) How is the hallucination rate of a few-shot setup in comparison to zero-shot setup (for both GPT and InstructGPT)? 
4) It seems that for RM model training, the paper used more than double the data used for SFT. Is there any reason for this? How much minimum data is decent enough for the RM model training? 
5) Do you plan to release the data used in SFT and RM?

limitations: Adequately addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the challenge of aligning large language models with user intents. They introduce a three-step process to do so: 1) Train supervised models on (human-collected) prompts and the (human-labeled) outputs 2) Use the models to generate labels which are then ranked by humans according to the desired metrics (bias, toxicity, etc.). Then train a Reward model on the data 3) Fine-tune a supervised policy using PPO algorithm with rewards produced by the Reward model. 

They show that the resulting models are better at following user instructions. They make fewer mistakes, use less offensive language, and have fewer hallucinations, as measured by automatic metrics and human evaluation.
strengths_and_weaknesses: Strengths:
- This work takes an important step in reducing bias, toxicity, and other ethical issues concerning large language models.
- The results align with the premises and claims. The discussion provides a good comparison between GPT3 and InstructGPT3 capabilities and where and why InstructGPT3 does not improve over GPT3.
- The paper is clear and easy to follow. It does not overclaim and cites relevant work where appropriate. 

Weaknesses:
- This approach relies heavily on human-collected and labeled data. Accessibility of such data, esp. for small research organizations are not clear. Also, the compute power required to fine-tune these models is quite high. As a result, the benefits of this research might not be applicable to the larger research community. The authors do not mention whether they intend to release their dataset or software.
- The approach does not provide a comprehensive solution to the general problem of bias and toxicity. The current approach is mainly data-driven and relies on reward signals which amplifies user preferences, etc. Thus an ill-intended user is now better equipped to mis use such models for their own benefit. 
- It's not clear whether current approach can be applied to smaller LLMs (~1-10B parameters).

questions: - Is this approach specific to large language models or can it be used for smaller LLMs (~1-10B parameters)?
- What are the possible solutions to avoid manipulation or misuse of alignment by certain individuals? Is it possible to introduce "inhibition" so that models refrain from following user's instructions if deemed biased, toxic, or inappropriate?
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies fine-tuning language models to produce outputs thare are more aligned with what users would like them to do. Given the increases focus on language models and their failure modes (hallucinations, toxic responses etc.) this is an important research direction.
strengths_and_weaknesses: + large scale evaluation in terms of model sizes (GPT-3 size) and number of considered tasks
+ effective approach to improve outputs of LMs with respect to what users expect
+ the problem is clearly significant as language models are an increasingly important research direction

questions: * Is the comparison between the SFT and PPO models fair? SFT was trained on 13K labeled prompts while PPO collectively uses 13K + 33K (both labeled) + 31K unlabeled prompts. Disclaimer: I did not read the appendix.
* Do you truly need a reward model? You could directly train the final model on the 33K PPO dataset using expected risk which uses the labeler rankings as a reward. You would not use the 31K unlabeled prompts, but the result would give you an indication of the value of these unlabeled prompts.
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a instruction tuning process for gpt3. It involves three stages: (1) Collect demonstration data, and train a supervised policy. (2) Collect comparison data, and train a reward model. (3) Optimize a policy against the reward model using PPO. It is compared to a few ablation baselines, or models trained on other instruction datasets.
It is compared with a number of strong baselines, and the performance is strong. It is shown that with the instruction tuning, the model can beat a 100x larger LM. (but keep in mind that larger LM is not deliberatedly tuned on instructions.)
strengths_and_weaknesses: Strength:
The collected data would be a good resource for further research into instructed LMs.

The comparison is solid and the performance is strong. It is shown that with the instruction tuning, the model can beat a 100x larger LM. (but keep in mind that larger LM is not deliberatedly tuned on instructions.)

Weakness:
Collecting data/ranking, is of course, costly.

It seems to me that this work is kinda typically adding more data/human labor, and the performance goes up. While the performance is good, the novelty is limited. 

More analysis experiment could be done. (see my questions)
questions: Would one ablation baseline be directly do RL with human feedback ? (without training the reward model)

If you have a smaller size of human feedback, how would the performance be affected?
limitations: Yes.
review_title: null

Meta-review: metareview: This paper demonstrates that a smaller model (1.7B InstructGPT) trained with human feedback (using a learned reward model and PPO) is competitive with a much larger pretrained model (175B GPT-3). The results are clearly convincing. The main downside is that there isn’t much analysis on the proposed method (e.g. the influence of the RL algorithm and the amount of human feedback needed). That said, the approach is novel and the results are impressive, which points out new directions for collecting better pretraining data. Thus, I lean towards acceptance.
confidence: Less certain
Decision: Accept