Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: The authors use GFlowNets to tackle the exploration problem in RLHF. This allows earning policies that are proportional to the score but are not focused on optimizing for the top scores since they can often be noisy. The authors evaluate the performance of GFlowNets on a point robot task with multimodal reward distributions and demonstrate superior performance against some SOTA RL baselines. They additionally demonstrate the robustness of GFlowNets against noisy labels.  
strengths_and_weaknesses: Strengths:
- This is a promising research direction, and very well timed, since with the onset of LLMs RLHF is coming increasingly in the limelight
- GFlowNets can be very powerful for dsitributional reward settings and noisy preferences, and I feel exploring this direction might open up potential applications into many other multimodal scenarios
- The experimental setup has been communicated in a clear and reproducible manner

Weaknesses:
- Generally, I think the paper is well-written, but needs some adjustments in terms of clarity of thought in certain sections and grammatical errors. Please see the suggested changes for that
- To me, what comes out as a potential conclusion is that methods that work with the reward distribution can potentially perform better than RLHF methods that optimize for the best reward. This is not a new argument in the standard RL literature, since methods in Distributional RL (e.g. C51, IQNs) [1,2] have already talked about the benefits of distributions. What did not come out very clearly to me was why exactly we need GFlowNets. Potentially, this can be addressed by either showing that GFlowNets are better than these methods (i.e. adding them as baselines, or citing publications that already talk about this issue) or discussing how the setting that the authors tackle is different from what these methods address. I would recommend looking into this and making the necessary changes.

[1] [https://arxiv.org/pdf/1707.06887.pdf](https://arxiv.org/pdf/1707.06887.pdf) 
[2] [https://arxiv.org/pdf/1806.06923.pdf](https://arxiv.org/pdf/1806.06923.pdf)
suggested_changes: - Grammatical mistakes. Please check the paper thoroughly for these. Here are some examples:
    - Therefore, Generative Flow Networks (GFlowNets) Bengio et al. (2021)  **is** recently proposed to make up for the insufficient exploration of RL … — should be **have been**
    - … and conducting experiments in a **diversity** reward distribution environment — should be **diverse**
- I think the environment needs a bit more explanation in the appendix
    - What is a preference, and how does it exactly work?
    - What is the scale of reward achievable — when do we consider the environment solved?
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The paper introduce GFlowNetsHF - a framework incorporating GFlowNets instead of Reinforcement Learning (RL) to improve upon the Reinforcement Learning with Human Feedback (RLHF) approach for improving the performance of large language models.
strengths_and_weaknesses: Strengths:
The authors propose a new method based on GFlowNets for incorporating human feedback that can outperform RLHF because it can score answers and not just rank them like RLHF. Thus, it can learn the distribution of rewards. This method is also more robust to noisy labels which is not an uncommon occurrence in human-labeled data. 

Weaknesses:
- Quite a few assumptions have been made about the background of the reader.
- The paper can definitely improve upon the clarity and completeness of the work. Many fundamentals for understanding the research problem have been glossed over.
- As for reproducibility, the experimental results section is poorly explained and the appendix does not cover all the details of the experimental setup.
suggested_changes: It would be great if the authors could explain the details of the paper better. The problem statement, motivation, background and approach must be thoroughly covered in the main paper and the appendix could contain the experimental details better.
review_title: null

Decision: Invite to present