Kati8/ICLR_2023_2025 huggingface dataset:
authors: Puhua Niu
authors: Shili Wu
authors: Mingzhou Fan
authors: Xiaoning Qian
reviews: 
Comments: This work shows that there is an equivalence (in expectation) between training GFlowNets (GFN) with the Trajectory Balance objective, and a special form of non-stationary-reward policy gradient (PG) method. This is derived through the equivalence (in expectation) between TB and a forward KL-Divergence shown in prior work. This enables the algorithm to be treated as a PG method, onto which one can apply methods like TRPO.

The authors perform some empirical evaluations of their method on grids, bit sequences, and Bayesian networks.
Strengths: This paper continues the work of relating GFlowNets and other frameworks, here RL, and of building novel methods upon that. In that sense the work here is a novel and interesting contribution.
Weaknesses: The two main weaknesses of this paper are presentation and strength of empirical results.

Presentation: what the authors are trying to do isn't incomprehensible, I kind of get it and staring at the appendix helps, but I honestly think  section 3 could be heavily reworked to be more concise and tell a story, e.g. make it clear at every step what the symbols are, where they come from and how they make us progress in going from TB to RL with a non-stationary reward.

Results: 
- Hypergrids and bit sequences are not very hard nor interesting problems, they were meant to be toy examples in the papers in which they were introduced. Learning to produce Bayesian nets is interesting, but I don't think the authors properly compare to Deleu et al.
- There isn't much in terms of "why does this work [better]?" which is usually a good thing to have in a paper.  


Now I realize this paper may be more "theory" than experiment, but it does feel unfortunate that there's a mismatch between the "we're doing PG therefore we can bring in the whole RL arsenal" and the lack of big results.

This may be tangential, and just a semantics argument, but I find it slightly far-fetched to call this a policy gradient algorithm, or maybe even RL. The reason: the reward changes at every state, _and_ in a self-referential manner. The closest we have is maybe methods like Random Network Distillation, but even then the update of $R$ is decoupled from the update of $\pi$. Just because the update equation "looks like" PG, doesn't mean it's the most "scientifically useful" way to describe it. Methods like TRPO certainly assume a fixed reward, so I don't even know if what's being done here is correct.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: The paper proposes a new approach for improving the training of GFlowNets from the perspective of policy-based rewards and guided backward policy. The proposed method includes a coupled training strategy that jointly solves GFlowNet training and backward policy design, which aims to improve training efficiency. The method is tested a few benchmark including hypergrid, bit sequence generation and bayesian structure learing.
Strengths: This paper tries to connect the field of RL and GFlowNets, which forms an interesting problem. The experimental part is well-explained.
Weaknesses: The main concern for the paper is that the experimental evaluation part is too toy -- which focuses on some synthetic problems including hypergrid and bit sequence generation only, and the evaluation metric does not follow previous paper, and some of the claims are not well supported.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 1 poor
reviews: 
Comments: This paper reformulates the the GFlowNet training problem as a RL problem over a special MDP. A policy gradient method is proposed to do RL training in this setting. The authors also formulate the design of backward policies in GFlowNets as a RL problem and propose a coupled training strategy. Theoretical and empirical results are provided to validate and help understand the proposed method.
Strengths: **originality**
- The main novelty of the paper is on formulating the GFlowNet problems as RL problems and allow the use of RL in these problems. The problem formulation and the training strategies proposed can be considered novel results. 
- The theoretical results are interesting and can be considered novel

**quality**
- The paper is overall well-written, with only minor isuses

**clarity**
- The paper is quite clear

**significance**
- The new problem formulation and training strategies discussed in this paper allow RL to be used in GFlowNet training problems, together with theoretical and empirical results, can be a significant contribution.
Weaknesses: Related work:
- It seems to me the related work can be improved, for example, what is the most relevant standard RL algorithm? And how is your proposed policy gradient method different in design? Additionally, GFlowNet and RL are discussed together in some other papers in the literature, such as the "GFlowNet Foundations" by Bengio et al. How is the analysis in your work related to these previous works? 

Additional technical details: 
- Would be nice to have more technical details, for example, how does your method compare to others in terms of computation efficiency? Will formulating into RL problems make training much slower. 

Ablations:
- I think more ablation on the proposed method with different hyperparameters can help us understand better how it can be applied/tuned to different problems. 

Other issues: 
- End of page 5, a figure reference seems to be broken. Is this figure in the paper?
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 4 excellent
reviews: 
Comments: This work introduces a way to train GFlowNets through policy gradients from the RL literature. The work proposes a coupled training strategy to train the forward and backward policies in GFlowNets.
Strengths: 1. This work discusses an interesting direction of training GFlowNets and using learning from RL policy-based methods to introduce a policy-based GFlowNet.
2. The discussion and perspective around backward trajectories is interesting and a useful way to improve GFlowNet training.
3. The gradient equivalence discussion and analysis is useful to understand the theoretical claims and some of the motivation behind this work.
Weaknesses: 1. The related work section could be made more exhaustive by adding the other GFlowNet losses and their references.
2. It will be useful to expand the number of environment configurations. For hypergrid, N=2 and N=3 are the only options used and using a higher value will help.
3. Previous work has analyzed number of states visited for hypergrid domain, which has not been included here. 
4. Adding other GFlowNet based baselines, such as Detailed Balance, would be useful as it is an important and commonly used objective.
5. Some of the domains that were used in the previous work have been not included here. In order to corroborate that the proposed method can be used over different settings, including them would be helpful.
6. For bit sequence domain, including all bits used in the previous work would be beneficial for a fair comparison across methods.
7. Related work could be expanded and made more exhaustive, including the recent improvements in GFlowNets as well.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good