Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new objective function for Generative Flow Networks (GFlowNet), which aims to generate objects with probability proportional to a given energy function. The proposed objective function is termed trajectory balance. This objective comes from a constraint that the flow of each complete trajectory should be consistent by either forward computation or backward compution. This constraint is formulated by Eq. (13), and is further converted to an objective function given by Eq. (14).

Trajectory balance suggests parameterizing the forward and backward probabilities (P_F and P_B) between connected states, as well as the flow volume Z of each state. The parameters can be trained by stochastic gradient descent. The authors further prove that, if Eq. (14) is minimized to zero, the GFlowNet can generate samples exactly from the target distribution. Experiments further show that trajectory balance outperforms the conventional two objective functions (flow matching and detailed balance) in terms of convergence, diversity of generated samples, and robustness to long action sequences and large action spaces.
strengths_and_weaknesses: **Strengths**:

This paper proposes an effective new objective function of training GFlowNets termed trajectory balance (TB). As a new learning method for generating complicated compositional objects, GFlowNet has recently gained an increasing research interest. Compared with conventional objectives, i.e., flow matching (FM) and detailed balance (DB), trajectory balance is empirically observed to overcome the difficulty of slow credit assignment, and gains a faster convergence speed and more diversified generated samples. This paper clearly sheds light on the research of generated flow methods, providing a new inspiring methodology for applying GFlowNet to various domains.

**Weakness**:

Although this paper demonstrates the success of TB through substantial experiments, it is not that clear why it performs better than FM and DB. It is argued that one benefit of TB is that the global minimizer can define a policy that samples exactly from the target distribution. However, this also holds for FM and DB. Another benefit claimed for TB is the improved credit assignment, but very few discussions are presented. Specifically, why does considering complete trajectories lead to better credit propagation in GFlowNet training?

Another concern is that the paper claims that the global minimizer of TB gives rise to exactly the target distributions. However, this is on the assumption that the parameterization of GFlowNet has sufficient representation power, i.e., Eq. (14) can be trained to exactly zero (Proposition 1). This may not always be the case if the task is too complicated or the training data is non-realizable. The authors may want to remind the readers of the above issue in the introduction and relevant discussions.
questions: Please refer to the weakness.
limitations: Please refer to the weakness.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new objective for GFlowNets and shows improved performance in experiments.
strengths_and_weaknesses: Strengths:
1. Clear presentation and introduction of the background of GFlowNets.
2. Theoretical backup on the proposed objective.
3. Clear performance improvements.

Weaknesses:
1. Figures 2 and 3 are a bit hard to read. 
questions: Can we add more explanations on how to interpret figure 3?
limitations: N.A.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new learning objective for GFlowNets, trajectory balance TB), as a more efficient alternative to previously used objectives. A GFlowNet is a trained stochastic policy or generative model, trained such that it samples objects $x$ through a sequence of constructive steps, with probability proportional to $R(x)$, where $R$ is some given non-negative integrable reward function.

More specifically, GFlowNets aim to approximate a Markovian flow F on G such that

$F(x) = R(x)~~ \forall{x} \in X$

Generally speaking, a GFlowNet could be any learning algorithm consisting of:
(1) a model capable of providing the initial state flow and the forward action
distributions for any nonterminal state s, determining a Markovian flow F.
(2) an objective function, such that if the model is capable of expressing any action distribution and the
objective function is globally minimized, then the constraint is satisfied for the corresponding Markovian flow F .

This work focuses on finding a more appropriate objective function, distinct from those already tested and used in previous works. Namely,
Flow matching (FM) objective and Detailed balance (DB) objective. Both are inefficient in credit propagation across
long action sequences.

The proposed alternative is called "trajectory balance", as the title indicates. Basically, this approach relies on the "trajectory balance constraint", which in fact balances the Markovian flow F (and its forward and backward policies) over complete trajectories. 

A set of experiments to test the performance of TB is provided in Section 5. More specifically: (1) The use of an Hypergrid environment to understand better the behavior with respect to MCMC and PPO, and more interestingly when using GFlowNets with FM and DB. (2) Experiment on Small drug molecule synthesis where TB, FM and DB are compared. (3) Autoregressive sequence generation in (3.1) bit sequences' problem and (3.2) Anti-Microbial Peptide (AMP) generation. 

strengths_and_weaknesses: Positive aspects:

* Well presented and original idea (within GFlowNets' world)
* Good quality and clarity, even though in the experimentation part there is little space for further analysis
* Already successfully applied (as indicated in the last paragraph of Section 1)

Less positive aspect:

* It could be a subjective point, but the contribution is quite specific to be applied in a relatively new "tool/model" and solving a quite particular task on it.
questions: 1- One evident question would be: why publish this idea now? I mean, it results at least surprising that it has been already used in papers already published. I see all of them belong to 2022, so it may be for distinct reasons, but I feel curious about the chronological order in publication.

2- Runtime gain (numerical speedup) is only explicitly commented (x5) in the drug molecule synthesis problem. I assume this is because it is more significant there. However, how does it affect the other problems? Is there any quantitative estimation of the general profit with respect to FM and DB? What aspects/factors this improvement will depend on?
limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
Trajectory balance: Improved credit assignment in GFlowNets
In this paper, the authors improve the original generative flow networks’ objective function by modeling it on top of the entire trajectories generated from the policy network. This new objective function improves the convergence, diversity, robustness etc. compared to existing methods.

strengths_and_weaknesses: 
Unfortunately I don’t have any experience or knowledge working with generative flow networks, and therefore I can only provide my evaluation and feedback as an outsider.

1) Originality: The paper discusses a relatively new technique of generative flow network and proposes a novel formulation of objective functions.

2) Quality & Clarity: The paper is very well written. The preliminary section provides complete and valuable background knowledge for readers not familiar with the topic. 
The paper is supported both with solid math and well designed experiments. The results of the experiments are also fully discussed.

The code is given, and the results can be easily reproduced.
The comments and remarks are interesting and inspiring. The related work is very complete and adequate, and the connections to some other algorithms or methods are discussed.

3) Significance: In this paper, the generative flow networks are extended and potentially can be used in more complicated real-life problems. Although it is not clear how well the method can be scaled to high-dimensional continuous state problems such as robot arm target reaching tasks.


questions: 
1) I am entirely sure why the proposed algorithm improves “credit assignment”. Maybe the authors will elaborate on this? I do see improved results in terms of the correlations between final reward and policy probability across different tested environments & tasks. 
But why does it improve the credit assignment? Maybe it improved something else instead?
Is there a metric or discussion to support this claim?

2) Does the algorithm scale to high-dimensional problems? A lot of the robotics tasks also involve actions that incrementally modify the state of the robot to solve the tasks (in gym for example). How well does the proposed algorithms work with these problems?
Also does the algorithm work with problems that have continuous state space?

3) The fonts of the figures are way too small (for example Figure 2, Figure 4), and it is hard to read without going back to the pdf version.


limitations: 1) As far as I know, I don’t think it will have any potential negative societal impact directly.
2) In terms of the limitations of the paper itself, I think it would be great for the authors to comment on how well the algorithm will scale for high-dimensional robotics problems. As I have no related knowledge, I am not entirely sure how difficult the tasks of drug molecule synthesis and AMP generation are and how well the baselines should perform.
review_title: null

Meta-review: metareview: The paper presents a new method for credit assignment in training GFlowNets, based on trajectory balance equations. All reviewers were in agreement the method offered theoretical guarantees as well as strong empirical results, and offers promise in terms of developing the growing field of Generative Flow Networks.
confidence: Certain
Decision: Accept