Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper propose a novel GFlowNet training objective called subtrajectory balance $SubTB(\lambda)$ that can learn from partial The paper proposes a novel GFlowNet training objective called subtrajectory balance $SubTB(\lambda)$ that can learn from partial action subsequences of varying lengths. Empirical results demonstrate that $SubTB(\lambda)$ can improve the convergence of GFlowNets in some simple environments.
strength_and_weaknesses: The proposed method is novel, and the paper is well-written. However, I have some concerns and thus do not recommend acceptance. 

Q1. My biggest concern is that the experiments are quite simple. How well does $SubTB(\lambda)$ perform over complex tasks in high-dimensional environments?

Q2. The motivation of $SubTB(\lambda)$ is not very celar. More theoretical analysis is required to explain why $SubTB(\lambda)$ improves the convergence of GFlowNets.

Q3. The current version lacks discussion about the limitations of $SubTB(\lambda)$.

clarity,_quality,_novelty_and_reproducibility: The paper is novelty, but the motivation is not very clear.
summary_of_the_review: I have some concerns and thus do not recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers the recently developed GFlowNets, a family of models using a policy network to sequentially sample from an unnormalised target distribution defined on a discrete domain. Three main objectives are typically used to train such models, each enforcing a flow matching 
(ensuring same incoming and outgoing flow at each node), a detailed balance (at each transition) and a trajectory balance condition (at global level). The work proposes a new objective unifying the last two conditions. The objective (i) extends the notion of balance to portions of trajectories, (ii) computes the balance contribution for each possible sub-trajectory and (iii) finally averages all these contributions. Importantly, the average used in the objective is governed by a single parameter, which is experimentally shown to control the bias-variance tradeoff. Indeed, a large parameter value offers similar behaviours to the trajectory balance objective (low bias, high variance), while a small parameter value shows similar effects to the ones obtained by the detailed balance objective (high bias, low variance). This bias-variance trade-off enables to reduce the time required to train GFLowNets and to generate better and more diverse solutions compared to the ones from previous objectives, as demonstrated by experiments on a hyper grid toy case, on small molecule synthesis and on three sequence generation tasks.
strength_and_weaknesses: **Strenghts**
- The paper is well-written. However additional details can be included in order to make it more self-contained (see comments later)
- The idea is novel, original and the technical content is correct.
- Several tasks are used for experimental evaluation. 
- Sequence generation tasks provide convincing evidence.
- Code is available. However, I haven’t run it to check if experiments are reproducible.

**Weaknesses**
- The sampling strategy doesn’t come with statistical guarantees on the quality of the obtained samples (like MCMC does). However, this is an intrinsic limitation of GFlowNets rather than of the proposed strategy.
- No supporting theory is developed to analyze the bias-variance tradeoff.
- Some experiments (hypergrid, bit sequences) miss baselines. Additionally, some experiments miss explanations of observed phenomena. Please refer to subsequent comments. 

**Clarifying questions to improve the quality of the paper**

About training details:
- Can you provide an algorithmic table to describe the training of GFlowNets and the computation of the proposed objective in order to make the paper self-contained? Additionally, is the partial trajectory objective computed at the end of or during each episode in a TD-like fashion?

About hypergrid experiments:
- The experimental setting allows only for actions that increase coordinates at each time step. This is quite a strong condition and I wonder what are the effects of such condition. Specifically, It seems that the proposed approach is biased and it doesn’t satisfy general conditions, like detailed balance and ergodicity used in Markov chains (Note also that the proposed strategy is in effect a Markov chain). Consequently, the proposed strategy doesn’t provide any guarantee of correct sampling from the target. For instance, this is visible in Figure 3 (e.g. 2-D grid for 16x16, 60x60 and 16x16-hard and almost all cases with 4-D grid), where even the unbiased approach using the trajectory balance objective doesn’t converge to 0. Note that the phenomenon becomes even more pronounced with the increasing of dimensions. Can you comment on these aspects? Additionally, how does the proposed strategy perform by also allowing actions that decrease the coordinates?
- The baseline using the detailed balance objective is missing in Figure 3. Can you please include it in order to have a more complete picture of what is going on?
- Regarding the analysis of gradient bias. Why in Figure 5 (bottom) small-batch SubTB provides a better gradient estimate than small-batch TD? The explanation in the text seems overly stated (indeed the superiority is visible only at 500 training steps and in between 1500-2000 steps), can you be more precise? Additionally, in Figure 4 (right), SubTB is extremely biased compared to TB even for full batch size (K=10). Can you elaborate more on that?

About bit sequences experiments:
- Can you add the baseline using the detailed balance objective in Figure 7?

clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is clear and well-written. Some suggestions are provided above to make the paper more self-contained.

**Quality**

The technical content of the paper is correct and an extensive experimental evaluation is conducted in order to support the proposed idea.

**Novelty**

The authors propose a new objective unifying existing criteria used to train GFlowNets. The idea is inspired by unifying attempts in reinforcement learning between temporal difference and Monte Carlo estimators. An extensive experimental evaluation is conducted to fill the missing gap on the theoretical side.

**Reproducibility**

Code is made available in the submission. I haven’t run it to check its reproducibility though.
summary_of_the_review: The paper proposes a new objective to train GFlowNets. The objective allows to tradeoff between existing local and global criteria (using the detailed and the trajectory balance conditions, respectively). Extensive experimental analysis provides support on the benefits of this unified view. The quality of the paper can be improved by including additional details about the training, including missing baselines and smoothen some strong statements in the explanations of the experimental analysis. Overall, the paper contributes to the advancement of GFlowNets from an empirical perspective.

---- POST REBUTTAL ------

First of all I would like to thank the authors for the clarifications to my questions. Also, I appreciated the inclusion of the baseline (based on detailed balance) in Figure 3, which allows for a better understanding.

Indeed, experiments in Figure 3 are the most important in the paper, as providing an empirical analysis of the bias-variance trade-off and giving insights about the behaviour of the strategy. From these figures I currently see three main potential issues:
1. While it is clear that SubTB($\lambda$) has the "lowest" curve, its convergence occurs at the same number of training iterations of the other approaches. Therefore, why is it possible to claim that the convergence is accelerated (as mentioned for example in the abstract)?
2. It is unclear why the baselines have strange behaviour. Indeed detailed balance is achieving both lower bias and lower variance than trajectory balance. This suggests that there is some additional factor influencing the results, probably the hyperparameters?
3. Hyperparameters seem to play a major role also for SubTB($\lambda$). Indeed it is unclear what is the effect of initialization (for instance note that in 16 x 16 hard the proposed strategy seems to start from a different condition from the other approaches) as well as how $\lambda$ is chosen.

In summary, the proposed objective unifies existing local and global objectives for GFlowNets and represents an interesting and novel idea. However, in absence of a theoretical analysis, the paper should provide a solid sensitivity analysis on the hyperparameters and utlimately a methodology for choosing them.

Based on these considerations and also based on others' reviews, I reduce my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper discusses an improved training procedure to solve GFlowNet problems. The paper begins by outlining GFlowNets, a DAG with exactly one root node where "actions" are decisions about which edge to follow in the DAG. When the agent reaches a terminal node (a node with no edges leaving that node), the "episode" ends. The goal is to find a policy such that the probability of reaching a terminal state is proportional (up to a constant) to a reward function R.

GFlowNets have strong assumptions not in MDPs (absence of cycles) and a different objective (probability of reaching terminal states), the authors review previous work which shows one can solve the GFlowNets problem with a solution involving an edge flow function which satisfies certain constraints. 

From there the authors outline previous methods used to solve the edge flow problem, such as Detailed balance and Trajectory balance all of which involve minimizing an objective, which once minimized is guaranteed to result in a policy matching the desired distribution.

The authors note that in  Malkin et al. (2022), it is shown that a flow solves the balance equation from the DB method if it holds for sub-sequences of nodes in trajectores. Using this observation, the authors present a new objective subtrajectory balance objective, resulting in their method.

They then hypothesize on why their new method brings a benefit.

The authors then repeat the experiments from Malkin et al. (2022), showing better results and an experiment on gradient variance and gradient bias.



strength_and_weaknesses: Strengths:

- well written, structured and easy to follow
- idea is plausible and understandable
- experiments improve on state of the art

Weaknesses (in no particular order)

Novelty+Relevance: the paper uses a slight modification on Trajectory balance objective, replacing whole trajectories with sub-trajectories. In the paper, the authors show how it provides better experimental results but don't argue how it advances theoretical understanding of GFlowNets. This issue is compounded since it's work on GFlowNets, which have such strong assumptions (must not contain cycles) that it's not applicable to most problems (chess, go, atari...)

Soundness: The authors claim the improvements they show in experimental results come from their novel SubTB method, but there seems to be another key difference between their work and Malkin et al. (2022). In Malkin, they claim they facilitate exploration by choosing a policy based on a "tempered (higher temperature) version" of P_F, while the authors use a "training policy that is a mixture of P_F with a uniform policy." It becomes unclear if gains are due to a different exploration policy or SubTB.

Another point that is unclear is the following: by using the criteria from eq. 11 with the O(n^2) different subsequences, the gradient's variance is reduced (as the authors claim), allowing for a larger learning rate (0.007 vs. 0.001 in Malkin et al. (2022)). The question is the following: if something is done to reduce the variance of TB, (for example using batches from a replay buffer) so that TB can be trained with a larger learning rate, will the author's method still be better? So is it the sampling of sub-sequences that makes the difference, or the variance reduction that comes from averaging the loss from lots of sequences.


clarity,_quality,_novelty_and_reproducibility: 

The clarity of the authors work would be helped if figure 3 specified what the x axis is (number of steps, number of episodes...)

I don't find the experiments to be reproducible. Even if the authors just made a slight modification to Malkin et al. (2022), they don't discuss how they mixed P_F with a uniform policy for exploration, i.e. the constant they chose for \epsilon. Other key details, such as the general architecture of their estimator are also missing. 
summary_of_the_review: I do not recommend this paper for publication, since it's only a small modification of previous work that doesn't advance the reader's understanding of science. Plus, there's an array of open questions about why the author's method delivers the observed performance gains.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new algorithm for resolving the bias-variance tradeoff in training of GFlowNets. The algorithm is inspired from TD(\lambda) in the RL literature and proposes to train the model over partially incomplete training trajectories. Empirical results demonstrate the strength of the approach on environments used in prior literature on GFlowNets.

On the positive side, most of the reviewers appreciated the idea, the writing of the paper, and some of the empirical results. On the negative side, there were a set of overlapping concerns with the execution and the claims of the work. The work builds on a recent body work that lacks theoretical results (GFlowNets) --- which is fine in my opinion as long as there are empirical benefits --- but there is an extra onus on the authors to set a high bar for empirical rigor. This goes beyond listing hyperparameter selection protocols and should include justifications grounded in practice, such as some reviewer suggestions on doing a sensitivity analysis. A few concerns from the reviewers that could have been addressed better with empirical evidence: 
- Choosing best hyperparameters from a grid search can leak information; is there an automated procedure e.g., held-out performance? if not, is there a sensitivity analysis to justify the robustness of different hyperparameter choices? 
- One of the reviewers highlighted that the different methods do not even start from the same initial performance, which is  indeed a bit concerning. Here, the authors claim it *could be* because they do not plot the L1 at iteration 0 --- evidence of the corrected numbers would have been preferable as opposed to speculating about a potential bug. 
- Another reviewer was curious about the potential use of averaging of sequences for variance reduction (eg, using replay buffers), which is a reasonable question that merits a careful discussion/ablation to better understand the gains of the proposed method.

Overall, I encourage the authors to view some of the reviewer suggestions constructively to improve the empirical rigor of the work for a future submission.
justification_for_why_not_higher_score: Lack of rigor in empirical protocols as it relates to choosing of hyperparameters and testing of ablations suggested by the reviewers. Given that the work is primarily of an empirical nature, this seems critical for acceptance.
Decision: Reject