Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: GFlowNets learn stochastic policies to sample discrete objects proportionally to a non-negative reward. The paper proposes path regularization to improve exploration and generalization in GFlowNets. The authors begin by defining a notion of directed distance within the GFlowNet DAG, followed by the transport cost between forward policies of neighboring states. The authors then establish an upper-bound on this transport cost and discuss efficient implementation of the transport cost. The authors discuss two particular instantiations of the path regularization. Specifically, the authors claim that minimizing the transport cost results in faster adaptation to high reward regions, whereas maximizing the transport cost leads to improved coverage of the state space. This is accompanied by empirical results on various tasks studied in prior work on GFlowNets. 
strength_and_weaknesses: **Strengths**

- The proposed regularization scheme is well formulated and described in sufficient detail. I think the optimal transport perspective fits training of GFlowNets quite well, and is an interesting direction to be explored. 

**Weaknesses**

- While I believe the optimal control perspective is interesting, the particular instantiation proposed in the paper is not too clear and well motivated. In Section 3.1 the authors discuss the cases of maximizing and minimizing the path regularization. For motivating the minimization of the regularization term the authors argue that making the forward policy at neighboring states closer results in better generalization - but this point is not super clear to me (and also doesn't seem well supported by the experiments). In general choices in the method are clearly described but quite poorly motivated. 
- In Section 3.3 the authors make the observation "any two neighbor states $s < s′$ do not have the same child state". I do not believe this observations holds true in the DAGs studied in the paper. For example, in the EB-GFN setup, the permutations of actions in a subtrajectory result in the same state. 
- The improvement due to the regularization seems marginal at best, based on the empirical results reported in the paper. Another thing missing in the experiments is careful analysis of the effect of the regularization, i.e. how does varying $\lambda$ affect the performance. It is also not clear to me why the Max-OT is missing in the EB-GFN experiments? (Minor - in Table 3 specifically the authors report results with a precision on 10^-5 but as the oracle used in the task is a neural network that precision seems insignificant)
clarity,_quality,_novelty_and_reproducibility: **Clarity**

While most of the details are described clearly, as discussed in the previous section, a lot of the choices are not motivated very clearly. There are also quite a few grammatical errors some of them listed below
- GFLowNets, Gflownets, GFlownets - in several places throughout the paper should all be GFlowNets (as stated in the abstract)
- Page 2, last paragraph line 1 - "we provide backgrounds" -> "we provide background "
- Page 5, below caption, "Remind that" -> "Recall that"

**Quality and Novelty**

The regularization scheme proposed in the paper is novel and explores an interesting direction to improve training of GFlowNets. However, the marginal empirical improvement limits the significance of the contribution. 

**Reproducibility**

The authors have included code to reproduce their results and present sufficient details in the paper. The proofs are for the most part accompanied by appropriate assumptions and details. 
summary_of_the_review: In summary, the paper explores an interesting direction of optimal transport to improve GFlowNet training. Some key details of the method are not well motivated and some observations seem to be incorrect. The empirical improvements are also marginal. I am thus leaning towards rejection but I encourage the authors to address the feedback during the discussion and rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a novel path regularizer for Generative Flow Networks (GFlowNets, Bengio et al., 2021), based on concepts from Optimal Transport (OT). This regularizer, defined in terms of directed distances in the GFlowNet, is in general expensive to compute. Fortunately, the authors show that this regularizer can be upper-bounded by a term more convenient for optimization. Moreover, under some conditions on the structure of the GFlowNet that are met in practice for the applications considered in prior works, the authors show that this regularizer can be OT regularizer can even be computed in closed form. This regularizer was tested in conjunction with the GFlowNet on a synthetic environment, discrete probabilistic modeling, and biological sequence design, and shows advantages in terms of performance and diversity compared to the baseline model without the regularizer.

---

*Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations.*
strength_and_weaknesses: **Strengths**: This regularizer is a well-founded way to improve diversity of samples from a GFlowNet. The upper-bound provides an effective strategy to add path regularization to GFlowNets. The authors evaluated this regularizer thoroughly on multiple environments over multiple metrics of interest in previous works.

**Weaknesses**:
 1. Since the core of the paper is about the added benefit of the regularizer over standard GFlowNets, I would have appreciated a more thorough study of the effect of the regularization constant $\lambda$ on the performance and diversity. What trade-offs should one expect from setting the regularization to some value? Currently, the regularization constants (provided in Appendix E) are task-specific, which is expected, but there is no mention how those regularization constants were chosen: did you choose them based on a validation set? What was the metric you optimized for?
 2. The definition of the cost function $C_{ij} = \min_{\tau} (-\log P(\tau\mid u_{i}))$ in Equation 9 seems incorrect (except for $u_{i} \equiv v_{j}$): for example you consider that if $u_{i} \not\rightarrow v_{j}$, then $C_{ij} = -\log (P_{B}(s\mid u_{i})P_{F}(s'\mid s)P_{F}(v_{j}\mid s'))$. However, we can imagine having a trajectory $\tau = u_{i} \rightarrow s \rightarrow s'' \rightarrow s' \rightarrow v_{j}$ that has a smaller $-\log P(\tau\mid u_{i})$; intuitively, it might be more unlikely to move from $s$ to $s'$ via an intermediate step $s''$, rather than directly.
 3. Related to 2., the proof of Theorem 3.2 might also be erroneous, for similar reasons. The proof in Appendix D.2 starts with $C_{ij} = -\log (P_{B}(s\mid u_{i})P_{F}(s'\mid s)P_{F}(v_{j}\mid s'))$, even though there is no assumption in Theorem 3.2 preventing us from having a situation as above with $\tau = u_{i} \rightarrow s \rightarrow s'' \rightarrow s' \rightarrow v_{j}$. All we can say, and that's what you properly used in the proof of Theorem 3.1 for the upper bound, is that $C_{ij} \leq -\log (P_{B}(s\mid u_{i})P_{F}(s'\mid s)P_{F}(v_{j}\mid s'))$.
 4. There are some confusions regarding the notions of entropy and cross-entropy. The formula in Equation 19 only corresponds to the entropy of $P(\tau)$ if $\pi_{\theta} = P$. Moreover since $P_{B}^{\star}$ is not a probability distribution, the cross-entropy $H(P_{F}(\cdot\mid s_{t}), P_{B}^{\star}(\cdot\mid s_{t}))$ is not properly defined. One can understand what this means from the Proof in Appendix D.1, but you should be precise in your use of these terms, especially since you are drawing conclusions about the behavior of the regularizer from maximizing entropies (Section 3.1) and minimizing cross-entropies (Section 3.2).
 5. The behavior of Trajectory Balance with and without OT regularization seem to be very similar on the synthetic task (Figure 2). It would be informative to add shaded areas to show the variance for the 10 runs, and evaluate if the effect of the OT regularization is statistically significant on this environment.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well written, and easy to follow. It provides all the necessary background on both GFlowNets and Optimal Transport to fully understand the paper. The work is novel.

There are some minor typos, that do not impact the clarity. I have noted a few:
 - Section 2.1, paragraph 2: "moving" -> "removing"
 - Definition 3.1: "shortage" -> "shortest"
 - Page 5 (paragraph below Fig. 1): "form" -> "from"
 - Page 5 (same paragraph): "designed" -> 'design"

The authors also provided the source code for their experiment as part of the Supplementary material, and provide all the details about the hyperparameters they chose in the Appendix. Unfortunately due to the lack of time for the review, I did not check the source code in details, nor some of the proofs in details in the Appendix.
summary_of_the_review: This paper provides a practical solution to improve GFlowNets, and is completely novel. I am currently leaning towards acceptance, but there are some technical aspects raised here that may have significant effects on the results presented in this paper (notably Theorem 3.2). Therefore I can only recommend borderline acceptance for now.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a way to regularize GFlowNet policies using an optimal transport cost that can encourage either similarity or dissimilarity of policies at states that are close in the state space. This cost is converted into several proxy regularization terms that can be added to the trajectory balance objective in on-policy training of GFlowNets. Experiments are performed on three domains from recent work.
strength_and_weaknesses: Comments on experiments:
- (++) Breadth of experiments: there is hypergrid, sequence design, energy-based modeling -- which all use different codebases in the original implementations from the papers that introduced them. I appreciate the engineering work and the variety of applications considered (simple reward matching, active learning, energy-based modeling).
- (--) Many of the results are unconvincing. 
  - For the hypergrid environment, the difference between algorithms is almost invisible, and it could be compensated by small changes in learning rate. I would suggest trying  larger values of H and D, where sparsity may become more important, and showing more clearly the dependence on the most important hyperparameters (regularization weight -- how was it chosen?).
  - In the EB-GFN experiments, there is practically no difference between the proposed algorithm and baselines. Furthermore, the values highlighted in Table 4 are sometimes not actually the best one in their columns, which is misleading.
- (+/-) It is interesting that the Max OT approach performs best in the active learning settings. What is the interpretation of this result? Are more entropic policies beneficial in the AL setting? This is a potential strong point that should be investigated further.
- (-) In practice, what is the computational overhead of UB and Min/Max OT?
  - There are simpler (and less computationally expensive) ways to regularize a GFlowNet to either encourage or discourage sparsity. The simplest is just to penalize the entropy of the backward policy. It would be interesting to see how these compare to the OT-motivated regularization proposed here.

Major comments on writing and theory:
- (++) The motivation for regularizing the GFlowNet policy makes sense and, as far as I can tell, the math is right.
- (-) Large parts of the background on GFlowNets, especially Appendix B, closely follow or even copy [Malkin et al., 2022], abbreviated [TB], in the flow of exposition and in the notation. For example, the first sentence of "Learning GFlowNets" (p.14) is identical to the first sentence of 2.2 in [TB], and all choices of notation are the same. Borrowing the structure and notation is fine, but it should be attributed, with a sentence such as "this exposition closely follows that of ...".
  - In addition, in the section "Learning GFlowNets" on p.3, the proof of correctness of TB is misattributed (in fact, it appears in [TB], not [Bengio et al.]). 
- (-) In Definition 3.1, the distance between states is only defined for states that are ancestors of one another. (By the way, in that definition, one should not call a sequence of actions a "transition".) However, on p.4, it is generalized to distances between any pair of states. 
  - This generalization should be explicitly stated, and it should also be stated in what sense the resulting d(s,s') is a "distance". For example, it is not symmetric, and it is possible that the distance between two distinct states is 0 if the transition probability between them is 1. Thus all we can say about d is that it is a pseudoquasimetric (since it is nonnegative, d(s,s)=0, and it satisfies the triangle inequality).
  - The cost matrix in equation (9) is only an approximation to the true distance, since the true distance may be the NLL of a trajectory with more than three edges and may not just go "back" and then "forth". This should also be stated.

Minor comments:
- This sentence on p.2 does not make sense to me: "Compared with KL divergence, the biggest problem of KL divergence is that it was infinite for a variety of distributions with unequal support". Does this mean that KL divergence cannot be used as the distance metric because KL(p||q) is infinite if (support of p)-(support of q) has nonzero measure under p?
- The "pseudo backward policy" requires some more details. It is not actually a conditional distribution (may not sum to 1). So what exactly is meant by the cross-entropy in (17)?
- In (19), $\pi_\theta$ was not defined. (Is it the untempered forward policy?)
- The paper would benefit from some proofreading for small writing bugs, which nonetheless do not interfere with understanding of the paper. Here are a few:
  - Inconsistent capitalization and grammatical number in the name of the algorithm throughout the text, such as  "GFlownet", "GflowNet", "GFLowNet" instead of "GFlowNet". GFlowNet is singular and GFlowNets are plural.
  - Similar comment about "trajectory balance objective" (sometimes called "trajectory balanced objective" and inconsistently capitalized)
  - "natural interpreted" -> "naturally interpreted" (p.2)
  - "shortage path" -> "shortest path" (several places)
  - "can be define" -> "can be defined" (p.4)
  - Some equations are not in math mode (p.6)
  - Incomplete citations:
    - [E. Bengio et al., 2021] was published in NeurIPS 2021.
    - [Dai et al., 2020] was published in NeurIPS 2020.
    - [Deleu et al., 2022] was published in UAI 2022.
    - [Grathwohl et al., 2021] was published in ICML 2021.
    - [Malkin et al., 2022] will be published in NeurIPS 2022.
clarity,_quality,_novelty_and_reproducibility: Quality and clarity: The exposition is good, but could give better attribution to past work (see above). Experiments are unconvincing and lack interpretation.

Originality: Good. This is the first attempt to answer the question of which solution to the GFlowNet policy optimization problem -- which generally has a high-dimensional family of global minima -- should be chosen. 

Reproducibility: Good. Code is provided, based on code from past work (though I did not run it myself).
summary_of_the_review: There is a potential for a good contribution here: optimal transport is an interesting way to answer the question mentioned in "Originality" above. However, I vote to reject because the weaknesses listed above are dominating: in particular, the lack of (1) experiments that give insight about hypothesized benefits of the prosed regularization and (2) consideration of simpler algorithms, like entropy regularization on $P_F$ or $P_B$, that may have a similar effect.

Post-rebuttal update: After reading all reviews and responses, I increase the recommendation from 3 to 5; see comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes three different ways to regularize GFlowNet training with Wasserstein distance. Implementations of the regularization term and an upper bound of it are derived for tractable training. The path regularization is claimed to be capable of generating more diverse candidates via maximizing the Wasserstein distance or improve the generalization by minimizing the Wasserstein distance. The proposed methods are evaluated on a series of GFlowNet-related tasks, including hypergrid, biological sequence design and synthetic probabilistic modeling. 
strength_and_weaknesses: ## Strengths
The proposal of using optimal transport theory to regularize GFlowNet training is novel and interesting. Three different regularization terms are proposed accordingly.
1. The authors propose the definition of directed distance given the "back-and-forth trajectory" idea in EB-GFN paper.
2. Tractable calculation methods are derived with theoretical insights to achieve the proposed algorithms.
3. A series of experiments are conduct to verify the advantage of the path regularization method.

## Weaknesses
1. In Sec. 1, it is written that the motivation of minimizing path regularization is to enhance flow on high flow trajectories. This is true, but what I do not understand is the connection with generalization. First, GFlowNet is a way to amortize the MCMC sampling problem, and since there is no change of the target distribution between the training and test time, I do not think "generalization" is a proper notion to discuss the performance of GFlowNet. Second, such motivation is contrary to the one of GFlowNet, who aims to sample *proportional* to a given reward function, rather than finding the high-score objects. Such difference is the core motivation for why people need GFlowNets instead of directly using reinforcement learning or other related methods.

2. In Sec. 1 and Sec. 3.1, the authors also say maximizing the path regularization term gives more diverse effects. I agree with this point, but since minimizing and maximizing the path regularization term gives different regularization effects, I wonder which effect is needed by GFlowNet? It seems the authors only simply testify the two methods (Min OT & Max OT) without further analysis. Maybe different tasks would require different behaviors (and thus need different regularization effects)? For example, in the toy hypergrid experiment, if we make the reward function to be more sparse, would the Max OT method show more benefit? I think it would be great if the author could provide more analysis on this point.

3. Ablation about the hyperparameter $\lambda$, namely the coefficient of the regularization term, is missing. It would be great to have a plot / table of the performance of different methods with different values of $\lambda$ (e.g. both positive and negative values) on hypergrid (since it is the easiest task, but it would be better to have ablation results on other tasks).

4. The forward and backward policy are in symmetric positions, which says, either of them is able to determine the whole GFlowNet distribution. If the method proposed in this work is effective, then one thing worth trying is to put a similar technique to regularize the distance of backward policy on intermediate states.

5. The result of "Max OT" is missing in the probability modeling tasks. Is there any particular reason?

### Minors
1. In Sec. 4.3, there is a gap between the MMD values in EB-GFN paper and this paper. I feel this is about the number of samples used to calculate the MMD. Theoretically, MMD value should always be positive. However, it could be negative if estimated with a small number of samples, for example in original ALOE paper they use a fixed 4000 samples. In EB-GFN paper, the MMD is calculated with 10 times average, each time with a different set of 4000 samples (not fixed samples), and the results are positive.  It is OK to compare with a consistent metric like in the current paper, but it would be great if the numbers of this part could be re-calculated for better reliability. 

2. One of the advantage of Wasserstein distance is that it does not require the distributions to have the same support, as mentioned in Sec. 1. In molecule tasks, this unequal support would really matter as the action space is more structured in graph generation. In this sense, the molecule task should be a better playground for the proposed method (otherwise, why not directly use KL or other simple divergences? They are much easier to compute than OT). 

3. There are a couple of typos and misuses of notations. 
clarity,_quality,_novelty_and_reproducibility: ### Clarity & Quality
This paper is OK w.r.t. readability. But I think it would be better to move the content of Sec. 3.3 to the end of Sec. 3.1 to give a clear demonstration of the specific algorithm. 

### Novelty
The idea of this paper is novel.

### Reproducibility 
The corresponding code implementation is provided.
summary_of_the_review: This work is with reasonable writing and experiments. However, I have some concerns about the motivation, and also feel more experiments could be conduct to amplify the story of this paper. I would definitely consider raising the score if some of the main weaknesses could be reasonably addressed :)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary**

The paper proposes a OT regularizer for the  GFLow Net cost to promote diversity and generalization.
 In order to encourage diversity authors propose path regularization  maximizing the OT distance between two forward policies of two neighboring states. The cost in the optimal transport used is intuitively the shortest path between the states. for generalization authors propose to minimize this distance. 

An upper bound is provided to the OT cost that is otherwise expensive to compute on the path for all states, and a closed form can be derived in a specific setup. 

Reviewers discussed the paper with the authors  and thought that the strength of the work is in its use of OT to regularize GFlow nets that is novel  however the motivation linking the OT distance to the   generalization and diversity is lacking and there is   weak empirical evidence for the stated claims. 


justification_for_why_not_higher_score: The paper lacks theoretical motivation and convincing experimentation that supports the claims
Decision: Reject