Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper propose GFlowNet, a generative method that can turn a given positive reward into a generative policy that samples with a probability proportional to the return. 

main_review: The most interesting part is that this novel generative method is based on flow network and local flow-conditions. And then the authors cast the flow consistency equations into an objective just like casting the Bellman equations into TD objectives. Compared with other methods, GFlowNet is designed especially for diverse candidate generation.

Overall, I vote for accepting. This current submission is overall well written, clearly illustrated and appropriately structured. The motivations are well explained and experimental results are strong.


Question:

* Looking at Figure 4, it seems that MARS has the trend to have a higher average reward if the generative models are trained more than 10^6 molecules. "For GFLowNet and for MARS, the more molecules are visited, the better they become, with slow converge towards the proxy's max reward", which method converges slower? Are there any theoretical/experimental results?

* In equation (9), the hyperparameter $\epsilon$ is added to avoid numerical issues. It would be better if the authors can provide more details on why $\epsilon$  can trade off how much pressure on large versus small flows. And add a brief description about how to setting this hyperparameter for different cases.

UPDATE:
I appreciate the authors' response to my questions and concerns.  I will keep my original score.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an approach (called GFlowNet) that uses ideas from reinforcement learning to learn a "policy" for sampling from a discrete distribution over a set of objects ("terminal states"), where these objects can be traversed between using a sequence of "actions". Specifically, the authors consider distributions proportional to some (non-negative) reward function, such that one is more likely to sample better performing points (although I believe the approach is general enough to sample from any unnormalized distribution).

In the experimental section the authors first consider a toy problem to highlight the advantages of their approach before moving on to consider the problem of designing new molecules, where the "state" is the current molecule, and the "actions" are defined using a vocabulary of fragments which one can add to the current state (as well as the stop action).  Here, the authors show GFlowNet is able to find  a set of better and more diverse molecules more quickly than previous MCMC and PPO approaches.


**More specific details about GFlowNet:**   
In GFlowNet, states and actions can be described using a flow network (i.e. a transportation network). Here the states are represented using nodes and the actions as edges (the edge weight, ie flow, describes the probability of taking an action). The authors always consider starting from the same state and actions cannot return to previously seen states (ie fragments can be added to molecules but not taken away). Moreover, one can arrive at the same state with different actions (i.e. different fragments could be used to create the same molecule). Overall, this means that the flow network can be represented using a DAG (directed acyclic graph, see Figure 1). 

The core idea of GFlowNet is then about how to learn the policy over the actions (ie the flows) such that when sampling actions from this policy, you sample from a predefined distribution over terminal states. To this end, the authors derive a flow matching objective (equation 9 -- the flow incoming to a state must match the outgoing flow) which can be differentiated through to learn the parameters parameterizing the policy. The authors prove that this loss can be used off-policy.

main_review: # 1 Summary of My Review
I found the approach presented in this paper very interesting and it appears to achieve impressive results. However, on a more negative note, some of the paper was hard to follow and the clarity could be improved in places (see main review below). I believe this would be easy enough to fix, and therefore, I have gone for a higher "overall score" and think this paper should be accepted. I have gone with a lower confidence score as I'm less knowledgable about how this idea relates to recent work in RL.
 
# 2 Main Review
## 2.1 Originality
In the molecule domain, the idea of using RL-based ideas to sample from the distribution of all molecules (weighted by their scores), rather than just find the top molecule, seems novel. Obtaining a set of diverse **and** well-performing molecules is an important problem and the proposed approach is an interesting way to do so.

Unfortunately, I am less familiar with the recent, more general RL literature in this domain so feel less qualified to discuss the paper's originality in this respect, but the approach is fairly general and could hopefully also be of interest here.

The idea of visited states being equivalent (and so the state-action space being best represented with a DAG rather than a tree) has been considered in the context of Monte Carlo Tree Search, which the authors might briefly want to mention, e.g see the discussion about "transpositions" in Section 5.2.4 of:   
> Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S. and Colton, S. (2012) ‘A Survey of Monte Carlo Tree Search Methods’, IEEE Transactions on Computational Intelligence in AI and Games, 4(1), pp. 1–43.

## 2.2 Quality
a. As far as I can tell the paper is technically sound (although I did not look through the proofs in the appendix). The approach to make the method computationally tractable (leading to Eq 9) seems reasonable although I would have been interested to hear how dependent the results presented later are on the value of $\epsilon$ chosen.

b. One other currently popular way to sample from the distribution over molecules is using autoencoders with autoregressive decoders, for instance the junction tree VAE of Jin et al. (2020) for which the fragment-based scheme in this paper is said to be based upon. This seems like an obvious baseline to compare to in the experiments but is omitted.

c. I found some of the experimental evaluation for the molecule task a little confusing: the authors invent a method for sampling from a distribution, yet evaluate it mostly in terms of how well it discovers the modes of the distribution (rather than how well it samples from the distribution as a whole). I realize that this is useful when the distribution is defined by a reward function, but what about when sampling more general distributions? (related to question 3.a. below)

## 2.3 Clarity

a. One area for improvement is perhaps the paper's clarity. For instance, section 2 is rather dense and I had to read it twice to understand what was going on. Perhaps adding more signposting to the beginning of this section would allow the reader to follow its various parts.

I also got a bit confused by the notation in this section. For instance $\tau$ suddenly appears in section 2.2 to denote the trajectory, whereas earlier on in this section I believe $\vec{a}$ was used instead (eg line 123). (Or do these variables mean different things....?)    

Likewise, I got a bit confused with Equation 10. I think the second part of this should be $L_{\theta^\ast}(\tau) = 0 \quad \forall \tau \sim P(\tau)$, i.e. with $\tau$ in place of $\theta$? (On a more minor note I was also uncertain as to why $L_{\theta^\ast}(\tau) $ had to equal zero, the approach can also deal with loss functions with any minimum right?)

b. In section 4.1 (ie the toy experiment on the grid domain) I didn't understand why the states $s$ could not live in the same space as $x$ (defining the reward)? To convert from $s$ to $x$ you had to add the vector $-\mathbf{1}$. This just seemed to add unnecessary complexity to the setup and made it harder to understand?

c. I've only briefly glanced over the code, which hopefully should aid reproducibility. One thing that could be added here though is a description of which scripts to run (including runtime arguments) to actually reproduce the experiments. 


## 2.4 Significance
This paper presents interesting theoretical results as well as strong empirical performance. In the molecule optimization task it is nice to see the focus on obtaining a _diverse set_ of top molecules rather than just the top molecule score (although it would be nice if the top-1 score was also reported somewhere for completeness). The authors consider more interesting (and difficult) molecular optimization tasks (based on docking) than much of the previous work, and it is exciting to see improvements wrt recent (and reasonable) baselines. 

I think the ideas that GFlowNet is built around are interesting, particularly of how to sample from a distribution of discrete objects when the transition function (i.e. "action" using the paper's terminology) is irreversible. Hopefully this will inspire further work in this direction.


# 3 Clarifying Questions

a. My understanding is that (if trained/run for long enough) the MCMC baseline and GFlowNet should ultimately sample from the same distribution, i.e. that defined by:
$\pi(x) \propto R(x)$. However, in figure 3 it appears that GFlowNet's distribution is different (the mode is higher and the distribution has wider tails). Does this just mean the MCMC method has not converged? Is there any way to compare against the "true" distribution (for instance running the MCMC method for much longer, or using very many random trajectories and importance sampling)?

b I am currently having a bit of trouble understanding figure 4. On lines 330-336 the authors state that when sampling 300k molecules using random trajectories that they obtain over 100 molecules with scores above 8 (233 such molecules to be precise). Looking at figure 4 the top-100 average for MARS is below 8 after visiting 300k molecules. Does this mean that it is worse than random search? Is this expected?  

c For the MCMC methods (e.g. MARS) did the authors consider thinning?

d A related question. For GFlowNet I assume every state is valid? If so I was wondering if the authors ever considered counting the reward for every state visited by GFlowNet rather than just the terminal ones. In some ways can one think of the "stop action" as a learnable thinning function?

e I apologize if I missed this but what is the initial state for the molecule tasks?


# 4 Very Minor Comments

i. Lines 105, 138 and elsewhere on page 3 the word "surjective" is used. I think "non-injective" is meant here instead...? 

ii. I found the figure on the right below line 265 helpful for understanding the toy problem (so thank you to the authors for this). However, its axis is too small and unreadable -- this should be made larger. Similarly, Figure 7 in the appendix could be bigger (especially given that there are no space constraints in that section).

iii. There are some weird black horizontal-ish lines on Figure 3 (at least on the two pdf viewers that I tried). Did these mean anything or were they just some left over plotting artifacts?

limitations_and_societal_impact: # 5 Limitations

a. The authors briefly talk about the limitations of the approach in section 5. The main limitation they draw attention to is the challenge of moving closer to the local maxima of the reward function in the latter stages of optimization. To resolve this they discuss combining their method with local optimization techniques; however, I wonder whether the temperature approach they discuss in the earlier part of their paper (combined with some annealing scheme) could also be used here?

b. One limitation the authors do not mention is how the method scales in terms of the size of the state and action space. The loss function requires for every current state the sum over all previous states and actions that may have led to the current state (see term 1 of Eq.9). I assume this may become intractable for very large state-action spaces (and the flows one is trying to model become very small). Can one approximate the sum using a subset? Also what about continuous state/action spaces?




# 6 Societal impact
The authors state that they foresee no negative social impacts of their work (line 379). While I do not believe this work has the potential for significant negative social impact (and I'm not quite sure if/how I'm meant to review this aspect of their work), the authors could always mention the social impact of increased automation, or the risks from the dual use of their method, etc.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tries to tackle a scenario where the goal is not to learn to generate a sequence of actions that has the highest return but rather to generate a diverse set of trajectories with high returns. They do so by using the given positive return function (which is given by the oracle function) to formulate a generative policy that samples with a probability proportional to the return.  They used a flow network to express the MDP and its value function where the value of a state and action pairs is defined by the flow between the two nodes in the flow network, and finally, the flow consistency equations are used as a learning objective.
main_review: The idea of the paper is very interesting but the paper is hard to follow as the paper borrows notations and concepts from RL but not in the exact same setting so it makes it a bit confusing for the readers. I personally feel like the paper could be made more intuitive and much easy to follow if it uses more direct and intuitive concepts and pushes less hard to connect with concepts in RL. 

There are some minor comments and questions:
Comments:
Line 71: The RL “agent” should have large support of what?
Line 84: Missing citations of PPO and MCMC
Figure 3: MARS looks to approximate better the proxy dataset distribution.
Figure3: Didn’t understand the shifting of the density to the right.


limitations_and_societal_impact: The paper is a bit hard to follow.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper considers generative processes for which state transitions form a DAG. The authors view such a generative process as a flow in a graph—with a single source and sinks connected to final objects. The authors set objects' probabilities being proportional to the reward function and train a generative model by enforcing flow-matching conditions. The main contribution of the paper is a new generative approach for unnormalized distributions.
main_review: The proposed approach seems novel and is well described in the paper. The authors provide theoretical and practical details, as well as intuition behind the method. Experimental evaluation is sufficient, and results are demonstrated on a toy problem as well as on a practical task of molecular property optimization. Provided results in supplementary (Fig. 13 and Fig. 14) demonstrate huge advantage of the proposed approach.

Some comments are provided below:
1. More details on relation of the proposed approach and Bayesian networks would be profitable. Currently it seems as if model definition using a Bayesian network would yield a similar approach (e.g., flow-matching condition will be the same as total probability formulta).
2. Experimental results should be provided on standard datasets. For example, optimization of logP and QED as provided in multiple papers, including GrammarVAE (https://arxiv.org/pdf/1703.01925.pdf). Such experiments are necessary to confirm that the proposed approach indeed yields good results against strong baselines.
3. Diverse generation as mentioned in the title of the paper, is not built in the model and is a side-effect of the approach. I would recommend conducting experiments on the objective functions with high mode imbalance (e.g., 10k molecules with high reward in one mode and 5 molecules with high reward in another).

Update: The authors answered my questions, I keep my original score.
limitations_and_societal_impact: The authors adequately addressed the limitations.
review_title: null

Meta-review: After an extensive back-and-forth discussion, all reviewers felt positively about the paper and its contribution, and lean towards acceptance.

However, there were issues regarding clarity in the submitted draft — this has been largely addressed through clarifications from the authors, but it is essential that the discussion here is incorporated into the final version of the manuscript (and in particular, the responses to reviewer 1AuD).
Decision: Accept (Poster)