Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to incorporate intrinsic reward augmentation in GFlowNets to improve exploration in sparse reward settings. It extends the intrinsic rewards to both state and edge intrinsic rewards improving on convergence and mode discovery in GflowNets in Gridworld experiments and more complex molecule generation tasks.

strength_and_weaknesses: Strengths: this paper is clear and well written, it provides a self contained and comprehensive literature review, it motivates the problem well and provides compelling results.
Weaknesses:
Can the authors provide citations/derivation for eq 4,5 and 6 in particular the telescoping derivation?
The experiments mention a coefficient parameter alpha that is never mentioned in the theoretical presentation of the model? Where is this used?
How important is the choice of intrinsic motivation function, is it enough to recover more structured forms of exploration? How far ahead can the flow information capture relevant future rewards? 
What are the main limitations of the GAFlowNets in exploration, can the authors frame the proposed approach in terms of where would this kind of exploration shine vs where would it fail?

clarity,_quality,_novelty_and_reproducibility: The paper provides a joint form of dealing with state and edge based GFlowNet using RND base intrinsic rewards which is novel to the best of my knowledge. The paper is clear and justifies experimentally the experimental claims, the exact network choices could be further explained to improve reproducibility of results. Will the code be made publicly available?


summary_of_the_review: The paper tackles a problem of lack of intrinsic rewards in GFlowNets and provides a joint edge and state based flow intrinsic reward that captures well different modes and converges faster.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces  Generative Augmented Flow Networks (GAFlowNets) that incorporate intermediate rewards into GFlowNets. Original GFlowNets  learn from the reward of the terminal state, and do not consider intermediate rewards. That is a problem in environments with sparse rewards. For this problem taking intermediate feedback signals into account helps to provide an exploration. (Bengio et al., 2021b) already proposed to incorporate intermediate rewards in a state- based manner. This approach  can result in slower convergence and large bias empirically, although it can explore more broadly. Incorporating the edge-based augmentation for efficiency, this paper proposes a joint way to take both edge-based and state-based augmented flows into account. Experiments on the GridWorld and molecule domains show the effectiveness of the proposed framework. 

strength_and_weaknesses: The strength of this paper is a novel learning framework to incorporate intermediate rewards into GFlowNets. Incorporating intermediate rewards into GFlowNets has been already proposed. This paper formulate the problem by clarifying existing unsolved  problems and proposes a novel approach.
The algorithm is described in details. Enough experiments with further ablation study.
This is not weakness, but more evaluations of differences of learned structures between GFlowNets and GAFlowNets would support claims more clearly
clarity,_quality,_novelty_and_reproducibility: The objective, the design of the  proposed framework, the algorithm are clearly described. Enough experiments are done to support claims with enough  evaluations. Conditions of experiments are described in details that is.helpful for its reproducibility.
summary_of_the_review: This paper formulate the problem by clarifying existing unsolved  problems of GFlowNets and proposes a novel approach. This paper proposes a joint way to take both edge-based and state-based augmented flows into account.The paper is enough described and organized to support claims with good experimental results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper argues that intermediate rewards are also crucial in the learning of Generative Flow Network, and proposes a novel GFlowNet learning framework, dubbed GAFlowNet, to incorporate intermediate rewards. The paper specifies intermediate rewards by intrinsic motivation to deal with the exploration of state space for GFlowNets in sparse reward tasks. The experiment results on the GridWorld domain and the larger-scale and more challenging molecule generation task show the effectiveness of the method in terms of convergence, diversity, and performance.
strength_and_weaknesses: Strength:

(1)The authors focus on the intermediate rewards in the learning of Generative Flow Network, and propose a novel GFlowNet learning framework, dubbed GAFlowNet, to incorporate intermediate rewards.

 (2) The paper proposes three different kinds of augmentation methods, namely, edge-based intermediate reward augmentation, state-based intermediate reward augmentation, and joint intermediate reward augmentation. And, the advantages and disadvantages of different augmentation methods are analyzed intuitively and clearly.

 (3) In general, the experiments are comprehensive and reasonable.

 (4) In general, the paper is well-organized and easy to follow.

Weakness:

(1)  The author just considers combining intermediate rewards to further improve the performance of GFlowNet in particularly challenging sparse reward tasks, which is not novel enough.

(2) More novelty degree measure methods can be considered to further refine the experiments.

Minor errors:
(1) In Figure.5, the first and second rows correspond to the number of discovered modes and empirical L1 error respectively, which doesn’t correspond to the descriptions in the caption of Figure.5 and the analysis of the experiment in 5.1.1.

(2) The position of the figures can be adjusted to make the paper read more fluently, for example, Figure.4 has been mentioned earlier in the paper.

clarity,_quality,_novelty_and_reproducibility: The novelty is limited.
summary_of_the_review: Overall, this paper proposes an interesting method. But, why intermediate rewards play a critical role in RL  is not clear.  It is unclear that why the improved Generative Augmented Flow Networks (GAFlowNets) can work better than the previous Generative Flow Network. 


The rebuttal is good, and I thus increase the score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an extension of Generative Flow Networks (GEFlowNets). The main idea is to incorporate intermediate rewards as well as terminal rewards into the objective, which was missing in GEFlowNets. The paper proposes to combine two variants: edge-based augmentation and state-based augmentation and shows that adding intrinsic rewards leads to the same solution when intrinsic rewards converge to 0 over time. The empirical results on a grid world and a molecule generation task show that the proposed method significantly outperforms GEFlowNets and other RL baselines including PPO when combined with RND-like intrinsic rewards. Besides, it discovers more diverse set of solutions compared to the previous approaches thanks to the additional intrinsic rewards. 
strength_and_weaknesses: [Strength]

* The proposed idea of incorporating intermediate rewards to GEFlowNets is novel.
* The result convincingly demonstrates that the proposed method significantly outperforms the previous approach (GEFlowNets) both in terms of performance and the diversity of solutions.
* The paper is well-written.

[Weaknesses]

* **Lack of result on standard RL benchmark:** To my understanding, the proposed method is general enough to be applied to any RL problems as it can incorporate intermediate rewards, and the introduction was also written in such a way. However, all of the experiments were conducted on the "reward-at-termination" types of environments, while intermediate rewards were given by intrinsic rewards (not from the environment). It would be much more comprehensive to evaluate the proposed method on a standard RL benchmark (e.g., Atari) that contain "intermediate extrinsic" rewards. 

* **Lack of clarity about intermediate rewards / intrinsic rewards:** Related to the above point, I felt that the paper often assumes that intermediate rewards = intrinsic rewards without a clear distinction, which I found confusing. In page 5, for example, it says "our edge-based reward augmentation could motivate the agent to discover other paths ...". Does the diversity of the solution come from the incorporation of intermediate rewards (i.e., the edge-based augmentation) or the combination of the edge-based augmentation and a particular intrinsic rewards designed for exploration? I believe that the paper meant the latter. I would be good to make this distinction clear throughout the paper.

* **Lack of clarity about edge-based augmentation vs state-based augmentation:** I found it difficult to understand the different behaviors between the edge-based augmentation and the state-based augmentation. Why does the edge-based augmentation tend to explore "locally" while the state-based augmentation explores more "globally"? I could not figure it out from the equations immediately. Further clarifying this point would be helpful for justifying the joint approach.
clarity,_quality,_novelty_and_reproducibility: 
* Clarity: In general, the paper is well-written except a few points I raised above (see weaknesses). 

* Quality: The paper provides compelling experiments against the related prior approach (GEFlowNets). The comparison to standard RL algorithms (e.g., PPO) could be a bit misleading without presenting results on standard RL benchmarks. 

* Novelty: Although the overall method builds upon GEFlowNets, the incorporation of intermediate rewards is novel.

* Reproducibility: The paper provided some details in the appendix and proposed to release the code upon publication. 
summary_of_the_review: GEFlowNets are an interesting alternative to existing RL algorithms, which has been receiving attention. This paper extends GEFlowNets nicely to support intermediate rewards, which sounds like an exciting direction towards solving RL problems with GEFlowNets. Although the results are limited to the intrinsic rewards on specific types of problems, the comparison to the prior work GEFlowNets looks quite promising and convincing. I am willing to increase the score if the authors reflect (or provide a reasonable response to) some of my comments above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed Generative Augmented FlowNets, which are a variation of GFlowNets that can incorporate intermediate rewards to aid intrinsic exploration. Specifically, the paper combines edge-based augmentation with state-based augmentation proposed in prior work to improve convergence and mode discovery. The empirical results on grid world and molecule generation are very promising, and outperform several baselines while discovering a diverse set of solutions. 
The reviewers were impressed with novelty of the solution and the strong results. There were some concerns around clarity of descriptions of intrinsic rewards and the augmentations but the author rebuttal seems to address those. I encourage the authors to make these revisions to strengthen their paper.
justification_for_why_not_higher_score: The technical ideas are relevant to a specific community within the field, who will need to dive deep into the paper to understand the details.
justification_for_why_not_lower_score: This is a neat paper that will benefit from a spotlight so that more people in the community will gain awareness of GFlowNets and related ideas.
Decision: Accept: notable-top-25%