Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets

Authors: Haoran He, Can Chang, Huazhe Xu, Ling Pan
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Backward Sampling, Goal-Conditioned Learning, Generative Flow Networks

Abstract:
Generative Flow Networks (GFlowNets), a new family of probabilistic samplers, have demonstrated remarkable capabilities to generate diverse sets of high-reward candidates, in contrast to standard return maximization approaches (e.g., reinforcement learning) which often converge to a single optimal solution. Recent works have focused on developing goal-conditioned GFlowNets, which aim to train a single GFlowNet capable of achieving different outcomes as the task specifies. However, training such models is challenging due to extremely sparse rewards, particularly in high-dimensional problems. Moreover, previous methods suffer from the limited coverage of explored trajectories during training, which presents more pronounced challenges when only offline data is available. In this work, we propose a novel method called \textbf{R}etrospective \textbf{B}ackward \textbf{S}ynthesis (\textbf{RBS}) to address these critical problems. Specifically, RBS synthesizes new backward trajectories in goal-conditioned GFlowNets to enrich training trajectories with enhanced quality and diversity, thereby introducing copious learnable signals for effectively tackling the sparse reward problem. Extensive empirical results show that our method improves sample efficiency by a large margin and outperforms strong baselines on various standard evaluation benchmarks. Our codes are available at \url{https://github.com/tinnerhrhe/Goal-Conditioned-GFN}.

Decision: accept (poster)
Reviewer scores: 8, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730459448634

thread_type:
official_review

<thread_object>
creator: reviewer_iekk

<content>
summary:
The paper tackles the challenge of training goal-conditioned Generative Flow Networks (GC-GFlowNets) in environments with sparse rewards and limited offline data. It introduces Retrospective Backward Synthesis (RBS), which synthesizes new backward trajectories to enrich training data, improving sample efficiency and diversity. Experiments demonstrate that RBS significantly improves performance and generalization in various benchmarks.

strengths:
This paper proposes a novel method called Retrospective Backward Synthesis (RBS), which synthesizes new backward trajectories in goal-conditioned GFlowNets to improve the quality and diversity of training trajectories. This approach introduces rich learnable signals, effectively addressing the sparse reward problem.

weaknesses:
1.The experimental tasks are relatively simple and insufficiently comprehensive.

2.The latest goal-conditioned reinforcement learning algorithms are not selected for comparison.

questions:
1.	Age-Based Sampling is a very straightforward technique. How does it compare to previous methods like Prioritized Experience Replay (PER)? Did the authors attempt using PER as well?

2.	Regarding the experimental setup, since you’ve compared your method with reinforcement learning approaches, I assume that these experiments share the same tasks as those in reinforcement learning. If that’s the case, why weren’t newer RL methods selected as baselines? Additionally, for tasks like bit sequence generation, TF binding generation, and AMP generation, is DQN an appropriate baseline?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730539580074

thread_type:
official_review

<thread_object>
creator: reviewer_xjji

<content>
summary:
This paper addresses key challenges in goal-conditioned Generative Flow Networks (GFlowNets), specifically the problems of sparse rewards and limited trajectory coverage. The authors introduce Retrospective Backward Synthesis (RBS), a method that generates additional backward trajectories to expand the training data. Their approach aims to improve both the quality and diversity of training trajectories, providing more learning signals in scenarios with sparse rewards. Empirical evaluations demonstrate improved sample efficiency and performance compared to baseline methods across multiple benchmarks.

strengths:
* The paper is well-written and straightforward to understand.
* Retrospective Backward Synthesis (RBS) is introduced with clear motivation, and the paper also presents training techniques such as backward policy regularization. 
* Empirical results demonstrate that the proposed method outperforms baselines, showing improved performance and sample efficiency.

weaknesses:
* The evaluation tasks do not include key benchmarks like RNA Generation from Pan et al. (2023a), which limits direct comparison.
* The differences between the proposed RBS method and OC-GAFN are not clearly articulated. A more comprehensive discussion is needed to clarify the specific advantages of the RBS method. 
* It remains unclear how goals are defined across the evaluated tasks, which could impact generalizability and reproducibility.

questions:
Please address my concerns in the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730592757520

thread_type:
official_review

<thread_object>
creator: reviewer_hych

<content>
summary:
The paper proposes Retrospective Backward Synthesis (RBS), a novel method to enhance the training of goal-conditioned Generative Flow Networks (GC-GFlowNets). GC-GFlowNets have shown potential in generating diverse sets of high-reward candidates but face challenges due to sparse reward structures and limited coverage of explored trajectories, especially when using offline data. To address these limitations, RBS synthesizes backward trajectories that originate from a desired goal, enriching the training data with high-quality, diverse samples. This approach helps transform unsuccessful action sequences into positive learning experiences, thereby improving sample efficiency and generalizability.

The authors introduce additional techniques, such as reward signal intensification and backward policy regularization, to stabilize training and prevent mode collapse. Empirical results across various benchmarks, including GridWorld and bit sequence generation, demonstrate that RBS outperforms state-of-the-art methods in terms of success rates, sample efficiency, and scalability. Notably, RBS achieves nearly 100% success in large-scale tasks where competing approaches fail, highlighting its robustness and potential for further advancements in GC-GFlowNets.

strengths:
- Backward-Looking Strategy for Enhanced Training: The proposed Retrospective Backward Synthesis (RBS) method utilizes a backward-looking strategy to synthesize trajectories from the goal state, significantly enriching training data. This approach effectively improves sample efficiency by converting failed experiences into successful learning signals, addressing the sparse reward problem.
- Empirical Validation of Sample Efficiency: The paper presents strong empirical results across a range of benchmarks, demonstrating that RBS markedly improves sample efficiency. The method achieves nearly 100% success rates in complex tasks where state-of-the-art baselines fall short, underscoring its practical impact.
- Clear Writing and Presentation: The paper is well-written and presented, with clear explanations, structured methodology, and comprehensive experimental results. The clarity facilitates a strong understanding of both the theoretical and practical aspects of the proposed approach.

weaknesses:
- Scalability and Continuous Environments: The paper’s experiments focus on relatively simple and discrete environments, raising concerns about how well the Retrospective Backward Synthesis (RBS) method would scale to more complex, continuous, real-world tasks. The absence of testing in high-dimensional or continuous state-action spaces limits insights into its broader applicability.
- Tuning Challenges: The method's reliance on hyperparameters, such as reward scaling and backward policy regularization, introduces tuning challenges. While these components are beneficial for stabilizing training, they require careful adjustment, potentially impacting the ease of replication and practical deployment in varied scenarios.
- Lack of Comparison with Model-Based RL: Despite the inherent use of backward trajectory synthesis, which resembles model-based planning, the paper does not compare RBS with established model-based RL approaches such as MBPO or Dreamer. This omission makes it difficult to assess how RBS performs relative to other methods that also utilize environment models for planning and sample efficiency.

questions:
- In algorithm 1) line 6, how do we guarantee that the backward policy could reach $s_0$ from $y$ each time?
- Are tuning for reward intensification and backward policy regularization difficult? What the the effect of hyper-parameters on the performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730813443859

thread_type:
official_review

<thread_object>
creator: reviewer_fnhf

<content>
summary:
The authors introduces a novel method, Retrospective Backward Synthesis (RBS), aimed at enhancing the training of goal-conditioned Generative Flow Networks (GFlowNets) by synthesizing new backward trajectories. RBS augments "virtual" backward trajectories in
goal-conditioned GFlowNets to enrich training trajectories with enhanced quality and diversity. RBS improves the sample efficiency and performance of GFlowNets across a range of tasks, including sequence generation and biological sequence design.

strengths:
1. The paper identifies and targets a significant issue in the training of goal-conditioned GFlowNets, offering a practical and innovative solution. Augmenting backward trajectories for training is interesting.
2. Comprehensive empirical results are provided, demonstrating the effectiveness of RBS over existing methods on multiple benchmarks.

weaknesses:
1. Limited Discussion on Potential Drawbacks: The paper does not sufficiently address the potential limitations of RBS. For instance, there is no discussion about the computational overhead of synthesizing backward trajectories, nor is there a mention of whether the method is robust to different types of reward structures or environment dynamics.
2. Relevance and Scope of Application: The improvements are made specifically within the context of GC-GFlowNets, which may limit the applicability of the method.
3. Comparison with Diffusion Policies: Given the similarities between the proposed RBS and diffusion policies, a direct comparison would be valuable to understand the unique contributions and differences of RBS.
4. Assumptions on Environment Dynamics: It is unclear whether the proposed RBS method assumes or requires any particular properties of the environment, such as determinism or stochasticity. If the backward dynamics are infeasible or the environment is highly stochastic, the performance of RBS may be affected, and this should be addressed.
5. Quality of Synthetic Trajectories: The paper should include a discussion on how to ensure the quality of the synthesized backward trajectories, especially in cases where such trajectories may not correspond to realistic or feasible paths in the actual environment.
6. Lack of Comparison with Goal-Conditioned RL: Without a comparison to goal-conditioned RL, it is difficult for readers to fully appreciate the relative strengths and weaknesses of GC-GFlowNets. Including such a comparison would provide a more complete picture of the method's positioning within the broader field of goal-directed learning.
7. The authors may further investigate existing literature on augmenting backward trajectories for sample-efficient RL or backward learning in goal-conditioned RL, which makes the paper more comprehensive.

questions:
1. How does RBS compare with diffusion policies, and in what scenarios does RBS offer distinct advantages?
2. Does RBS assume deterministic or stochastic environments, and how does it handle situations where the backward dynamics are not straightforward?
3. How can the authors ensure that the synthesized backward trajectories are meaningful and do not lead to false positives in the learning process?
4. Could the authors include a comparison with goal-conditioned RL methods to highlight the specific benefits of using GC-GFlowNets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731925728243

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all of the reviewers for their time and insightful comments. Furthermore, we are very glad to find that reviewers generally recognize the effectiveness, novelty and significance of our work:
- **Method**: The paper identifies and targets a significant issue in the training of goal-conditioned GFlowNets, offering a practical and innovative solution [**fNHf**]. This approach effectively improves sample efficiency by converting failed experiences into successful learning signal [**HyCh**]. RBS is introduced with clear motivation [**xjJi**]. This approach introduces rich learnable signals, effectively addressing the sparse reward problem [**HyCh, iEkK**]. This paper proposes a novel method called RBS [**iEkK**].

- **Presentation**: The paper is well-written and presented, with clear explanations, structured methodology, and comprehensive experimental results [**HyCh**]. The paper is well-written and straightforward to understand [**xjJi**]. 

- **Experiments**: Comprehensive empirical results are provided [**fNHf**]. The paper presents strong empirical results across a range of benchmarks, demonstrating that RBS markedly improves sample efficiency. Empirical results demonstrate that the proposed method outperforms baselines across a range of benchmarks, showing improved performance and sample efficiency [**HyCh, xjJi**].

Meanwhile, we thank all the reviewers for their helpful and constructive feedback to improve the quality of our work again. We have carefully updated our paper to incorporate the valuable suggestions from the reviewers. We summarize the revisions and added experiments as follows:

- [**fNHf**] We add additional experiments to demonstrate the computation efficiency of our method in Appendix D.1. Results are illustrated in Figure 15(a).
- [**fNHf**] We demonstrate that RBS-GFN can gain further performance improvements in a different reward structure, i.e., dense rewards. Please see the results in Figure 16(b) and Appendix D.6.
- [**fNHf, HyCh**] We evaluate RBS-GFN on stochastic environments to demonstrate its generalizability in Appendix D.5. Results are shown in Figure 16(d).
- [**fNHf, HyCh, iEkK**] We compare our method, i.e., RBS-GFN, with an advanced model-based goal-conditoned RL method. RBS-GFN consistently outperforms this method significantly. Details can be found in Figure 16(a) and Appendix D.4.
- [**HyCh**] We add an additional ablation study to show that RBS-GFN is robust to our introduced decay hyperparameter $\beta$. We provide the results in Figure 15(b) and Appendix D.3.
- [**xjJi**] We further demonstrate the superior efficacy of RBS-GFN on the RNA generation benchmark. Results are given in Figure 15\(c\) and Appendix D.2.
- [**iEkK**] We compare our proposed age-based sampling technique with PER. Results shown in Figure 16(c) and Appendix D.7 demonstrate that our method is more effective.
---
All changes in the new pdf are highlighted in red. 

We hope to have addressed all the raised concerns and would be happy to respond to further questions and suggestions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Retrospective Backward Synthesis (RBS), a novel method that addresses challenges in training goal-conditioned Generative Flow Networks (GC-GFlowNets) by enriching training data with synthesized high-quality, diverse backward trajectories. Additional techniques are introduced, such as reward signal intensification and backward policy regularization, to prevent mode collapse. Empirical results across various benchmarks demonstrating its superior performance in discrete environments.

The reviewers generally acknowledge the paper's contribution in its potential for advancing GC-GFlowNet applications, considering it both novel and solid. The paper is well-written, with a clear motivation, structured methodology, and comprehensive experimental results. Empirical evaluations demonstrate that RBS effectively addresses the sparse reward problem and significantly outperforms state-of-the-art methods in terms of sample efficiency, success rates, and scalability.

Despite these strengths, some weaknesses and limitations are noticeable in the original manuscript. First, the experiments are limited to discrete environments, raising concerns about the scalability of RBS to continuous real-world tasks. The evaluation tasks also omit some key benchmarks, such as RNA generation. Comparisons with alternative approaches, such as diffusion policies, goal-conditioned RL, and model-based RL methods, are absent, leaving the unique advantages and positioning of RBS unclear. Additionally, the paper does not adequately address the potential computational overhead of synthesizing backward trajectories or its robustness to different reward structures and environment dynamics. Furthermore, questions about the quality of synthesized trajectories, the feasibility of backward dynamics in stochastic environments, and the tuning challenges associated with hyperparameters like reward intensification remain unanswered.

The authors’ responses offer detailed clarifications to these questions and provide additional experimental results. Notably, they address the inherent difficulty of the benchmarks used in the paper, compare their approach with goal-conditioned RL, present experiments on RNA generation, demonstrate generalization to different environments and reward structures, and explain hyperparameter tuning along with its robustness.

Three out of four reviewers recommend acceptance. While the remaining reviewer did not provide follow-up feedback on the authors' response, it seems that the concerns raised have been addressed. Additionally, although two positive reviewers did not respond to the authors’ rebuttal, their initial assessments were favorable. I support the reviewers' recommendation and suggest that the authors carefully address the main points raised in the reviews, particularly focusing on additional analyses and experiments, when preparing the final version.