COFlowNet: Conservative Constraints on Flows Enable High-Quality Candidate Generation

Authors: Yudong Zhang, Xuan Yu, Xu Wang, Zhaoyang Sun, Chen Zhang, Pengkun Wang, Yang Wang
Primary area: reinforcement learning
Keywords: generative flow network, offline RL, molecule design

Abstract:
Generative flow networks (GFlowNets) have been considered as powerful tools for generating candidates with desired properties. Given that evaluating the property of candidates can be complex and time-consuming, existing GFlowNets train proxy models for efficient online evaluation. However, the performance of proxy models is heavily dependent on the amount of data and is of considerable uncertainty. Therefore, it is of great interest that how to develop an offline GFlowNet that does not rely on online evaluation. Under the offline setting, the limited data results in an insufficient exploration of state space. The insufficient exploration means that offline GFlowNets can hardly generate satisfying candidates out of the distribution of training data. Therefore, it is critical to restrict the offline model to act in the distribution of training data. The distinctive training goal of GFlownets poses a unique challenge for making such restrictions. Tackling the challenge, we propose Conservative Offline GFlowNet (COFlowNet) in this paper. We define unsupported flow, edges containing unseen states in training data. Models can learn extremely little knowledge about unsupported flow from training data. By constraining the model from exploring unsupported flows, we restrict COFlowNet to explore as optimal trajectories on the training set as possible, thus generating better candidates. In order to improve the diversity of candidates, we further introduce a quantile version of unsupported flow restriction. Experimental results on several widely-used datasets validate the effectiveness of COFlowNet in generating high-scored and diverse candidates. All implementations are available at https://github.com/yuxuan9982/COflownet.

Decision: accept (poster)
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729839615218

thread_type:
official_review

<thread_object>
creator: reviewer_7qzz

<content>
summary:
In this paper, the author introduces COFlowNet, an offline Generative Flow Network (GFlowNet) designed to generate high-quality and diverse candidates without relying on online evaluations. The effectiveness of COFlowNet is validated through experiments on commonly used datasets.

strengths:
+ Adapting GFlowNets for offline scenarios is a valuable idea.
+ The proposed method shows superiority over baseline models like QM-GFlowNet and FM-GFlowNet.

weaknesses:
1. The authors conduct experiments on Hypergrid and molecule design tasks. Could the proposed method be generalized to more tasks and real applications? It is recommended to conduct experiments on additional tasks.

2. How much time and computational cost are saved by training in the offline setting? Could the authors compare the training cost between the proposed method and traditional methods?

3. It would be better to use \textbf{} for the first sentence in figure and table captions to indicate their targets clearly.

4. It would be  better to indicate the meaning of 'mixed', 'expert', and 'random' in Figure4.

5. Do number of modes and ℓ1 error are commonly used in evaluation of GFLowNets? Are there more metrics to measure the performance?

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685175900

thread_type:
official_review

<thread_object>
creator: reviewer_9ymw

<content>
summary:
The paper present a method to the problem of offline GFlowNet training. Many conventional approaches require approximating training scores/rewards of candidate model outputs to provide a variety of generated candidate solutions, however, this approach is sensitive to out-of-distribution data. In this work the authors remove any reliance on proxy models for estimating the candidate to reward optimization objective for GFlowNet training. The authors propose to constrain the actions that a GFlowNet model may take to generate an output sample such that only actions supported by the training data are permissible. The authors show that their flow regularization only positively affects training on the supported flow paths, and that their approach is truly offline and can improve candidate model diversity.

strengths:
The authors demonstrate originality in their work by clearly outlining their research problems in the context of other popular works and systematically laying out solutions for them with corresponding theorems and proofs. While maintaining an easy-to-read and high-level dialogue in addressing their approach’s significance, aim, and scope, the authors also manage to spare no mathematical details when necessary. Theorem 1 and the writing in sections 3.3 and 4 show formally the details required to both implement and understand the proposed approach.

weaknesses:
In theorem 2, the use of the phrase “be more close to policy of offline dataset” seems too imprecise to be in the theorem statement. Since the proof is rather short this could perhaps be moved to the end of the proof, so that reader’s still get the high-level point that the author is trying to convey. Additionally, the conclusion of the proof for theorem 2 seems to be absent, although the technical/algebraic details seem sound. 

Although the authors effectively demonstrate the efficacy of COFlowNet on the Molecule Design problem, providing experimental results on datasets from other domains would help show the success of their method beyond a shadow of a doubt.

questions:
1. No references are provided for vanilla GFlowNet and QM-GFlowNet. There is no justification that these are the state-of-the-art methods.
2. Although COFlowNet w/o shows that the regularization term enhances performance, but can not beat the competing method QM-GFlowNet. The experiment results are weak. If the experiment results can not be improved, please provide a more in-depth analysis of where COFlowNet falls short compared to QM-GFlowNet.
3. Are there other potential ways of increasing diversity during GFlowNet regularization that were explored aside from the quantile matching algorithm? 
4. Can COFlowNet be easily extended to other domains where RL is popular, like game playing and autonomous vehicle driving for example?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714392138

thread_type:
official_review

<thread_object>
creator: reviewer_z4pm

<content>
summary:
This paper tackles the generation task, which aims to search for state transitions with high rewards. The fundamental task strongly impacts generating or searching for plausible candidates of sequences, where its possible applications include molecule design or path generation.
To this end, the authors have proposed COFlowNet, which focuses on finding high-reward state transitions by simply regularizing unsupported flows. The efficacy of the method is confirmed in Hypergrid and Molecule Design tasks.

strengths:
**Strength 1:** The main strength of this work is the broad impact on any candidate-generation tasks.
- I feel that the method can be applied to any candidate-generation task, such as molecule design. Also, as pointed out in the paper, flow generation can be further used to search plausible flows to find the optimal decision-making in the RL study.

weaknesses:
**Weakness 1:** The comparison with prior works and the evaluations seem limited.
- By referring the baseline paper of GFlowNet, it is compared with other non-flow-based approaches such as MCMC (MARS) and PPO. However, COFlowNet is only compared with GFlowNet, which makes it hard to figure out the quantitative gains of COFlowNet over MCMC and PPO. It is better to add these two baselines in the experiments.
- Moreover, only the main experiment considered in this paper, which includes the comparisons with others, is Molecule Design. (I have not published works for this topic) In literature, is there any other task to demonstrate the efficiency of the proposed method? A single demonstration seems to be limited to say the consistent and meaningful gains of COFlowNet over others.

**Minor Comment 1:** For consistency, please choose the one among COFlowNet or COFlownet.

questions:
**Questions 1:** Regarding the weakness in literature, is there any other task to demonstrate the efficiency of the proposed method? If so, it would be much better to add one or two more results of COFlowNet.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733052454498

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A brief rebuttal to reviewers' comments.

comment:
Dear Area Chair and Reviewers,

We deeply appreciate the efforts made by the Area Chair and reviewers on our manuscript. We have point-by-point responded to all the concerns raised by the reviewers and revised our manuscript accordingly. 

**As the discussion period coming to an end, we would like to summarize our responses.** The major **C**oncerns of the reviewers are two folds: (**C1**) *Experiments on more tasks are required to further validate the effectiveness of the proposed COFlowNet,*  and (**C2**) *Performance of COFlowNet is slightly worse than that of QM-GFlowNet.* For your convenience, We provide a brief rebuttal below.

**(C1) Additional experiments on more tasks.** Following the suggestion of reviewers, we conducted additional experiments to evaluate COFlowNet on two more tasks: **Anti-Microbial Peptide Design** and **Item Recommendation**. The results demonstrate that COFlowNet exhibits excellent offline learning capabilities and consistent effectiveness across various tasks. We have incorporated these results and detailed analyses into our revised manuscript (**Section B**).

**(C2) Performance of COFlowNet.** COFlowNet is an offline flow-based model, and can solely learn from collected data, which covers limited state space. However, online models, including GFlowNet and QM-GFlowNet, have a better ability to explore the whole state space by interacting with the proxy model. Consequently, given a good proxy model, offline models can hardly outperform online models in generating high-score candidates. We have provided in-depth analyses in our responses and updated the manuscript (**Section 4.2.3**) to thoroughly clarify these distinctions.

We sincerely hope that our responses can address the reviewers' concerns, and warmly invite reviewers to engage in further discussions with us. Thank you once again for your valuable time and continued support. Wishing you all the best in your endeavors!

Best regards,

Authors of submission 9900
</content>
</thread_object>

</thread>
<thread>
creation date:
1733189851892

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Gentle Reminder: Discussion Period Nearing Closure

comment:
Dear Reviewers,

We sincerely appreciate your valued efforts and continued support of our work. **We have carefully responded to each of your comments and thoroughly revised the manuscript accordingly.** We are deeply honored that our detailed responses can address your concerns during the rebuttal process.

As the discussion period is coming to a close on **December 2nd (today)**, and If you have no further comments, **we kindly remind you to consider updating your score.**

Thank you again for your time and consideration!

Best regards,

Authors of submission 9900
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents the Conservative Offline GFlowNet (COFlowNet) framework, aimed at enhancing the quality and diversity of generated candidates under offline setting. Overall, the paper demonstrates strong theoretical rigor, meticulous technical details, and comprehensive experimental evaluations. The majority of reviewers expressed satisfaction with the authors’ responses and provided positive feedback on the paper:

1.It introduces an offline learning framework for GFlowNets, offering a novel research perspective in this domain.

2.The method exhibits strong generalizability, achieving impressive experimental results across multiple practical tasks.

3.The comparison between offline and online methods is clear and insightful, effectively highlighting their respective application scenarios and advantages.

Thus, it is recommended that this paper be accepted.