Generating GFlowNets as You Wish with Diffusion Process

Authors: Yuxin Li, Wangbo Zhao, Dongwen Tang, jiyao liu, Xiahai Zhuang, Guang Li, Dianbo Liu, Yang You, Kai Wang
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: GFlowNet, Parameter generation

Abstract:
Generative Flow Networks (GFlowNets) are probabilistic samplers that learn stochastic policies to generate diverse sets of high-reward objects, which is essential in scientific discovery tasks. However, most existing GFlowNets necessitate training, becoming costly as the diversity of GFlowNets expands and trajectory lengths increase. To alleviate this problem,  we propose a method to Generate high-performing GFlowNet parameters based on a given model structure, called GenFlowNet. Specifically, we first prepare an autoencoder to extract latent representations of GeFlowNet parameters and reconstruct them. Then, a structure encoder is trained alongside a conditional latent diffusion model to generate the target GFlowNet parameters based on the given structure information. To the best of our knowledge, it is the first exploration to generate parameters of a probabilistic sampler using the diffusion process. It enables us to obtain a new GFlowNet without training, effectively reducing the trial-and-error cost during GFlowNet development. Extensive experiments on diverse structures and tasks validate the superiority and generalizability of our method.

Decision: reject
Reviewer scores: 3, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730099516758

thread_type:
official_review

<thread_object>
creator: reviewer_h5ma

<content>
summary:
This paper explores parameter generation for GFlowNets, as GFlowNets require high costs to train, e.g., sampling an exponential number of trajectories. To generate parameters, this paper proposes a two-fold method: (1) generating a latent representation of parameters via reverse diffusion given the structural information of an environment and (2) decoding this representation into parameters. The overall method is similar to a prior study [1], but extends it to consider the condition specifying the environment. The experiments show that the proposed generative method can adapt to unseen environmental structures.

---

[1] Wang et al., Neural Network Parameter Diffusion

strengths:
- This work is the first to investigate the parameter generation for GFlowNets, which can also be naturally extended to reinforcement applications.
- The proposed method demonstrates that generated parameters can achieve similar or superior performance compared to parameters obtained from conventional training of GFlowNets.

weaknesses:
My primary concerns arise from doubts regarding the practical utility of the proposed methods.

- **About motivation.** This paper argues that generating the parameters given environmental information, e.g., state dimensions, makes it easy to obtain parameters for new environments. However, one might consider defining a forward policy conditioned on such information, e.g., $ P_F(s|s'; \text{Structure}) $, and training it. Given this alternative, what benefits does the parameter generation provides?

- **About extensibility.** Although the method considers environment-specific structural information, e.g., state dimension in a hyper-grid, it seems challenging to incorporate most components of GFlowNets in practice. For example, in specifying information of environments, how should one consider different reward functions, e.g., addressing different properties, or different action spaces, e.g., fragment-based and reaction-based transitions?

- **About experiments**. The considered tasks are limited to show usefulness of the proposed method (only considers eight different structures for a hyper-grid). Furthermore, the most experiments only consider the relatively simple task, i.e., a hyper-grid, and simple structural variations, i.e, changes in dimensions, without exploring more advanced tasks, e.g., RNA sequence generation, or complex environmental variations. Although the experiments consider a molecular generation task, it does not specify what $I$ is.  

- **About results.** In Table 4 and 5, the performance improvements seem too minor. Especially, in Table 4, it is hard to understand why $N=2$ and $N=5$ yield the similar generalization performance. Can you clarify more details on this?

questions:
- What is $I$ in Table 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730309297248

thread_type:
official_review

<thread_object>
creator: reviewer_jptd

<content>
summary:
The authors proposed using a VAE to learn the latent space of GFlowNet parameters, followed by using DMs to generate the GFlowNet parameters. They used different synthetic data for the evaluation.

strengths:
If the method proves effective in real scenarios, it could help improve training in situations where GFlowNet struggles; however, the current experiments do not support this outcome.

weaknesses:
- The experiments are based solely on synthetic data, which does not strongly support most of the claims, such as those in Figure 1.

- While the authors mention where GFlowNet training struggles—such as with increasing trajectory length in the abstract—they do not clarify whether they successfully addressed these issues.

- The novelty is somewhat limited, as it relies on an existing latent diffusion model without any modifications.

questions:
- What are the specific distinctions that make adapting existing parameter generation methods challenging for GFlowNet parameters?

- The dataset preparation for the AE component lacks clarity. Could you please elaborate on this? How many models did you generate? How many samples were created? Which datasets were used for this purpose?

- Including additional real-world datasets would be greatly appreciated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730753456092

thread_type:
official_review

<thread_object>
creator: reviewer_wwiu

<content>
summary:
This paper presents a novel idea to generate GFlowNets, which are deep learning models that hierarchically generate sequential actions in parameter space. They use an autoencoder to create a latent mapping of parameters and use a conditional diffusion model in the latent space to generate a proper latent representation of parameters. This method enables generalization over the parameter space, allowing us to obtain new GFlowNets without training.

strengths:
1. Very novel idea (more like crazy idea).

weaknesses:
There are several concerns:

1. Scalability may be limited.


2. The motivation is unclear—why is this approach necessary?


3. The empirical results do not reflect real-world performance.

questions:
1. Can this method be used to measure uncertainty from a Bayesian perspective? I'm asking because this work seems to be connected with AutoML and hypernetworks.


2. Can you provide a clearer motivation for why we need this?


3. Is this method scalable to large-scale tasks that require very complex parameterizations (e.g., LLMs, text-to-image models) within GFlowNets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731291216248

thread_type:
official_review

<thread_object>
creator: reviewer_ipe3

<content>
summary:
The authors propose a method to generate GFlowNets based on previously trained policy. Their method (GenFlowNet) condenses policy parameters using an autoencoder. Then, it employs a latent conditional diffusion process in the latent space to create new policy parameters conditioned on an encoding of the target architecture.

strengths:
* To the best of my knowledge, the first work on developing generalizable initializations of parameters for GFlowNets.

weaknesses:
* Experiments lack proper description. For instance, are different Rewards used to train the auto-encoder in the hypergrid task? Or do you fix the same $R\_0$, $R\_1$, $R\_2$, and $H$ for all GFlowNets comprising the training set? Is the aim solely to create new architectures for the same reward --- that has been learned before? I have several questions regarding the experimental setup below. If the method cannot generalize to unseen rewards, I don't see how it can be useful.

* While the hyper grid task and molecule generation are challenging ones, the experimental suit is rather slim compared to other recent works in the GFlowNet literature

* There are limitations that the authors do not properly address. For instance, how does GenFlowNet behave when for varying rewards? How does it fare when the forward policies are not MLPs?

* No error bars or standard deviation.

questions:
* What is the unit for "Time usage" in Table 1? Seconds, hours?

* In line 307, the authors highlight the "superior performance in sampling distribution accuracy". This seems like an overstatement, given the very small gaps in Table 1 and the lack of uncertainty measurements.

* Authors state GenFlowNets enable the generation of accurate GFlowNets without training. However, Figure 4 makes it look like the parameters generated by GenFlowNets are used as initialization. Is this correct? Please elaborate.

* It seems odd that a training-free GFlowNet would perform better than a trained one (assuming the latter is properly trained). Could you share a rationale for this?

* For section 3.3, are the GFlowNets drawn from the GenFlowNet trained for the hypergrid task? Please provide more details
</content>
</thread_object>

</thread>
<thread>
creation date:
1732681015354

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear ACs and reviewers,

We sincerely appreciate the time and effort provided by all reviewers and ACs in our work. In particular, we are encouraged to see that Reviewer iPE3 finds that our method **is "the best work on developing generalizable initializations of parameters for GFlowNets"**. Reviewer Wwiu thinks the proposed method **” is a very novel idea“**. Reviewer JPtD **“The method proves effective in real scenarios"**. Reviewer h5Ma acknowledges that the method proposed in this paper **"can achieve similar or superior performance compared to conventional training methods"**.

We addressed each of the reviewers' comments individually, and we will continue to refine our work based on the valuable feedback provided.

Thanks,

Authors of submission 3398
</content>
</thread_object>

</thread>


Metareview: