Title: Discovering Latent Graphs with GFlowNets for Diverse Conditional Image Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: generative models

diffusion model

diversity

gflownet.
Abstract: Capturing diversity is crucial in conditional and prompt-based image generation, particularly when conditions contain uncertainty that can lead to multiple plausible outputs. To generate diverse images reflecting this diversity, traditional methods often modify random seeds, making it difficult to discern meaningful differences between samples, or diversify the input prompt, which is limited in verbally interpretable diversity. We propose \modelnamenospace, a novel conditional image generation framework, applicable to any pretrained conditional generative model, that addresses inherent condition/prompt uncertainty and generates diverse plausible images. \modelname is based on a simple yet effective idea: decomposing the input condition into diverse latent representations, each capturing an aspect of the uncertainty and generating a distinct image. First, we integrate a latent graph, parameterized by Generative Flow Networks (GFlowNets), into the prompt representation computation. Second, leveraging GFlowNets' advanced graph sampling capabilities to capture uncertainty and output diverse trajectories over the graph, we produce multiple trajectories that collectively represent the input condition, leading to diverse condition representations and corresponding output images. Evaluations on natural image and medical image datasets demonstrate \modelnamenospace’s improvement in both diversity and fidelity across image synthesis, image generation, and counterfactual generation tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRainbow is a plug-and-play framework that adds explicit diversity to any pretrained conditional image generator. Instead of varying random seeds or paraphrasing prompts, Rainbow decomposes the input condition into multiple latent representations. Concretely, it inserts a latent graph—parameterised and sampled by a Generative Flow Network (GFlowNet)—into the prompt-encoding stage. Each GFlowNet trajectory captures a different plausible interpretation of the condition; feeding these trajectories through the frozen generator yields a batch of diverse, yet still faithful, images. Experiments on natural and medical datasets show simultaneous improvements in diversity (e.g., higher LPIPS / dispersion scores) and fidelity (lower FID) for image synthesis, conditional generation and counterfactual tasks, demonstrating Rainbow’s generality and effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStenghs:\n1. Rainbow can be bolted onto any pretrained conditional generator without retraining the base model, making it easy to adopt across diffusion, GAN and autoregressive backbones.\n2.  Instead of random seeds or ad-hoc prompt edits, the method explicitly decomposes the condition into multiple latent routes, yielding diversity that is semantically tied to ambiguities in the original prompt.\n3. Leveraging GFlowNets’ trajectory-sampling provides a mathematically grounded way to cover high-reward—but diverse—modes of the latent graph, avoiding mode collapse common in MCMC or pure random seeding.\n\n\nWeaknesses:\n1. Comparisons are mostly against “different seeds” or prompt-paraphrasing. State-of-the-art diversity techniques—e.g., DDIM stochasticity schedules, classifier-free guidance sweeps, Multimodal MCTS, or Latent Diffusion Ensembles—are missing, making it hard to gauge Rainbow’s real advantage.\n2. Sampling diverse trajectories may lower prompt adherence or introduce artifacts. The paper reports FID and a single diversity metric but provides no human evaluation or CLIP-based alignment score to show that each variant still obeys the original condition.\n3. GFlowNets are known to be sensitive to reward scaling and exploration temperature. The paper does not include convergence curves, variance across runs, or guidelines for hyper-parameter selection, which may hinder reproducibility.\n4. The abstract states that each trajectory “captures an aspect of the uncertainty,” but no qualitative or quantitative analysis is given to link individual graph paths to human-interpretable factors (e.g., colour palette, viewpoint).\n\nQUESTIONS:\nHow do graph depth, branching factor, and node-feature type affect both diversity and fidelity?\nCould you provide an ablation that varies these hyper-parameters?\n\nLIMITATIONS:\nsee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"Rainbow,\" a novel framework designed to enhance the diversity of conditional image generation models. The core problem addressed is the inherent ambiguity in input conditions (e.g., text prompts), which can correspond to multiple plausible image outputs. The authors' key idea is to decompose a single input condition into multiple, diverse latent representations. This is achieved by introducing a latent graph structure and employing Generative Flow Networks (GFlowNets) to sample a set of diverse trajectories (subgraphs) over this graph. The method's efficacy is demonstrated across multiple domains, including natural image generation (Flickr30k), 3D medical imaging (Brain MRIs), and 2D medical imaging (Chest X-rays). The results show consistent improvements in both diversity (measured by Vendi Score) and image quality (measured by IS and FID) compared to strong baselines like various versions of Stable Diffusion, CADS, and PAG. The paper also provides compelling evidence for the interpretability of the learned latent graphs and the positive impact of the generated data on downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nNovel and Sound Methodology: The primary strength of this work lies in its innovative integration of GFlowNets with latent diffusion models. Using GFlowNets to explicitly model a distribution over diverse graph structures that represent conditional uncertainty is a principled and powerful approach. It directly tackles the mode-seeking behavior of many generative models and provides a more structured way to explore the solution space than simple noise manipulation. The technical execution, including the use of the Detailed Balance objective and the design of the graph generator and decoder, is well-reasoned and clearly explained.\n\nComprehensive Empirical Evaluation: The authors have conducted an extensive set of experiments across three distinct and challenging domains. The visual comparisons (e.g., Figure 2, \"Sunset scene with mountain\") are convincing. Rainbow clearly generates a wider variety of seasons, lighting, and compositions compared to the baselines, which often produce repetitive or stylistically limited outputs. The quantitative analysis (Figure 4, Table 1, Table 3) supports the claims. Rainbow consistently achieves a better trade-off between diversity (higher Vendi Score) and fidelity (higher IS, lower FID) than the baselines.\n\nModel-Agnostic and Flexible Framework: An advantage of Rainbow is its applicability to any pretrained conditional generative model. By operating on the condition representation and keeping the core generative model frozen, it serves as a versatile plug-in. This is impressively demonstrated by its successful application to different Stable Diffusion versions for natural images and chest X-rays, and a MONAI-optimized LDM for 3D brain MRIs.\n\n\nWeaknesses\n\nComputational Overhead: The paper acknowledges that the method requires higher computational resources due to the parallel generation of M trajectories (graphs). While the results justify the cost, a more detailed analysis of the trade-off between the number of graphs (M) and performance versus computational cost would be beneficial. The ablation study (Figure 10) shows that performance degrades as M decreases, but the cost-benefit analysis isn't fully fleshed out.\n\nClarity on Graph Hyperparameters: The choice of the latent graph size (N nodes) and sparsity (ρ) seems somewhat arbitrary. The ablation study explores these parameters for the Chest X-ray experiment, but a more in-depth discussion on the intuition behind selecting these values for different domains would strengthen the paper. For instance, why was N=20 chosen for natural images but N=8 for brain MRIs? Is there a principled way to set these, or is it purely empirical?\n\nLimited Discussion on Failure Cases: While the presented results are strong, the paper would be more complete with a discussion of failure modes. Are there types of conditions or prompts where Rainbow struggles to generate diversity or maintain fidelity? For example, how does it handle highly complex prompts with many entangled concepts?\n\nDifference with Subspace-based Methods: It seems that the core goals of rainbow and subspace-based representation learning methods are very similar. Why not use subspace methods to learn different semantic representations?\n\nQUESTIONS:\n1. Regarding the latent graph interpretation (Section 4.2.2): How were the sets of edges corresponding to specific concepts (e.g., \"spring edges\") identified? Was this an automated clustering process based on edge frequency, or was there manual selection involved after clustering the output images? Please clarify the procedure.\n\n2. Could you elaborate on the choice of the RNN in the graph decoder ($Q_D$)? Since the final graph is an unordered set of edges, is the sequential processing of the RNN critical? Have you experimented with non-sequential architectures like a simple pooling or a Transformer encoder?\n\n3. In Table 1, the performance of the age/sex classifiers on real data is provided as a reference. What is the performance of these classifiers on the training set of the real data? This would help contextualize the generalization gap and better interpret the results on the synthetic data.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRainbow is a new method for generating diverse images from text prompts or other conditions. The main idea is that when given a prompt like \"sunset scene with mountain\" there are many valid ways to interpret it - different seasons, lighting, perspectives, etc. Current methods struggle to capture this diversity, which was shown in different works. Rainbow addresses this by using Generative Flow Networks (GFlowNets) to create multiple graph trajectories that represent different interpretations of the input condition. Each trajectory gets decoded into a different latent representation, which then generates different image from the trajectory.\nThe authors test this on three datasets: natural images, brain MRIs, and chest X-rays. Results show Rainbow generates more diverse images while maintaining quality compared to baselines like Stable Diffusion.\nThe key innovation is connecting GFlowNets (which are good at sampling diverse solutions) with diffusion models (which generate high-quality images). The learned graphs also seem interpretable - certain edges correspond to concepts like \"winter\" or \"devices.\"\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n- Novel combination of GFlowNets with diffusion models that hasn't been explored before\n- Strong experimental results across three very different domains (natural images, brain MRI)\n- The learned graph structures show some interpretability - can identify edges corresponding to seasons or medical devices\n- Comprehensive evaluation with multiple metrics (IS, VS, FID, CLIP scores)\n- Works with any pre-trained conditional diffusion model without major modifications\n\n### Weaknesses:\n\n- Computational cost seems high (M=40 parallel trajectories) but no timing comparisons provided\n- Hyper parameter selection (nodes, graphs, sparsity) appears arbitrary and domain-specific\n- The method might be forcing diversity even when the prompt is very specific\n- Unclear how stable the interpretable edges are across different training runs\n– minor thing  hard to distinguish (trajectories vs graphs) as notation\n\nQUESTIONS:\n1. How did you decide on using graph structures to represent uncertainty? Were other structured representations considered or related works can be improved for that then?\n2. What is the computational overhead of Rainbow compared to standard diffusion models - can you add this to the paper to make it clear? How does inference time scale with the number of graphs M?\n3. When given a very specific prompt with little ambiguity (like a red circle on white background), does Rainbow still try to force diversity?  How does it handle low-uncertainty conditions?\n4. Are the interpretable edges (e.g. seasons) consistent across different training runs or did you select the best examples?\n5. Is there a principled way to choose M, N, and $\\rho$ for a new domain, or does it require extensive hyperparameter search -- not sure I feel the intuition here?\n\nLIMITATIONS:\nThe method requires training additional components top of pre-trained models, which adds complexity.\nNo principled way to set hyperparameters - each domain uses very different values\nThe approach may not scale well to high-resolution images due to memory requirements of maintaining M parallel trajectories\nThe interpretability of edges is only demonstrated through cherry-picked examples; no systematic evaluation of interpretation stability\nAssumes all input conditions have meaningful uncertainty to exploit, which may not always be true\nThe connection between graph sparsity and semantic diversity is assumed rather than proven", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework named Rainbow, designed to enhance the diversity of images produced by conditional generative models. The core problem it addresses is that a single input condition or prompt often has inherent uncertainty and can correspond to multiple plausible visual outputs, which standard generative models fail to capture, often producing repetitive results.\n\nRainbow is proposed as a model-agnostic layer that can be integrated with any pretrained conditional generative model, such as a Latent Diffusion Model (LDM), without altering its core components. The method operates in a multi-step process:\n\n* Initial Representation: An input condition (e.g., a text prompt or a set of attributes) is first encoded into an initial latent representation using a standard condition encoder.\n\n* Latent Graph Discovery with GFlowNets: The framework's key innovation is the use of Generative Flow Networks (GFlowNets). A GFlowNet is trained to sample a set of diverse trajectories over a latent graph structure. Each trajectory represents a different plausible interpretation of the uncertainty contained within the initial condition.\n\n* Diverse Condition Generation: These diverse graph trajectories are then decoded into a set of distinct condition representations. Each new representation captures a unique aspect of the input condition's ambiguity.\n\n* Diverse Image Generation: Finally, each of these diverse condition representations is fed into the frozen, pretrained generative model (e.g., an LDM) to produce a corresponding output image. The result is a set of varied images from a single initial prompt.\n\nThe authors evaluate Rainbow on both natural image (Flickr30k) and medical imaging (3D Brain MRI, Chest X-ray) datasets. The results are compared against baselines like Stable Diffusion and other diversity-enhancing methods using metrics for diversity (Vendi Score) and quality (Inception Score, FID). The experiments aim to show that Rainbow can generate a more diverse set of high-quality images that better reflect the inherent ambiguity of the input conditions.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n*Methodological Novelty: The paper introduces a novel framework by coupling Generative Flow Networks (GFlowNets) with latent diffusion models. This approach, which samples multiple graph trajectories to capture multimodal uncertainty, is a notable contribution to the field of conditional image generation.\n\n* Consistent Performance Across Domains: The method demonstrates strong and consistent empirical gains across three distinct domains: natural images, 3D brain MRIs, and 2D chest X-rays. The quantitative improvements in both diversity (Vendi Score) and fidelity (IS, FID) over strong baselines validate the effectiveness of the approach.\n\n* Latent Graph Controllability: A significant strength is the demonstration that the learned latent graphs are not just a passive component but can be actively manipulated. The experiments showing that injecting \"season-specific\" or \"support-device\" edges leads to targeted semantic edits suggest that the graph encodes meaningful, editable concepts.\n\n* Transparent Experimental Design: The paper provides a high degree of transparency, detailing hyperparameters and outlining the training algorithm with pseudocode. The inclusion of ablation studies on key parameters like N, M, and ρ further enhances the reproducibility and clarity of the work.\n\n## Weaknesses\n\n* Opaque Edge Semantics: While the latent graph is shown to be controllable, its components lack inherent interpretability. The mapping from graph edges to human-readable concepts is not automatic and must be inferred through post-hoc analysis. This \"black box\" nature of the graph's semantics is a core limitation, as it makes it difficult to understand or directly control the specific type of diversity being generated.\n\n\n* Indirect Reward Signal for GFlowNet: The GFlowNet is trained to generate diverse graphs based on a reward signal derived from the final denoising error of the diffusion model. This is a highly indirect objective. The GFlowNet is not explicitly guided to find graphs that correspond to semantically meaningful variations, but rather graphs that result in low-error reconstructions. This could potentially bias the model toward modes of variation that are easy for the LDM to render, rather than ones that fully capture the true underlying uncertainty in the prompt.\n\nQUESTIONS:\n1. **Reward design** – The current GFlowNet reward is the exponential of the final denoising MSE, implicitly tied to modes the frozen LDM already renders well. Have you experimented with composite or alternative rewards (e.g., CLIP‐based semantic distances, pairwise contrastive terms) to reduce this potential bias, and if so how did they affect diversity and quality?\n\n2. **Scaling the latent graph** – Reported models use up to *N = 20* nodes and *M = 40* trajectories. What practical or theoretical bottlenecks did you encounter when trying larger *N* or *M*, and do you foresee algorithmic modifications (e.g., sparse batching, trajectory pruning) that would let Rainbow scale to higher‐resolution video or multimodal inputs without prohibitive memory growth?\n\n3. **Automated graph interpretability** – Qualitative edits (e.g., “season” edges, “support‐device” edges) suggest that some latent edges align with human concepts, yet Appendix C lists graph interpretation as future work. Could you outline planned or preliminary methods (attention tracing, concept activation maps, etc.) for systematically mapping edges to semantic attributes?\n\n4. **Evaluation beyond automated metrics** – Diversity and fidelity are assessed with Vendi, IS, FID, and CLIP similarity. Are there plans to incorporate human studies or downstream‐task evaluations (e.g., radiologist scoring, prompt relevance surveys) to validate that Rainbow captures semantically meaningful uncertainty rather than pixel‐level variation?\n\n5. **Clinical deployment pathway** – For the 3-D MRI and chest-X-ray settings, Appendix C notes the need for anatomical-plausibility checks before use in clinical pipelines. Could you elaborate on specific validation protocols (e.g., expert grading, quantitative anatomical metrics) and on whether constraints or priors could be integrated into Rainbow itself to enforce medical realism during sampling?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}