Title: Learning Diffusion Models with Flexible Representation Guidance
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Models

Representation Learning

Image Generation

Biomolecule Generation

Theoretical Analysis of Generative Model
Abstract: Diffusion models can be improved with additional guidance towards more effective representations of input. Indeed, prior empirical work has already shown that aligning internal representations of the diffusion model with those of pre-trained models improves generation quality. In this paper, we present a systematic framework for incorporating representation guidance into diffusion models. We provide alternative decompositions of denoising models along with their associated training criteria, where the decompositions determine when and how the auxiliary representations are incorporated. Guided by our theoretical insights, we introduce two new strategies for enhancing representation alignment in diffusion models. First, we pair examples with target representations either derived from themselves or arisen from different synthetic modalities, and subsequently learn a joint model over the multimodal pairs. Second, we design an optimal training curriculum that balances representation learning and data generation. Our experiments across image, protein sequence, and molecule generation tasks demonstrate superior performance as well as accelerated training. In particular, on the class-conditional ImageNet $256\times 256$ benchmark, our guidance results in $23.3$ times faster training than the original SiT-XL as well as four times speedup over the state-of-the-art method REPA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends REPA, a method originally developed to accelerate diffusion transformer training for image generation, to various modalities. While REPA focuses on aligning intermediate features between DINO and the diffusion transformer, REED further aligns representations from multiple modalities—such as text—during image generation. Additionally, REED is the first to introduce REPA-style regularization terms into protein sequence and molecule generation, demonstrating its applicability beyond vision tasks such as protein sequence generation and molecule generation.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\n- Novel generalization of REPA to multimodal representation\n\n- First to apply REPA-like regularization to non-visual generative tasks\n\n- Demonstrates consistent performance improvements over baselines across all tasks\n\n[Weaknesses]\n\n- While REPA was motivated by improving training efficiency, REED uses Qwen2-VL 7B for image generation—a much heavier model than DINO—raising doubts about actual training speed improvements. A comparison of GFLOPS or latency is needed to confirm whether REED's 1M iterations in Table 1 are more efficient than REPA’s 4M, and whether 200 epochs in Table 2 are indeed more efficient than REPA’s 800. Figure 1 should also use GFLOPS or latency on the x-axis instead of training epochs.\n\n- Section 2's theoretical characterization is not novel, and seems unnecessary in this paper. The connection between ELBO and diffusion models is well-known [A], and the effectiveness of timestep weighting [B] has been established in prior work. I feel Sections 2 and 3 are loosely connected.\n\n[A] Kingma et al., Variational Diffusion Models, NeurIPS 2021.\n\n[B] Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation, NeurIPS 2023.\n\nQUESTIONS:\nWhat would be the effect of using a smaller multimodal model, such as CLIP or SigLIP, instead of Qwen2-VL for image generation? Would REED retain its benefits while becoming more efficient?\n\nLIMITATIONS:\nThe section 5 should further discuss the computation overhead by using large external models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a systematic framework REED (Representation-Enhanced Elucidation of Diffusion) for incorporating representation guidance into diffusion models. The authors provide theoretical insights into how high-quality pretrained representations can enhance diffusion model training, proposing two novel strategies: (1) integrating multimodal representations with synthetic data, and (2) designing an optimal training curriculum that balances representation learning and data generation. The framework is demonstrated across three domains: image generation, protein sequence generation, and molecule generation, showing good performance and accelerated training compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n1. The paper is technically sound with well-supported claims through both theoretical analysis and extensive experiments. \n2. The paper tests the algorithm on three tasks: image generation, protein generation, and molecule generation.\n3. REED seems to show better performance compared to baselines.\n\n### Weaknesses\n\n1. Could you provide more details on how the synthetic data is generated by VLM? Such as instruction prompts and caption examples.\n2. Experiments use Qwen2-VL 2B to generate synthetic captions and use representations from Qwen2-VL 7B? What’s the purpose to use different VLMs?\n3. In Section 3.1, the paper mentions using features from both a primary modality x and a secondary modality y. However, in the experiments, the authors chose to use features from VLMs. While VLMs do incorporate both text and image information, this appears more like a fused feature representation rather than distinct modalities as suggested by the theory. This seems to deviate significantly from the theoretical framework, particularly regarding how hyperparameters control the alignment between two separate features.\n    \n    Additionally, if VLMs are used as the aligned features, the key difference from REPA (which uses DINO features) seems to be primarily the choice of features (VLM vs. DINO). Could you elaborate more on the main experimental distinctions between these two works?\n    \n4. In Appendix C.1.2, from equation (21) to (22), it seems impossible to derive $p(z|x_0)p(x_0|x_1)=p(z|x_1)p(x_0|x_1,z)$.\n\nQUESTIONS:\n1. The paper mentions using synthetic paired data to address the scarcity of multimodal data. Could the authors provide more details on how the synthetic data is generated? Clarifying this could strengthen the reproducibility of the results.\n2. Can you explain why use Qwen2-VL 2B to generate synthetic captions and use representations from Qwen2-VL 7B?\n3. Could you provide more details between the theoretical analysis and experiment details?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper introduces aframework for enhancing diffusion models through representation guidance, i.e. aligning the internal representations of diffusion models with pretrained representations like DINO features. The authors derive a variational bound incorporating auxiliary representations and show how a weighting schedule can control when and how these representations guide the denoising process. \ntwo key strategies are proposed: (1) multimodal representation alignment using synthetic paired data (2) training curriculum that phases in the diffusion loss. REED is evaluated across three tasks: image generation, protein inverse folding, and molecule generation. Showing improved performance and faster training compared to prior methods like REPA.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is built on a theoretical motivation, deriving a variational bound that incorporates pretrained representations. The experimental section demonstrates clear performance gains.\n\n- The paper is logically organized, moving from theory to methods and then to experiments. Key ideas, such as the use of synthetic multimodal data, are illustrated with figures.\n\n- The paper addresses the important challenge of improving the efficiency and quality of diffusion models. The application to protein and molecular domains shows potential for impact in scientific discovery.\n\n\n**Weakness**\n- The experiments could be stronger with more rigorous ablations to rule out simpler explanations for the performance gains. FOr example,  could the authors report results for a baseline where the full REED model is trained without the proposed curriculum? i.e. with constant weighting for both the diffusion loss and the representation loss? \n\n- The method's success is heavily dependent on extremely large, resource-intensive pretrained models like DINOv2. This reliance is a barrier to wider adoption, and application in resource-constrained settings, potentially limiting the practical impact of the work.\n\n- The work is largely an extension of REPA. While the new components are innovative, the core contribution can be seen as an incremental improvement that combines existing ideas rather than presenting a fundamentally new framework\n\nQUESTIONS:\n- Can you provide further empirical analysis supporting why the linear phase-in for the diffusion loss is better than simpler schedules (e.g., constant weighting)? Would adaptive or learned schedules yield further gains?\n- What failure modes might occur when the synthetic auxiliary modality is too dissimilar from the primary data? For example, could this lead to misaligned representations that degrade generation quality?\n- how does REED perform when using lightweight or less accurate pretrained representations? This is crucial for understanding the method's generalizability\n- how sensitive is REED to the differences between the pretrained representation space and the diffusion model's latent space?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified theoretical framework for integrating pretrained representations into diffusion model training, enabling flexible guidance via multimodal and hierarchical alignment. The proposed method, REED, demonstrates strong improvements in sample quality and training speed across diverse tasks including image generation, protein design, and molecule generation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper develops a rigorous and general variational framework for understanding and designing representation-guided diffusion models. This subsumes and explains prior works (e.g., REPA, RCG) as special cases, and provides new design insights (e.g., multi-latent structures, flexible scheduling).\n\n2. The proposed approach goes beyond simple alignment with a single pretrained representation. It supports multi-level, multimodal representations (e.g., combining image and text embeddings, or structure and sequence in proteins), and shows empirically that this leads to better controllability and generation quality.\n\n3. The paper motivates and validates a practical curriculum that introduces representation learning loss early and phases in diffusion loss, leading to faster convergence and improved stability.\n\n**Weakness:**\n\n1. While the equivalence to the RCG method is clearly established through the variational bound and decomposed probabilistic framework, the connection to REPA is less explicit and lacks rigorous justification.\n\n2. The paper would benefit from more comprehensive ablation experiments exploring how different choices of the curriculum schedules $\\alpha(n)$ and $\\beta(n)$ (which control the relative weighting of diffusion and representation alignment losses) affect model performance. Currently, the ablation on these schedules is limited; it remains unclear how sensitive the proposed method is to the selection or annealing strategy of these weights, and whether adaptive or learned schedules might further improve results.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}