Title: Boosting Generative Image Modeling via Joint Image-Feature Synthesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion models

Representation learning
Abstract: Latent diffusion models (LDMs) dominate high-quality image generation, yet integrating representation learning with generative modeling remains a challenge. We introduce a novel generative image modeling framework that seamlessly bridges this gap by leveraging a diffusion model to jointly model low-level image latents (from a variational autoencoder) and high-level semantic features (from a pretrained self-supervised encoder like DINO). Our latent-semantic diffusion approach learns to generate coherent image-feature pairs from pure noise, significantly enhancing both generative quality and training efficiency, all while requiring only minimal modifications to standard Diffusion Transformer architectures. By eliminating the need for complex distillation objectives, our unified design simplifies training and unlocks a powerful new inference strategy: Representation Guidance, which leverages learned semantics to steer and refine image generation. Evaluated in both conditional and unconditional settings, our method delivers substantial improvements in image quality and training convergence speed, establishing a new direction for representation-aware generative modeling.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel generative paradigm named joint image-feature synthesis ReDi. Inspired by REPA , this paradigm first performs understanding (feature synthesis) followed by synthesis (image synthesis) within the backbone's forward pass. The significance of this method lies in two aspects: (1) Its novelty may provide valuable insights for designing future unified understanding and synthesis algorithms , and (2) it has the potential to enhance the model's generalization capabilities and training speed.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**:\n\n1. The greatest strength of this paper lies in its highly interesting and novel algorithm. Additionally, its proposed representation guidance also offers valuable insights.\n\n2. This paper is written in a very logical manner and the experiments are adequate and sound.\n\n3. The experimental results are outstanding; ReDi outperforms REPA in both model performance and training speed.\n\n**Weakness**:\n\n1. The paper lacks some analytical experiments, such as: the FID change curve during training, a visualization of the \"synthesized\" representation's evolution during the sampling process, and a comparison of the \"synthesized\" representations between REPA and ReDi to better explain why ReDi performs better.\n\n2. In addition to DINOv2, the authors should explore other representations, such as MAE, LLM Embeddings (e.g., Gemma and T5 Embedding) corresponding to prompts, or even representations from a REPA pre-trained diffusion model.\n\n3. The statement \"This suggests an optimal intermediate subspace where compressed visual features retain sufficient expressivity to guide generation without dominating model capacity\" only provides an intuitive explanation of the experimental results. Is there a more in-depth explanation regarding Figure 7?\n\nQUESTIONS:\nNo.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

comments: SUMMARY:
The paper proposes to jointly model the image latents and semantic features (e.g., DINO) within the same diffusion process. During inference, it introduces representation-guided sampling to leverage the semantic features. Experimental results show that the proposed ReDi improves generative quality for both DiT and SiT, and achieves faster convergence during training.

STRENGTHS AND WEAKNESSES:
## **Strengths**
- The motivation of the proposed method is well-explained. In the introduction it discusses the gap in discriminative capabilities between LDMs and representation learning methods. 
- Compared to the existing work with similar motivation  e.g., REPA that achieves the semantic alignment via distillation, the paper proposes a new direction that encourages diffusion models to explicitly learn the joint distribution of the VAE latents and semantic features. It could offer new insights to the research community.
- The experiment results show the superiority of the proposed method

## **Weaknesses**
- There is no analysis of the generated DINO features. For example, it would be good to demonstrate their discriminative capabilities via linear probing results. In Figure 5, there are qualitative examples of the generated DINO features; however, it would be better to compare them with the ground truth DINO features as well.
- There is no experimental result with higher resolution (512 x 512) input. It would be good to show the proposed method can also effectively synthesize the semantic features and improve the generative quality of higher resolution inputs.

QUESTIONS:
- How about the speed? The sampling speed might be slower than the baselines due to the extra prediction of the DINO feature?

LIMITATIONS:
Yes, limitation of impact discussed in Appendix.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper presents a joint diffusion training approach using low-level VAE representations and high-level DINOv2 representations, called **ReDi**.\n- The joint training significantly improves the resulting diffusion model across the board—for both conditional and unconditional generation. The results are in the same vein as REPA (Yu 2024), but ReDi achieves even greater improvements. Additionally, joint training speeds up convergence substantially.\n- The paper also introduces a sampling-time self-guidance trick called **Representation Guidance**, where the guidance signal comes from the denoising high-level DINOv2 representation. This signal is amplified in a way similar to classifier-free guidance (CFG), and the method effectively narrows the performance gap between conditional and unconditional generation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The improved convergence speed and convergence quality have practical significance. Achieving **23× faster convergence** and **cutting FID in half** is a very strong result—this alone is worthy of publication.\n- In addition to the substantial FID improvement, the idea of Representation Guidance raises intriguing research questions (see Weaknesses).\n\n**Weaknesses:**\n\n- There is limited investigation into *why and how* joint modeling with DINO representations leads to improvements. Although REPA observed that aligning a diffusion model’s representation with DINO helps, a deeper mechanistic understanding is lacking. ReDi’s approach is also structurally different from REPA, which warrants a fresh explanation.\n    - Relatedly, Figure 7 shows that increasing the number of DINO channels can degrade performance. The paper suggests this happens because the model devotes too much capacity to modeling the DINO representation. While plausible, the experiments may not be well-controlled enough to support that conclusion. For instance, the loss weighting should be adjusted to maintain a constant ratio between the VAE and DINO losses. Without this control, it’s difficult to draw strong conclusions.\n    - Even with such control, an open question remains: different DINO channels may encode fundamentally different types of information, and perhaps only some of that information is beneficial for diffusion. The paper too quickly attributes the observed degradation to limited capacity, without adequately exploring whether the nature—not just the volume—of the DINO information plays a role.\n- There is also limited analysis of why **Representation Guidance** (a form of self-guidance) is effective. It’s curious that a signal generated by the model itself—when amplified—can lead to better diffusion outcomes. Understanding this phenomenon might uncover important new insights.\n\nQUESTIONS:\n- I’m personally curious about *why and how* jointly modeling “more stuff” (like DINO, which should make the task harder) actually leads to **faster convergence**. More broadly, I’d like to understand how joint modeling in general helps convergence.\n    - To answer these questions, it would be useful to test different joint representation choices. For example, one potential baseline is to jointly model with the **class label** itself.\n        - My hypothesis is that class labels might outperform DINO representations (though I could be wrong). If they do, it might suggest that the advantage comes from the sharpness of the conditioning signal.\n        - This would also shed light on why Representation Guidance helps: it could serve as a “sharpened” signal that accelerates denoising, akin to conditional generation. While it doesn’t provide a strong condition from the beginning, it offers a slightly stronger guide during sampling, enhancing overall generation quality.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new technique, called ReDi, to improve the training and inference of generative diffusion models for images.\nThe paper places itself in the context of transformer latent diffusion.\nThe main idea of the paper is to learn to noise/denoise both the image latent and a secondary image representation derived from a pre-trained DINOv2 model.\nThe secondary image representation acts as regularizer like is commonly done in self-supervised learning.\nFurthermore this secondary objective yields an additional signal for guidance during inference which the authors coin adequately as representation guidance.\nExperiments on ImageNet256 demonstrate an improvement over REPA, in particular when NOT using Classifier-Free Guidance (CFG).\nWhen using CFG, there's little room for improvement in the first place for ImageNet256, but the proposed method still manages to show a little improvement for class-conditional generation and a significant one for unconditional generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is easy to read and clear\n2. The method looks simple to implement\n3. The experiments show convincing gains over REPA\n4. Conceptually, I can see this applying to other domains than images but somehow this does require a (good) secondary representation to be available for the target domain.\n\nWeaknesses\n1. Experimental setting limited to ImageNet256\n2. Secondary representation limited to one choice (DINOv2), it would have been interesting to try several alternatives for secondary representations.\n3. Strength (4) raises the question of what is a good secondary representation, which in turn points back to weakness (2), this discussion is lacking in the paper \n4. Consequently, the paper feels more like an engineering a solution to natural image generation that probing the underlying workings of diffusion models.\n\nQUESTIONS:\nI do not have questions, my only suggestion would to dig deeper on trying various secondary representations to help uncover what a good representation should have.\n\nLIMITATIONS:\nI didn't see any discussion on limitations or potential negative social impact. \nI would assume that it shares those with generative modeling in general, e.g. that ultimately could allow to generate fake data that is indistinguishable from reality.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}