Title: Projection-Manifold Regularized Latent Diffusion for Robust General Image Fusion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image fusion

general image fusion

multimodal image fusion

latent diffusion models
Abstract: This study proposes PDFuse, a robust, general training-free image fusion framework built on pre-trained latent diffusion models with projection–manifold regularization. By redefining fusion as a diffusion inference process constrained by multiple source images, PDFuse can adapt to varied image modalities and produce high-fidelity outputs utilizing the diffusion prior. To ensure both source consistency and full utilization of generative priors, we develop novel projection–manifold regularization, which consists of two core mechanisms. On the one hand, the Multi-source Information Consistency Projection (MICP) establishes a projection system between diffusion latent representations and source images, solved efficiently via conjugate gradients to inject multi-source information into the inference. On the other hand, the Latent Manifold-preservation Guidance (LMG) aligns the latent distribution of diffusion variables with that of the sources, guiding generation to respect the model’s manifold prior. By alternating these mechanisms, PDFuse strikes an optimal balance between fidelity and generative quality, achieving superior fusion performance across diverse tasks. Moreover, PDFuse constructs a canonical interference operator set. It synergistically incorporates it into the aforementioned dual mechanisms, effectively leveraging generative priors to address various degradation issues during the fusion process without requiring clean data for supervising training. Extensive experimental evidence substantiates that PDFuse achieves highly competitive performance across diverse image fusion tasks. The code is publicly available at https://github.com/Leiii-Cao/PDFuse.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training-free and unified framework, named PDFusion, which incorporates the diffusion posterior sampling process into a pre-trained latent diffusion model. It introduces MICP and LMG to guide the generation process, aiming to improve fidelity and quality. Additionally, the framework claims the ability to handle degraded inputs inherently. Extensive experiments across various fusion scenarios and downstream tasks demonstrate the method's robustness.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The motivation is novel. PDFusion leverages a pre-trained latent diffusion model to guide the fusion process in the latent space.\n- The use of a canonical interference operator set allows PDFusion to handle various types of degradation and produce higher-quality fusion results.\n\n### Weaknesses\n\n- PDFusion is based on Stable Diffusion but is guided at the pixel level. In other words, the fusion occurs at the pixel level and is subsequently encoded into the latent space. This approach undermines the advantage of latent-space fusion, making it fundamentally similar to conventional DDPM approaches.\n- Table 1 reports an unusually large performance gain for PDFuse-E in metrics such as EI, SF, and AG. It is difficult to understand how degraded inputs, which have lost information, can produce outputs of higher quality than those generated from clean inputs. This discrepancy raises concerns regarding the credibility and reliability of the reported results.\n- While the method is described as training-free, it remains unclear whether the de-degraded outputs are guided using clean inputs. If so, this undermines the validity of the \"training-free\" claim. If not, it is unclear how the method ensures clean fusion outputs under pixel-level constraints with degraded inputs.\n- The paper does not clearly explain how the datasets $\\mathcal{D}^s$ and $\\mathcal{D}^d$ are constructed or obtained.\n- Every generation step includes a decoding operation. Given this, it is difficult to understand how PDFusion achieves faster runtime than DDFM and CCF, as reported in Table S6.\n\n>  If the authors can clarify my concerns, I would be inclined to significantly raise my score.\n\nQUESTIONS:\n- Empirically, the pre-trained VAE is unable to generate high-quality visible and infrared fused images, as it has never been exposed to such data during training. It remains unclear how the model is guided in the latent space and how it decodes these representations to synthesize high-fidelity fusion images. Additionally, Appendix A.3 is difficult to follow: the algorithm references Equations (10)–(15), but these equations are either missing or poorly integrated, resulting in a presentation that appears chaotic and incomprehensible.\n- The performance of PDFuse-E appears questionable, both in terms of underlying principles and the reported metric values.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PDFuse, a training-free image fusion framework built on pre-trained latent diffusion models with a novel projection–manifold regularization strategy. \n\nBy formulating fusion as a constrained diffusion inference process, it effectively integrates multiple source images to generate high-fidelity results. \n\nTwo core mechanisms, Multi-source Information Consistency Projection (MICP) and Latent Manifold-preservation Guidance (LMG), ensure both consistency with source inputs and alignment with the generative prior. \n\nExtensive experiments show that PDFuse delivers competitive performance across diverse image fusion tasks without requiring supervised training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The submission is complete, and the writing is generally clear and accessible to readers.  \n- The method is training-free, which reduces computational cost and simplifies deployment.  \n- Some quantitative results demonstrate the potential of the proposed approach.\n\n**Weaknesses**\n- The paper emphasizes the role of multiple source images (e.g., `i_m`, `i_n`), yet the visual diagrams (e.g., **Figure 1** or **Figure 2**) do not illustrate how these sources interact with the fusion process, making the method less intuitive to understand.  \n- The qualitative results are not convincing, **Figures 3, 5, and 6** do not clearly show visual improvements from the proposed method, and **Figure 5** appears unexplained or unreferenced in the main text.  \n- The paper feels overloaded with tables and figures, but lacks clear insights or takeaways. The **ablation study** section is shallow and does not offera  meaningful understanding of the proposed components or their contributions.  \n- In **Table 2**, comparisons to baselines like **EAT (TMM’25)** are incomplete due to missing metrics, making it difficult to assess the true advantage of the proposed method.\n\nQUESTIONS:\n### Questions for the Authors\n\n**Q1:** Lines 61–62 state that \"PDFuse constructs a canonical interference operator set.\"  \n- What exactly is meant by a *canonical interference operator set* in this context?  \n- Why is this set necessary, as mentioned again in Line 173?  \n- Please clarify its role in the method and consider adding an ablation or analysis to demonstrate its impact.\n\n**Q2:** The paper heavily emphasizes fusion from multiple source images (e.g., `i_m`, `i_n`), but **Figure 1** and **Figure 2** do not visually explain how these sources are processed or interact during inference.  \n- Can the authors revise or augment the diagrams to clearly depict the role and flow of multiple inputs?\n\n**Q3:** The qualitative results in **Figures 3, 5, and 6** do not convincingly show improvements of the proposed method.  \n- Could the authors enhance the visualization quality and include zoomed-in or region-focused comparisons to better highlight differences?  \n- Also, **Figure 5** is not referenced or discussed in the text—please explain its purpose and relevance.\n\n**Q4:** The paper includes many tables and figures, but lacks clear conceptual takeaways or technical insights.  \n- Can the authors restructure the results section to highlight key observations and explain what we learn from each experiment, rather than simply reporting numbers?\n\n**Q5:** The ablation study is quite shallow and lacks detailed analysis.  \n- Can the authors expand this section to better isolate the contributions of individual components (e.g., MICP, LMG, interference operators) with clear comparisons?\n\n**Q6:** In **Table 2**, the comparison with **EAT (TMM’25)** is inconclusive due to missing metrics.  \n- Can the authors either reproduce the missing metrics for a fair comparison or discuss this limitation more clearly to help contextualize their method’s relative performance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes **PDFuse**, a training-free, general-purpose, and degradation-robust image fusion framework based on pretrained latent diffusion models. The key idea is to reformulate image fusion as a posterior sampling process guided by two regularization mechanisms:\n- **MICP (Multi-source Information Consistency Projection)**: injects multi-source image content via a two-stage projection strategy.\n- **LMG (Latent Manifold-preservation Guidance)**: enforces generative consistency by aligning latent variables with the manifold prior.\n\nTo further enhance robustness, a set of degradation operators is embedded into both mechanisms, allowing PDFuse to adapt to various degradation scenarios. Extensive experiments across diverse fusion tasks (IVF, MEF, MFF) and semantic segmentation verify the superiority and robustness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. **Well-engineered and effective framework**  \n   The alternating design of MICP and LMG offers a principled mechanism to balance fidelity and generative quality, and integrates seamlessly into latent diffusion inference.\n\n2. **Training-free and broadly applicable**  \n   The method does not require retraining and generalizes across tasks and degradations, making it practically useful. The integration of an interference operator set further improves robustness.\n\n3. **Comprehensive experiments**  \n   Strong empirical results on multiple tasks, supported by ablation studies and semantic-level verification, demonstrate the method's effectiveness and generalization.\n\n---\n\n### **Weaknesses**\n\n1. **Limited theoretical novelty**  \n   The approach is largely driven by system design and heuristics. There is no formal analysis of the convergence, optimality, or theoretical behavior of the alternating regularization scheme.\n\n2. **Somewhat heuristic projection and loss design**  \n   The two-stage MICP and the handcrafted LMG loss rely on pre-defined forms (e.g., max intensity, Laplacian regularization), which may limit adaptability beyond the chosen tasks.\n\n3. **Lack of analysis on degradation generalization**  \n   The degradation operator set is conceptually flexible, but it’s unclear how well it handles unseen or compound degradations at test time.\n\nQUESTIONS:\n- How does the method scale to fusion tasks involving more than two source images?\n- Can the proposed degradation operators generalize to unseen types or compositions?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training-free image fusion framework built on a pre-trained latent diffusion model. This framework presents two regularization mechanisms: Multi-Source Information Consistency Projection (MICP) and Latent Manifold-preservation Guidance (LMG). Then, the unified image fusion modeling is realized through the alternating regularization of the diffusion process by these two regularization mechanisms. Additionally, the method constructs a set of interference operators and integrates them into the two regularization mechanisms to tackle various degradations and interferences. They provide extensive experiments to verify this work’s effectiveness across multiple fusion scenarios and under complex interference conditions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The idea of formulating the image fusion process as a latent diffusion process regularized by multi-source image conditions is novel.\n+ Designed two regularization mechanisms, MICP and LMG, to constrain the latent diffusion process, universally applicable to various image fusion tasks without requiring training. Meanwhile, a set of interference operators is constructed, which can be integrated into the regularized mechanisms to further enhance the overall robustness.\n+ The authors conduct extensive evaluations, proving the advanced performance of this work.\n+ The code is provided in the supplementary material, which is commendable.\nWeaknesses：\n- In the Multi-source Information Consistency Projection mechanism, the authors divide the optimization into two stages: the coarse structural component $\\mathcal{D}^s(\\hat{\\mathbf{z}}_0^t)$ and the fine detail component $\\mathcal{D}^d(\\hat{\\mathbf {z}}_0^t)$. However, it is puzzling that the subsequent equations optimize $\\mathcal{D}^s$ and $\\mathcal{D}^d$ without clearly defining their relationship.\n- After integrating the degradation operator $\\mathbf{\\Pi}$, the derivation from Equation (12) to Equation (16) is not entirely clear to me. Would it be possible to provide a more detailed explanation of this part?\n- When different source images encounter different degradation interferences, how to construct $\\mathbf{\\Pi}$ from $\\mathbf{\\Pi}_i$ and $\\mathbf{\\Pi}_j$. Although the authors provide a brief explanation in the supplementary materials, it would be helpful if more concrete details could be provided.\n- In the experimental section, the paper presents comparative results for both the base and enhanced versions of the method in infrared–visible image fusion, demonstrating promising performance; however, for multi-exposure and multi-focus image fusion, only the base version (PDFuse-B) results are reported.\n- The author conducted ablation studies on the MICP and LMG mechanisms, if further provided the regularization effects of these two mechanisms at each step of the latent diffusion process would offer a more intuitive demonstration.\n- Although the method achieves good results, its runtime is still relatively long (despite improvements in efficiency compared to similar diffusion-based methods).\n\nQUESTIONS:\n- Is this method entirely training-free, or does it still require fine-tuning? Please clarify this point.\n- Why are the values set to 0 and $\\hat{\\bm{z}}_0^t$ respectively in Equation (13) during the alternating optimization setup?\n- The paper only considers two source images as input. Is the method adaptable to more input images, such as a sequence of three images with different exposure settings for multi-exposure image fusion?\n- Some minor issues include: Table 3 lacks ↑ or ↓ indicators for the metrics, and some experimental details, such as the diffusion step size, are missing.\n- Besides, please address the concerns in the Weakness part.\n\nLIMITATIONS:\nThe authors discuss the limitations in the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}