DRMF: Degradation-Robust Multi-Modal Image Fusion via Composable Diffusion Prior

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 15:19:45

Abstract: Existing multi-modal image fusion algorithms are typically designed for high-quality images and fail to tackle degradation (e.g., low light, low resolution, and noise), which restricts image fusion from unleashing the potential in practice. In this work, we present Degradation-Robust Multi-modality image Fusion (DRMF), leveraging the powerful generative properties of diffusion models to counteract various degradations during image fusion. Our critical insight is that generative diffusion models driven by different modalities and degradation are inherently complementary during the denoising process. Specifically, we pre-train multiple degradation-robust conditional diffusion models for different modalities to handle degradations. Subsequently, the diffusion priori combination module is devised to integrate generative priors from pre-trained uni-modal models, enabling effective multi-modal image fusion. Extensive experiments demonstrate that DRMF excels in infrared-visible and medical image fusion, even under complex degradations.

Main paper: 
 introduction: 
 Multi-modal image fusion (MMIF) is an essential image enhancement technique that aggregates significant information from diverse sensors or modalities for a comprehensive scene representation [13] . Infrared-visible image fusion (IVIF) and medical image fusion (MIF) are the most representative MMIF tasks. IVIF aims to merge essential thermal radiation information from infrared (IR) images and rich texture from visible (VI) images. The fusion results can overcome the limitations of IR images suffering from noise and low resolution as well as VI images affected by illumination and camouflage [60] . MIF seeks to integrate functional and metabolic information in functional images with structural and anatomical information in structural images into a single image, aiding in subsequent disease diagnosis and treatment [7] . Sufficient information aggregation and pleasing visual outcomes enable MMIF to be widely applicable across various domains, such as intelligent medical service [67] , nighttime assisted driving [1] , object detection [10] , and semantic segmentation [58] .In recent years, MMIF has gained increasing attention, and numerous algorithms have been developed to achieve greater visual appeal. Owing to powerful feature extraction and representation abilities, auto-encoder (AE)-based [16, 17, 21, 63] , convolutional neural network (CNN)-based [27, 40, 51, 62] , generative adversarial network (GAN)-based [20, 28, 29] and Transformer-based [26, 44, 57] frameworks dominate the MMIF field. In addition, the diffusion model [8, 35] with strong distribution modeling and generative abilities shows great potential for MMIF [55, 64] . DDFM [64] migrates the generative prior of the pre-trained large-scale natural image diffusion model to MMIF using the score matching technique [38] . However, the implementation of score matching involves tedious design and requires re-optimizing an objective function for each specific scenario, making it time-consuming. Moreover, the distribution variation between natural and multi-modal images limits the performance of pre-trained diffusion models in MMIF.Additionally, most existing fusion methods are tailored for normal scenarios. As illustrated in Figure 1 (I), the VI image in a realistic night scene experiences severe illumination degradation. Consequently, even the advanced diffusion model-based fusion method (i.e., DDFM) struggles to represent the scene information effectively. Several works attempt to simultaneously enhance and aggregate complementary information in challenging scenarios [40, 48] . DIV-Fusion [40] first leverages the Retinex theory to extract reflectancerelated and illumination-related features from VI images. Then, it only integrates reflectance-related features in the subsequent fusion stage, mitigating the negative impact of illumination degradation. Nevertheless, since the illumination component is completely dropped, the fusion result of DIVFusion, shown in Figure 1 (I (d )), appears unnatural and suffers from noticeable color distortion. Note that source images in challenging scenarios may be affected by multiple degradations, including noise, low contrast, low resolution, low light, etc. As illustrated in Figure 1 (II), existing fusion methods struggle to effectively address composite degradation, leading to fusion results that exhibit noise and blurriness, which greatly restrict the practical application of MMIF. It is worth mentioning that some preprocessing techniques (e.g. DiffLL [12] for low-light image enhancement and CAT [2] for denoising as well as super-resolution) can be utilized to pre-enhance source images. Nevertheless, the poor coupling between multiple cascading tasks leads to error amplification and thus synthesizes unsatisfactory fusion results, as presented in Figure 1 (I (c )) and (II (c)).To overcome the above challenges, we propose a degradationrobust fusion method based on composable diffusion prior, termed DRMF, which is a novel diffusion-based paradigm for MMIF. DRMF fully leverages the inherent complementary potential of diffusion models driven by different modalities and degradations to mitigate various degradations. We begin by pre-training multiple degradationrobust conditional diffusion models, which are conditioned on degraded source images to estimate the distribution of corresponding high-quality images. Since generative diffusion priors are characterized by Gaussian noise, the priors of various diffusion models can be combined in the noise space. Specifically, during the sampling process, given the fusion sample 𝑥 𝑓 𝑡 at 𝑡-step, the diffusion model 𝜖 𝑖 𝜃 driven by 𝑖-modality is employed to estimate the noise (i.e., generative prior) 𝑛 𝑖 𝑡 to approximate high-quality revision of 𝑖-modality. Then, the diffusion priori combination module estimates the combination weights of different modality-driven diffusion priors 𝑛 𝑖 𝑡 and produces the next sample 𝑥 𝑓 𝑡 -1 . Gradually, DRMF can yield a high-quality fused image conditioned on degraded source images. As shown in Figure 1 , DRMF effectively addresses various complex degradations and generates impressive fusion results that emphasize significant targets and vividly portray scene details. To sum up, our major contributions are threefold:-We propose a novel diffusion model-based framework for MMIF, which harnesses the powerful generative properties of diffusion models to mitigate various complex degradations in source images. -We develop a diffusion prior combination module to aggregate generative diffusion priors from different modalities, effectively leveraging the complementary nature of diffusion models driven by various modalities and degradations. -Extensive experiments demonstrate the superiority of our method in the IVIF and MIF tasks, especially when source images are afflicted by composite degradations. 
related work: 
 Multi-modal image fusion. The booming development of deep learning has energized the field of MMIF. AE [16, 65] , CNN [51, 62] , GAN [20, 29] , and Transformer [26, 44] are the major network architectures used in deep learning-based MMIF methods. In response to the practical demands of subsequent high-level vision tasks, the image fusion community has proposed numerous semanticdriven approaches [20, 22, 41, 43] , which constrain the fusion network to retain richer semantic cues. Additionally, RFNet [53] , UMF-CMGR [47] , SuperFusion [39] , and MURF [54] jointly model image registration and fusion tasks, correcting the parallax and aberration in actual shooting. Furthermore, some universal methods [18, 26, 52, 56] uniformly fulfill various image fusion tasks. Specifically, realizing that source images are usually affected by degradation, Tang et al. [40] and Wang et al. [48] devised illumination-robust fusion approaches, which take into account low-light image enhancement and information fusion. However, these methods are specifically designed for illumination degradation and fail to address other common degradations, such as noise and low resolution. In this work, we utilize the diffusion models with powerful generative abilities to tackle the challenges of composite degradation. Diffusion model. Recently, the diffusion model [8] has gained widespread attention within the computer vision community, which samples high-quality images from the estimated target distribution. Benefiting from their powerful generative capabilities, diffusion models have found extensive applications in various lowlevel visual tasks, such as text-to-image generation [3, 35] , image manipulation [15] , super-resolution [36] , de-raining [30] , deblurring [50] , and low-light image enhancement [5] , consistently delivering remarkable results. In addition, composable diffusion models [9, 15, 23] are proposed to exploit multi-modality conditions for controlling the generation process more accurately. The diffusion model is also introduced to the field of MMIF. Dif-Fusion [55] for the first time employs the denoising network of the diffusion model as a potent feature extractor, while the whole image fusion process is detached from the diffusion model. DDFM [64] combines the score matching and a pre-trained unconditional diffusion model to transfer the generative prior learned from natural images to multi-modal images for image fusion. Notably, existing diffusion model-based methods are not specifically designed for the challenges in MMIF. As a result, the remarkable generative abilities of diffusion models are not unleashed. In this work, we design taskspecific conditional diffusion models tailored to cope with various degradations in multi-modal images. Then, we propose a diffusion prior combination module to achieve image fusion using generative priors driven by different modalities and degradations. 
method: 
 Given complementary degraded source images {𝑥 𝑖 𝑐 } where 𝑖 ∈ {𝑖𝑟, 𝑣𝑖, 𝑐𝑡, 𝑚𝑟𝑖}, our method starts from random Gaussian noises 𝑥 𝑇 and generates high-quality fused images 𝑥 𝑓 by step-by-step denoising conditional on {𝑥 𝑖 𝑐 }. We devise a diffusion prior combination module to combine the noises {𝑛 𝑖 𝑡 } from multiple diffusion models {𝜖 𝑖 𝜃 } and infer the next fusion sample during the denoising process. In particular, the estimated noises characterize the degradation-robust diffusion priors driven by different modalities and degradations, thereby we can achieve multi-modality information aggregation. Our sampling process adheres to the standard conditional diffusion model. Therefore, in the following we present the typical diffusion model, degradation-robust conditional diffusion models, and diffusion prior combination modules in sequence.Denoising diffusion probabilistic model [8] is a classical generative model that utilizes a Markov chain to transform complex data distributions into a simple Gaussian noise distribution. It then recovers the desired data distribution from the noise distribution through step-by-step denoising. As shown in Figure 2 , in the forward process, the given data distribution 𝑥 0 ∼ 𝑝 (𝑥 0 ) is progressively corrupted through a pre-defined Marchkov chain. This is accomplished by gradually introducing Gaussian noise, and the pre-defined Markov chain satisfies the following distribution:EQUATIONwhere 𝛽 𝑡 governs the noise variance added at 𝑡-step and 𝛼 𝑡 = 1 -𝛽 𝑡 .Thus the marginal distribution is derived as:EQUATIONwhere ᾱ𝑡 = Π 𝑡 𝑖=1 𝛼 𝑖 and β𝑡 = 1 -ᾱ𝑡 . As 𝑡 approaches a large value 𝑇 , ᾱ𝑇 tends to 0 and 𝑞(𝑥 𝑇 |𝑥 0 ) approximates the normal distribution N (0, 𝐼 ), the forward process is finished.The reverse process starts with a Gaussian noise 𝑥 𝑇 ∼ N (0, 𝐼 ) and progressively denoises to generate a clean image through a Markov chain, defined as:EQUATIONAs presented in [8] , the mean 𝜇 𝜃 (𝑥 𝑡 , 𝑡) is calculated based on estimated results of the noise estimation network 𝜖 𝜃 with parameters 𝜃 , and the variance 𝜎 2 𝑡 is a time-dependent constant. Specifically, the above processes are formulated as:EQUATIONUltimately, the optimization objective is defined as:EQUATIONwhere 𝑛 𝑡 is sampled from a standard Gaussian noise.Denoising diffusion implicit model [37] presents an accelerated deterministic sampling manner for pre-trained diffusion models, where the forward process follows a non-Markovian process. The marginal distribution 𝑞(𝑥 𝑡 |𝑥 0 ) still obeys Eq. ( 2), and the mean in Eq. ( 3) is re-formulated as:EQUATIONwhere x0 =𝑥 𝑡 - √ 1-ᾱ𝑡 𝜖 𝜃 (𝑥 𝑡 ,𝑡 ) √ ᾱ𝑡indicates the estimated 𝑥 0 at the current step, and 𝜆 𝑡 is a tunable hyper-parameter controlling the variance of the marginal distribution. Thus, 𝜎 2 𝑡 in Eq. ( 3) is set to 𝜆 2 𝑡 .Conditional diffusion models have been shining in the field of image generation [3, 35] , enhancement [5] , and degraded image restoration [30, 36, 50] . However, diffusion models in MMIF are primarily employed to extract features [55] and provide routine generative priors of natural images [64] without tackling degradations. In this work, we first present degradation-robust conditional diffusion models (DRCDMs) to restore degraded source images. As shown in Figure 3 , given the degraded image 𝑥 𝑖 𝑐 and the high-quality versions 𝑥 𝑖 0 , the forward process in the training stage follows Eq. ( 1), i.e.,𝑞(𝑥 𝑖 𝑡 |𝑥 𝑖 𝑡 -1 ) = N (𝑥 𝑖 𝑡 ; √ 𝛼 𝑡 𝑥 𝑖 𝑡 -1 , 𝛽 𝑡 𝐼 ).The reverse process is defined as𝑝 𝜃 (𝑥 𝑖 𝑡 -1 |𝑥 𝑖 𝑡 , 𝑥 𝑖 𝑐 ) = N (𝑥 𝑖 𝑡 -1 ; 𝜇 𝜃 (𝑥 𝑖 𝑡 , 𝑥 𝑖 𝑐 , 𝑡), 𝜎 2 𝑡 𝐼 ), where the posterior is further conditioned on degraded image 𝑥 𝑖 𝑐 as:EQUATIONSpecifically, the intermediate sample 𝑥 𝑖 𝑡 is concatenated with the degraded image 𝑥 𝑖 𝑐 and fed into the noise estimation network 𝜖 𝑖 𝜃 , where the degraded image plays a crucial role in providing valuable semantic and structural information. Furthermore, the optimization objective is modified as: During the inference stage, high-quality source images are generated via iterative denoising, starting from a standard Gaussian noise and guided by degraded conditional images. Note that in our solution, DRCDMs not only recover high-quality images from degraded ones but also offer degradation-robust generative diffusion priors, which are employed to yield high-quality fused images directly.EQUATION)After pre-training, DRCDMs have learned the prior distributions of high-quality uni-modal images. The priors are determined by noises predicted at each sampling step. Related works [9, 15, 23] confirm that combining noises from multiple diffusion models can generate a result containing corresponding prior distributions. Therefore, to directly generate a fused image that carries characteristics from high-quality uni-modal priors, we devise a diffusion prior combination module (DPCM) to flexibly merge complementary noises. In detail, as shown in Figure 3 , we invert the image from random Gaussian noise with pre-trained DRCDMs and update the mean during the DDIM sampling process according to the rule:EQUATIONwhere x𝑖0 𝑡 = 𝑥 𝑓 𝑡 - √ 1-ᾱ𝑡 𝜖 𝑖 𝜃 (𝑥 𝑓 𝑡 ,𝑥 𝑖 𝑐 ,𝑡 ) √ ᾱ𝑡denotes the estimated sample associated with 𝑖-modality conditioned on 𝑥 𝑖 𝑐 from the fusion sample at 𝑡-step. {𝛾 𝑖 𝑡 } 𝑇 𝑡 =1 is the sequence weights for each generative diffusion prior satisfying 𝑖 𝛾 𝑖 𝑡 = 1. In this way, the goal of DPCM is to estimate sequence weights for each modality. The learnable weights need to take into account several considerations. They 1) should measure the complementary properties of different modalities, 2) are expected to be learned from high-quality images, 3) are modulated by timestep 𝑡, and 4) are guided by the weight from the previous step. Therefore, the process of generating weights with a U-Net 𝜑 is defined as:EQUATIONThe current fusion sample 𝑥 𝑓 𝑡 , the predicted high-quality sample x𝑖 0 𝑡 for 𝑖-modality, and the weight 𝛾 𝑖 𝑡 +1 from the previous step are concatenated and fed into the U-Net. Calculating L by Eq. ( 11), Eq. ( 12) and Eq. ( 13);13Taking gradient descent step on ∇ 𝜑 L;14 until converged;We also devise corresponding losses to optimize the weight prediction network 𝜑. The fusion loss is defined as:EQUATIONwherex 𝑓 0 𝑡 = 𝑥 𝑓 𝑡 - √ 1-ᾱ𝑡 𝑖 𝛾 𝑖 𝑡 𝜖 𝑖 𝜃 (𝑥 𝑓 𝑡 ,𝑥 𝑖 𝑐 ,𝑡 ) √ ᾱ𝑡denotes a clean fused image predicted at 𝑡-step and x𝑖 0 is the high-quality source image restored by the pre-trained DRCDM. The max(•) means the maximum aggregation strategy and ∇ represents the gradient operator [41, 43] . As the denoising proceeds (i.e., as 𝑡 decreases), the estimated fusion results get closer to the desired ones, thereby we weight the fusion loss with time-dependent coefficient √ 𝛼 𝑡 . In addition, we introduce a regularization term to constraint the smoothness of the combination weights 𝛾 𝑖 𝑡 , which is defined as:EQUATIONThe full objective function for 𝜑 is a weighted sum of L 𝑓 and L 𝑟𝑒 :EQUATIONIn particular, the IVIF task is expected to retain salient objects in IR images. To achieve this, we design an additional mask-guided loss, utilizing the salient target mask 𝑚 to guide 𝛾 𝑖𝑟 𝑡 learning, as defined:EQUATIONThe training procedure of DPCM is summarized in Algorithm 1, which uses IVIF as an example and involves only fusion and regularization losses to ensure generality. During the inference stage, we employ DDIM [37] fast sampling and update 𝑥 𝑓 at each step according to Eq. ( 9), iteratively generating high-quality fused images. 
experiments 4.1 implementation details: 
 We train multiple DRCDMs separately. For IVIF, we initially train DRCDMs for VI and IR images on the LOL [49] and MSRS [42] datasets, respectively. LOL consists of paired low-and norm-light images. We down-sample the low-contrast IR images with downsampling factor of 1/4 and add streak noise with 𝜎 2 = 15 to simulate degraded images. The high-quality versions are provided by MSRS. Subsequently, the illumination-robust CDM pre-trained on LOL is used to enhance the VI images in MSRS. Finally, we employ enhanced VI and high-quality IR images from MSRS to train the DPCM, and the salient target mask 𝑚 is in Eq. ( 14) converted from segmentation labels in MSRS. For MIF, we introduce degradation in the form of Gaussian noise (with 𝜎 2 = 25) to MRI images and down-sample CT images using a down-sampling factor of 1/4.During DRCDM training, the batch size is set at 64, the learning rate is set at 2 × 10 -5 , the training iteration is set at 2 × 10 6 , and all images are randomly cropped into 64 × 64. We introduce the attention mechanism at 1/4 resolution for computational efficiency, and other settings adhere to the default settings of diffusion models [30] . When training the DPCM, we employ DDIM deterministic sampling with 𝜆 𝑡 = 0 for generating fused images and set the sampling step to 5. The batch size and training iteration are adjusted to 1 and 1 × 10 5 , and source images are randomly cropped to 128 × 128.We first evaluate the fusion performance of the proposed scheme in practical scenarios on LLVIP [11] , MSRS, and Harvard [45] datasets, which contain 216, 361, and 20 test images, respectively. Then, we utilize degraded images in these datasets to demonstrate the superiority of DRMF in solving composite degradation for MMIF. Moreover, the test set of LOL with 15 image pairs is used to validate the strong performance of DRCDMs in tackling degradation.We first consider practical scenarios, where all source images are captured directly by sensors. DRMF is compared with 8 SOTA methods, i.e., DeFusion [18] , TarDAL [20] , U2Fusion [52] , DIVFusion [40] , LRRNet [17] , MURF [54] , PAIF [25] and DDFM [64] . Visual comparisons for IVIF and MIF tasks are shown in Figures 4 and 5 . In nighttime IVIF scenarios, DRMF effectively lights up information in the dark and enhances the contrast of IR images. Our results thus provide bright backgrounds and significant targets. DIVFusion also lights up VI images, but it suffers from overexposure and color saturation reduction. Other approaches struggle to mine valuable information in the dark or even diminish salient targets. As in Figure 5 , only DRMF completely integrates functional and structural information in CT and MRI images, benefiting from the adaptive aggregation ability of DPCM. Quantitative results using MI [32] , SF [4] , SD [33] , and VIF [6] are displayed in Table 1 . A SOTA lowlight enhancement (LLIE) method (i.e, DiffLL [12] ) is employed to enhance low-light VI images for comparing fusion performance fairly. Orig. denotes the practical fusion results, whereas En. means fusion results with LLIE for VI images. Achieving the best SD and VIF means our method has optimal contrast and visual effects. The superior MI suggests that DRMF effectively transfers valuable information from source images to fusion results. The comparable SF shows that our fusion results integrate abundant textures. Figures 4 and 5 also show visual comparisons of various fusion methods in challenging scenarios with composite degradation. Specifically, VI images experience noise and low light, while IR images endure low contrast, low resolution, and streak noise. CT and MRI images suffer from low resolution and noise, respectively. These degradations are typical challenges in real-life shoots. For fair comparisons, we employ DiffLL to pre-enhance VI images, and fine-tune CAT [2] to denoise VI, IR, and MRI images. CAT is also fine-tuned to super-resolution IR and CT images. As shown in Figures 4 and 5 , our fusion results distinctly present scene information (e.g., abundant textures and prominent objectives in IVIF), even amidst complex image degradation. Additionally, only DRMF can synthesize the cleanest and sharpest fused images in MIF, owing to the exceptional degradation elimination capability of DRCDM. In contrast, other alternatives suffer from the harsh effects of composite degradation, leading to decreased fusion performance. Although employing cascaded pre-enhancement methods can mitigate these effects, it usually amplifies deficiencies in specific tasks, leading to unsatisfactory performance. Table 2 further substantiates the remarkable performance of DRMF in degraded fusion scenarios. Particularly, DRMF exploits the composable property of diffusion models to organically integrate enhancement into fusion tasks, avoiding incompatibility between various tasks and enabling impressive performance. In summary, both qualitative and quantitative experiments on IVIF and MIF demonstrate the superiority of DRMF in inhibiting degradation and aggregating complementary information.Low-light image enhancement. We verify the effectiveness of our DRCDM on low-light image enhancement (LLIE), which is a prominent image restoration task. DRCDM is compared with several mainstream algorithms, including KinD [61] , ChebyLighter [31] , RUAS [24] , CLIP-LIT [19] , CUE [66] , FourLLIE [46] , and DiffLL [12] . A typical example is shown in Figure 6 . It is evident that only KinD, CUE, DiffLL, and DRCDM can faithfully render the scene with natural brightness. Remarkably, DRCDM with powerful generative ability can mine information in the darkness, as highlighted by rectangular boxes. The quantitative assessments, involving PSNR, SSIM, MUSIQ [14] , and LIQE [59] , are presented in Table 3 . The best Pre-enhancement for other fusion methods. Our DRCDM can directly enhance source images as preprocessing for other fusion approaches. Figure 7 displays some representative examples. DRCDM yields high-quality source images for both actual and degraded scenarios. When existing fusion methods take enhanced source images as input, both qualitative and quantitative results show significant improvements. For instance, the background (e.g., fences) in VI images and structure in MRI images are presented more clearly, and the SD and VIF metric are increased remarkably. Object detection. Effective information enhancement and aggregation not only aid in visual perception, but also promote machine vision performance. Thus, we evaluate object detection performance on LLVIP to further reveal the superiority of DRMF, where the re-trained YOLO v8 [34] is employed as the detector. In Figure 8 , DRMF successfully detects all pedestrians in both practical and challenging scenarios. In contrast, source images and other methods suffer from varying degrees of missed detection. The quantitative results in Table 4 show that DRMF balances precision and recall, thereby achieving better average precision (AP). In particular, the best mean AP (mAP) across various IoU thresholds implies that DRMF can adapt to different IoU settings.Degradation-robust conditional diffusion model (DRCDM). DRCDM harnesses the potent generative abilities of diffusion models to eliminate the effects of various degradations. As presented 9 (c), when replace DRCDM with a typical conditional diffusion model, providing the prior of reconstructing inputs, the results are faithful to source images. Specifically, the fused images retain common information in source images but fail to eliminate nasty degradations. As shown in Table 5 , the higher MI means the fusion model without priors provided by DRCDM transfers more information from source images to fused images. Unfortunately, there is a noticeable degradation in textures and visual quality. Diffusion prior combination module (DPCM). DPCM aims to adaptively integrate diffusion priors driven by various modalities, thus preserving crucial information. We perform an ablation study using a simple average weighting strategy instead of DPCM. As depicted in Figure 9 (d), the fusion results tend to average the enhanced source images. Specifically, even with DRCDM providing robust priors, the fusion results still experience degradations due to the inability to adaptively aggregate relevant priors. The descent results in Table 5 also confirm this issue. In contrast, DRMF equipped with both DRCDM and DPCM can effectively cope with complex degradation and retain essential details and targets. 
conclusion: 
 This work presents a degradation-robust multi-modal image fusion framework based on composable diffusion priors. The degradationrobust conditional diffusion models are devised to eliminate unfavorable degradations in source images and offer robust diffusion priors. Furthermore, we develop a diffusion prior combination module to adaptively aggregate diffusion priors driven by various modalities and progressively generate fused images. Experiments on IVIF and MIF demonstrate the superiority of our method in suppressing degradation and aggregating information during both practical and challenging scenarios.  
drmf: degradation-robust multi-modal image fusion via composable diffusion prior: 
 Anonymous Author(s) Submission Id: 2367 Figure 1 : Fusion schematic in challenging scenarios for MMIF tasks. DDFM [64] and DiffLL [12] are Diffusion-based methods for image fusion and low-light image enhancement. DIVFusion [40] is an illumination-robust image fusion method, and CAT [2] is a SOTA universal image restoration method for preparatory denoising and super-resolution. 
 
 Appendix:  


Decision: The submission presents a novel and promising approach to multimodal image fusion that addresses significant challenges in the field. While there are concerns regarding the computational demands and the need for more rigorous experimental validations, the strengths of the paper—particularly its innovative approach and strong experimental results—suggest that it would be a valuable contribution to the conference. I recommend an Accept, encouraging the authors to address the highlighted concerns regarding computational efficiency and further experimental validations in future revisions or extensions of this work.

Reviews: review: paper_summary: paper_summary: Existing multimodal fusion algorithms are unable to solve the low-quality degraded image fusion, based on this, this paper proposes a degradation-robust multimodal image fusion method, which utilizes the powerful generative properties of diffusion model to offset various degradations in the image fusion process.
strength_weakness: strength_weakness: In this paper, a new multimodal image fusion framework based on diffusion models is proposed, which utilizes the powerful generative properties of diffusion models to mitigate various complex degradations in the source images. Also in this paper, a diffusion prior combination module is developed to aggregate generative diffusion priors from different modalities, effectively exploiting the complementary nature of diffusion models driven by various modalities and degradations.
limitations: limitations: The introduction of infrared-visible image fusion and medical image fusion as two side-by-side tasks is not considered reasonable, and image fusion in industrial field and medical image fusion can be treated as a kind of side-by-side task.

The ablation experiments in this paper are not sufficient, and detailed ablation experiments should be conducted to verify the necessity and role of the designed module.
paper_summary: paper_summary: This paper proposed Degradation-Robust Multi-modality image Fusion (DRMF), which leverages generative diffusion models to counteract various degradations during image fusion. By pre-training multiple degradation-robust conditional diffusion models for different modalities and integrating their generative priors, DRMF excels in infrared-visible and medical image fusion, even under complex degradations.
strength_weakness: strength_weakness: 1. Extensive experiments demonstrate the outstanding performance of the proposed model.
2. The figures are aesthetically pleasing and effectively convey the motivation and process of the proposed method.
3. This proposed Degradation-Robust Multi-modality image Fusion (DRMF) is a novel way to integrate generative priors from pre-trained uni-modal models, enabling effective multi-modal image fusion.
limitations: limitations: 1. Pre-training multiple degradation-robust conditional diffusion models may be computationally intensive and time-consuming.
2. Integrating generative priors from pre-trained uni-modal models might introduce complexity to ensure effective fusion.
3. There could be challenges in maintaining the balance between different modalities.
paper_summary: paper_summary: In this work, authors propose Degradation-Robust Multi-modality image Fusion (DRMF), which utilizes the robust generative properties of diffusion models to address various degradations in image fusion. They pre-train multiple degradation-robust conditional diffusion models for different modalities to handle degradations effectively. Following this, they develop a diffusion priori combination module to merge the generative priors from these pre-trained uni-modal models, facilitating efficient multi-modal image fusion.
strength_weakness: strength_weakness: 1.	The writing of the paper is good. The authors clearly identify the problems in existing MMIF methods and explain how their work addresses and solves these issues.
2.	Using the diffusion model to address various degradations in image fusion is intuitively feasible.
3.	The authors conducted comprehensive experiments to verify the effectiveness.
limitations: limitations: The model's performance appears to be good. However, the authors need to focus on discussing computational cost. Will the use of diffusion models and related designs lead to increased computational costs? How can the trade-off between performance and computational cost be achieved?
paper_summary: paper_summary: Based on the noise process of the diffusion process, the article believes that composable diffusion prior is helpful for multivariate image fusion and is robust to degradation. The article uses this to design a two-stage training Diffusion model. In the first stage, the diffusion model of each modality is trained. In the second stage, a diffusion priori combination module is trained separately to perform a weighted summation process of the noise of different modes. Advanced results on multiple datasets demonstrate its effectiveness.
strength_weakness: strength_weakness: 1. The article introduces a diffusion model to provide a good degradation-robust single-modal prior.
2. Use the single-modal prior input trained in stage 1 to dynamically weight the information of different modalities.
3. Some ablations show the effectiveness of the proposed method. And the detection task shows its potential of applicaition.
limitations: limitations: 1. What the shape of $\gamma_t^i$ is? $\mathbb R^{H\times W\times C}$ or just a scalar?
2. According to Eq. (10), is there an arrow from $x_t^f$ to Diffusion Priori Combination Module?
3. A major concern is that: after the noise prediction network $\epsilon_\theta^{i}$ are trained, the diffusion process $x_t$ in stage 1 and in the stage 2 are different. In stage 1, the $x_t$ is noisy (and degraded) single-modal visible image (or infared image), but in stage two, the $x_t$ is changed to fused image $x_t^f$. There exists an domain shift for the noise prediction network between stage one and two. The authors should consider it and explain why this does not affect the fusion performance. Moreover, the U-Net $\varphi$ only predict weights for combinate the predicted noise $\epsilon_\theta^i(x_t^f, x_c^i, t)$, it still can not mitigate the domain shift.

If the authors solves the concern 3., I am willing to rise my rating.

---

After rebuttal:

1. This explanation is quite useful, although domain shift still exists (even though the authors did not deny this), but the experimental figure shows it still works. Anyway, I rise the rating to "Weak Accept". Authors should add this discussion to the manuscript (This is important).