Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model

Authors: hao  zhang
lei  cao
jaiyi  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 06:01:50

Abstract: Existing multi-modal image fusion methods fail to address the compound degradations presented in source images, resulting in fusion images plagued by noise, color bias, improper exposure, etc. Additionally, these methods often overlook the specificity of foreground objects, weakening the salience of the objects of interest within the fused images. To address these challenges, this study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse. First, this framework integrates feature-level information integration into the diffusion process, allowing adaptive degradation removal and multi-modal information fusion. This is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion. Second, by embedding the combination of the text and zero-shot location model into the diffusion fusion process, a text-controlled fusion re-modulation strategy is developed. This enables user-customized text control to improve fusion performance and highlight foreground objects in the fused images. Extensive experiments on diverse public datasets show that our Text-DiFuse achieves state-of-the-art fusion performance across various scenarios with complex degradation. Moreover, the semantic segmentation experiment validates the significant enhancement in semantic performance achieved by our text-controlled fusion re-modulation strategy. The code is publicly available at https://github.com/Leiii-Cao/Text-DiFuse.

Main paper: 
 introduction: 
 Due to constraints in imaging principles and hardware technology, single-modal images fall short of accurately and comprehensively describing scenes, thereby limiting their utility in subsequent tasks. Hence, image fusion technology emerges as essential in this context [60, 28] . It aims to integrate useful information from multi-modal images, producing high-quality visual results that enhance both human and machine perception of scenes. Currently, image fusion technology has been integrated into various tasks, significantly advancing performance in related fields such as autonomous driving [47] , intelligent security [57, 25] , and disease diagnosis [14] .Over recent decades, rapid advancements in deep learning have propelled significant progress in image fusion. Deep learning-based methods have surpassed traditional approaches in fusion performance by a considerable margin. In the historical context, the evolution of image fusion closely aligns with the• A text-controlled fusion re-modulation strategy is designed, allowing users to customize fusion rules with language to enhance the salience of objects of interest. This interactively improves the visual quality and semantic attributes of fused images.• We evaluate our Text-DiFuse on extensive datasets and verify its advantages over state-of-the-art methods in terms of degradation robustness, generalization ability, and semantic properties. 
related work: 
 Deep Multi-modal Image Fusion. As mentioned earlier, the progress in deep multi-modal image fusion is closely tied to updates in neural network paradigms. Initially, AE-based fusion methods [19, 20] utilize pre-trained encoders and decoders alongside hand-crafted fusion rules, resulting in performance bottlenecks. Subsequent methods introduce CNN [6, 5] and Transformer [37, 58] for end-to-end fusion guided by specific unsupervised loss, yielding improved performance. The introduction of GAN is groundbreaking due to their inherently unsupervised nature, enabling the preservation of important multi-modal features [29, 56] . However, the instability of the adversarial game often leads to non-equilibrium appearances in fused images [30] . Furthermore, the diffusion model is highly anticipated for solving image fusion and is used in two main ways: injecting features into CNNs for separate fusion and diffusion [52] , or treating multi-modal images as conditions for implicit fusion [63] . However, both methods fail to utilize the diffusion model's degradation removal capabilities and struggle with complex degradation. In contrast, our Text-DiFuse embeds feature fusion into the diffusion process, ensuring robustness and aggregation of multi-modal information. Additionally, we use text combined with a zero-shot location model for user-customized fusion, enhancing object salience.Diffusion Model. The impressive performance of the diffusion model [41, 13] makes it top-notch in visual generation. It constructs a Markov chain by progressively adding noise forward, and then estimates the underlying data distribution and uses inverse sampling to generate images. This natural property of degradation removal has made the diffusion model excel in visual restoration tasks [46, 61] . However, the practical application of the diffusion model is hindered by its slow T-step continuous sampling. Recent efforts have focused on enhancing sampling efficiency and sample quality [50] . For example, DDIM [42] extends the original denoising diffusion probability model to non-Markovian scenarios, requiring only discrete time steps during sampling to reduce costs. Furthermore, iDDPM [35] introduces an enhanced denoising diffusion probability model, parameterizing backward variance through linear interpolation and training with mixed objectives to acquire knowledge of backward variance. This approach increases log-likelihood and accelerates sampling rates without compromising sample integrity. Therefore, our method employs iDDPM to expedite sampling while upholding the quality of fused images.Zero-shot Location. Establishing connections between unseen and seen categories using semantic information [18] , zero-shot location models [27, 33] can understand unseen images to identify and locate designated objects. Representative zero-shot location models include GLIP [22] , OWL-VIT [33] , and Grounding DINO [27] . Additionally, methods like DiffSeg [40] , PADing [12] , and SAM [17] achieve finer pixel-level object localization. These powerful zero-shot location techniques provide a solid foundation for the implementation of our text-controlled fusion re-modulation. 
problem statement and modeling: 
 Let us formally define the research problem of this work: achieving multi-modal image fusion under degraded scenes while supporting text-controlled fusion re-modulation of objects of interest. The multi-modal image pair captured under degraded conditions is formulated [{X, Y }|Ω], in which {X, Y } denotes clean multi-modal images (e.g., infrared and visible images), and Ω indicates composite degradation (e.g., color casts, noise, and improper lighting). We aim to process degraded multi-modal images to obtain a clean fused image: Z = Γ([{X, Y }|Ω]). The function Γ must handle two tasks: degradation removal R and information fusion F . There are two routes: concatenation (Γ = R + F ) and coupling (Γ = R F ). Concatenation overlooks the intrinsic connection between degradation removal and fusion, leading to limited performance (see comparative experiments). Therefore, we choose the coupling route and introduce the diffusion model for degradation removal. Then, the key question becomes how to integrate the diffusion model with information fusion. Notably, the diffusion operates as a continuous process with multi-step sampling, making it difficult to incorporate fusion. To address this, we propose a novel explicit coupling paradigm of information fusion and diffusion, as illustrated in Fig. 2 (a) . Specifically, the diffusion model is initially trained on data with compound degradation, incorporating the degradation removal prior into the encoder ε b E and decoder ε b D . During T-step reverse sampling, the multi-modal encoded features are continuously passed to the FCM for fusion, aiding in the reconstruction of the final fused image by the decoder. This essentially consolidates multiple diffusion processes into a single one, effectively integrating degradation removal and information fusion. However, the above methodology has not yet addressed the re-modulation of objects of interest in image fusion. To this end, we further develop a text-controlled fusion re-modulation strategy to highlight the objects of interest, thereby enhancing subsequent semantic decision performance. This strategy can be formulated as Z = Γ([{X, Y }|Ω], L), where L represents the user-defined language command. Specifically, we utilize text combined with the zero-shot location module to identify and locate the objects of interest. This knowledge triggers the re-modulation of diffusion fusion to enhance the saliency of the objects with in-built contrast-enhancement prior, thereby significantly improving perceptual quality for both humans and machines.Diffusion for Degradation Removal. Embedding the degradation removal prior into the encoderdecoder network of the diffusion model forms the foundation for diffusion fusion. We consider three primary types of degradation: color casts, noise, and improper lighting. They cover both nighttime and daytime negative imaging conditions and can be considered comprehensive to a certain extent. In our model, we separate brightness and chrominance components, and perform independent diffusion for them. Here, we describe and represent these two diffusion processes consistently. For clean components s (brightness or chrominance), the corresponding degraded versions Ω(s) involve composite degradation like color casts, noise, and improper lighting. These degraded components are fed into the encoder-decoder network as conditions. As shown in Fig. 2 (b), in the forward diffusion process, the original clean component s, denoted as s 0 at step 0, is progressively added with Gaussian noise over T steps to obtain s T . In the reverse sampling process, the encoderdecoder network is guided to estimate the mean µ θ (s t , Ω(s), t) and variance θ (s t , Ω(s), t) of s t-1 , progressively approaching the clean component with the conditions Ω(s). Drawing upon iDDPM [35] , the optimization can be defined as:∇ θ t -θ (s t , Ω(s), t) 2 +λD KL (q(s t-1 |s t , s 0 , Ω(s))||p θ (s t-1 |s t , Ω(s))), t > 1 (1a) ∇ θ t -θ (s t , Ω(s), t) 2 -λ log p θ (s 0 |s 1 , Ω(s)), t = 1 (1b)where ∇ θ means optimization by gradient descent, t denotes the added noise in the forward diffusion process, D KL is the regularization term based on KL divergence, and q and p θ represent the prior and posterior probability distributions, respectively. θ is the noise predictor.Fusion Control Module for Information Fusion. For information fusion, we design an FCM to aggregate encoded features during the diffusion process, as shown in Fig. 2 (c ). It primarily consists of convolution layers with CBAM [45] . In them, integrating spatial and channel attention mechanisms helps perceive the importance of multi-modal features on a wider scale, promoting rational feature fusion. To reduce the solution space, FCM generates weight coefficients for fusion rather than directly predicting fused features, allowing faster convergence in multi-step sampling of the diffusion process. EQUATIONThe FCM generates weight coefficients {ω X b t , ω Y t } for fusing these multi-modal features:EQUATIONwhere [Φ f t |Ω] is the fused feature with residual degradation at step t, and denotes the Hadamard product. Subsequently, the fused feature is fed into the decoder ε b D to predict the contained noise θ (t) at step t, and the relevant variable υ θ (t) for learning the variance:EQUATIONThen, the mean and variance of Z b t-1 can be obtained according to:EQUATIONEQUATIONwhere β t represents the variance associated with the forward diffusion process, using the notation α t = 1 -β t and α t = t s=0 α s . Additionally, we parameterize the variance between β t and β in the logarithmic domain using the technique of iDDPM [35] , where β = 1-αt-1 1-αt β t . Then, Z b t-1 can be computed according to:EQUATIONwhere z denotes the randomly sampled Gaussian noise z ∼ N (0, I) when t > 1, otherwise z = 0. According to Eqs. ( 4)-( 7), each sample will derive an Ẑb 0 :EQUATIONNotably, Ẑb 0 (Z b t , θ (t)) indicates the corresponding fake final fused image that is derived from the results of any step of sampling. Therefore, we construct constraints to guide the FCM in retaining beneficial information during the diffusion fusion process. Considering pixel intensity and gradient as two basic elements that describe images, we specify intensity loss L int and gradient loss L grad to emphasize the preservation of significant contrast and rich texture:EQUATIONEQUATION)where max is the maximum function, ∇ is the Sobel gradient operator, X b and Y are clean source components after diffusion. The total loss is summarized as:EQUATIONwhere γ int and γ grad control the balance of these terms, set to 1 and 0.2, respectively. After optimization, we obtain the clean fused brightness componentZ b = Z b 0 . The purified chrominance component X c is used as the fused image's chrominance component: Z c = X c 0 .Finally, stitching Z b and Z c yields the final fused image Z with accurate colors, minimal noise, and proper lighting. Through these designs, information fusion and diffusion have been fully and explicitly coupled, achieving multi-modal image fusion while removing compound degradation.The above diffusion fusion constitutes the basic version of our method. Now, we aim to expand it into a modulatable version, allowing users to re-modulate the fusion process based on personalized needs, enhancing the perception of objects of interest. Firstly, we use state-of-the-art zero-shot localization models to identify and locate objects of interest based on text commands. Specifically, we introduce OWL-VIT [33] for detecting objects of interest with open-word input. Then, SAM [17] provides pixel-level positioning of these objects, obtaining the mask M . Subsequently, M is fed into the re-modulation block to generate fusion modulation coefficients {κ X b , κ Y }. This block incorporates a built-in contrast-enhancement prior, aiming to maximize the contrast between the object area and the background in the fused image, thus improving the salience of the objects. Consequently, the multi-modal feature fusion in the diffusion process changes from Eq. (3) to:EQUATIONIn non-object areas, the original distribution of diffusion fusion should be maintained:EQUATIONThe modulated fused image enhances the saliency of objects compared to the before, making it more suitable for subsequent advanced tasks. We prove this in the re-modulation verification section.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] .Justification: This paper does not include theoretical results. Actually, it is a pioneering paradigm in which diffusion theory is explicitly embedded in multi-modal image fusion, achieving compound degradation removal while high-quality cross-modal information integration.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 Configuration. We evaluate our method on two typical multi-modal image fusion scenarios: infrared and visible image fusion (IVIF) and medical image fusion (MIF). For IVIF, we use the MSRS dataset [43] , with 485 training and 100 testing image pairs. For MIF, we use the Harvard medicine dataset3 with 160 training and 50 testing image pairs, covering CT-MRI, PET-MRI, and SPECT-MRI. Data augmentation like random flipping and cropping increases the training pairs to 12, 888 for IVIF and 6, 408 for MIF. Besides, generalization is evaluated on 60 pairs from LLVIP [15] and 25 pairs from RoadScene [49] . Competitors include 9 methods: RFN-Nest [20] , GANMcC [31] , SDNet [53] , U2Fusion [49] , TarDAL [25] , DeFusion [23] , LRRNet [21] , DDFM [63] , and MRFS [58] . Five metrics are used: EN [39] , AG [3] , SD [38] , SCD [2] , and VIF [11] . The Adam optimizer with a learning rate of 2e -5 is used for parameter updates. Experiments are conducted on an NVIDIA RTX 3090 GPU and a 3.80 GHz Intel i7-10700K CPU. Comparative Experiments. We first compare the basic version of our Text-DiFuse with current state-of-the-art fusion methods, and the qualitative results are shown in Fig. 3 . The first two rows depict IVIF results, demonstrating our method's ability to correct color casts, restore scene information under low-light conditions, and suppress noise. The last three rows display MIF results, which show that our method can highlight physiological structure information while maintaining functional distribution. In contrast, competitors are unable to achieve such information recovery and still suffer significantly weakened appearance. The quantitative results in Table 1 demonstrate our method's advantages over other fusion techniques. For further fairness, we introduce state-of-the-art low-light enhancement (CLIP-LIT [24] ), denoising (SDAP [36] ), and white balance (AWB [1] ) algorithms as the pre-processing steps for these competitors, with results presented in Fig. 4 and Table 2 . Clearly, our method still outperforms these comparative methods. This is because these added pre-processing steps for information recovery are entirely independent of information fusion, so they cannot mine habits that are more conducive to modal complementarity, leading to limited performance.Generalization Evaluation. Next, we directly test the model trained on the MSRS dataset on the LLVIP and RoadScene datasets to evaluate the generalization ability of the proposed method. We select a daytime scene with overexposure and a low-light nighttime scene, and the qualitative results are shown in Fig. 5 . It can be observed that our Text-DiFuse still maintains high-quality degradation removal and information fusion capabilities. In particular, it has two-way information recovery functions such as overexposure correction and low-light enhancement, producing visually satisfying fused results. By comparison, other competitors lose useful information masked by overexposure or low light. We further prove the good generalization ability of our method in Table 3 . In general, these results indicate that our Text-DiFuse can be applied more reliably in real scenarios.Re-modulation Verification. We verify the performance gains brought by our text-controlled fusion re-modulation strategy on the MFNet dataset [10] . We select 6 state-of-the-art RGB-T segmentation methods for comparison, i.e., MFNet [10] , FEANet [4] , EGFNet [7] , CMX [59] , GMNet [65] ,and MDRNet [48] . Besides, we train SegNext [9] on infrared images, visible images, the fused images generated by the basic version of our method, and the modulatable version respectively to achieve segmentation. It can be seen from Fig. 6 that different language commands derive customized fused results, which promote the completeness and accuracy of semantic segmentation while visually highlighting the objects of interest. The quantitative results in Table 4 further prove that our re-modulation strategy can improve the semantic attributes, achieving the best segmentation scores.Semantic Verification on Detection. We further verify the semantic gain brought by text modulation on the object detection task. Specifically, the MSRS dataset [43] is used, which includes pairs of infrared and visible images with two types of detection labels: person and car. Therefore, the text instruction is formulated as: "Please highlight the person and car", which guides our method to enhance the representation of these two types of objects in the fused image. Then, we adopt the YOLO-v5 detector to perform object detection on infrared images, visible images, and fused images generated by various image fusion methods. The visual results are presented in Fig. 7 , in which more complete cars and people can be detected from our fused images while showing higher class confidence. Furthermore, we provide quantitative detection results in Table 5 . It can be seen that the highest average accuracy is obtained from our fused images, demonstrating the benefits of text modulation. Overall, these results indicate that text control indeed provides significant semantic gains, benefiting downstream tasks.Ablation Studies. We conduct ablation studies to verify the effectiveness of specific designs, involving eight variants: I: removing FCM with using maximum rule; II: removing FCM with using addition rule; III: removing FCM with using mean rule; IV: removing FCM with using variancebased rule [34] ; V: removing L grad ; VI: removing L int ; VII: removing diffusion with using AE route; VIII: our full model. The visual results in Fig. 7 show that removing any of these designs results in a reduction of visual satisfaction. The quantitative scores in Table 5 also support this view.Overall, these designs in our Text-DiFuse collectively guarantee advanced fusion performance.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] .Justification: This paper aims to propose a new multi-modal image fusion algorithm that can adapt to degraded scenes. It is completely reproducible. First, we describe in detail the experimental conditions of this work in the experimental configuration section, including datasets, training details, and computing hardware. Second, we provide a URL link to the code of our method in the abstract section, including all the necessary functions for training and inference. We also provide a README file within it to outline the necessary environment configuration for running, as well as detailed usage instructions. Together these ensure the experimental result reproducibility in this paper. 
conclusion: 
 This paper proposes a new interactive multi-modal image fusion framework based on the textmodulated diffusion model. On the one hand, it is the first to develop an explicit coupling paradigm for information fusion and diffusion models, achieving the integration of multi-modal beneficial information while removing composite degradation. On the other hand, a text-controlled fusion re-modulation strategy is designed. It incorporates text combined with the zero-shot location module into the diffusion fusion process, supporting users' language control to enhance the perception of objects of interest. Extensive experiments demonstrate that our method achieves better performance than current methods, effectively improving the visual quality and semantic attributes of fused results. 
acknowledgement: 
 This work was supported by NSFC (62276192). 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] .Justification: The experimental results reported in this paper are the average of a large number of test results in the dataset. Therefore, they are statistically significant, being able to support and validate the contributions and claims of this paper.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] .Justification: In the experimental configuration section, we provide the computing resources required to reproduce the experiments in this paper, including an NVIDIA RTX 3090 GPU and a 3.80 GHz Intel i7-10700K CPU.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] .Justification: All data, codes, and methodologies involved in this paper comply with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] .Justification: We discuss the potential impacts of this work in the supplementary material. Specifically, this paper is devoted to solving the problem of multi-modal image fusion under degraded scenes to provide high-quality fused results suitable for human and machine perception. Therefore, it can be expected that this work will demonstrate positive social impacts in many fields. For example, it can help drivers better perceive the road conditions ahead in environments with poor visibility through information fusion, such as at night, to improve driving safety. For another example, it can help poor areas that only have low-quality medical imaging equipment to enhance the perception of the body's condition through information recovery and fusion, thereby assisting in disease diagnosis and treatment.As far as we know, this work does not appear to have any negative social impacts and the risks are extremely low.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] .Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] . Justification: All data covered in this paper are publicly available, and we provide accurate citations for them. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] . Justification: The code for our work is provided as a zip file, which already contains an MIT License.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 
 
 Appendix: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.• The full details can be provided either with the code, in appendix, or as supplemental material.https://www.med.harvard.edu/AANLIB/home.html 


Decision: The authors have provided a good rebuttal. All four reviewers agreed that this paper proposed a novel and effective multi-modal image fusion method, and their concerns have also been properly resolved by the authors’ rebuttal. The final scores of this paper are all quite positive:  “Weak Accept”, “Weak Accept”, “Accept”, and “Accept”. The meta-reviewer agreed with the reviewers and recommended the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes an interactive framework that can exploit the intrinsic connection between image restoration and multi-modal image fusion.
The authors embed information fusion within the diffusion process and address the "composite degradation challenge" i.e., multi-modal information integration with
effective information restoration from degradation like colour casts, noise, and improper lighting. Particularly, first, independent conditional diffusion models are applied
to each modality with compound degradation -- the degradation removal priors are embedded into the encoder-decoder network. A fusion control module (FCM) sits in
the multi-step diffusion process to manage the integration of multi-modal features and remove degradation during T-step sampling. Next, to interactively enhance
focus on objects of interest during diffusion fusion, the authors designed a text-controlled fusion re-modulation strategy that incorporates a text and a zero-shot OWL-ViT to
identify the objects of interest. In other words, this step performs a secondary modulation with the built-in prior to enhance saliency.
strength_weakness: strength_weakness: For now, I have minor concerns and mostly questions (as listed in the next section).
 - The authors should add a brief discussion on the competitors in supplementary material. For example, differences between TarDAL, DeFusion, LRRNet, DDFM, and MRFS.
 - Typo in Eq. 2: $\Theta_{t}^{X^{B}}$ should be $\Theta_{t}^{X^{b}}$.
 - Improve the caption of Figure 2. I had to read the entire paper to understand the figure (it should be self-explanatory).
 - Not much of a weakness, but the authors could improve the clarity of the paper if they added the tensor dimension of each variable in Figure 2.
questions: questions: - In the proposed method, input visual image X is broken into brightness and chroma components. I wonder if this step is absolutely necessary -- or can we skip $\eta^{c}_{
    theta}$ and directly combine both $X$ and $Y$ as three-channel images $\mathbb{R}^{H \times W \times 3}$.
 - What if I use InstructIR (for image restoration) followed by MaxFusion (for multi-modal fusion) -- how would it compare with the proposed method?
    - (InstructIR) https://arxiv.org/pdf/2401.16468 | Github: https://github.com/mv-lab/InstructIR
    - (MaxFusion) https://arxiv.org/pdf/2404.09977 | Github: https://github.com/Nithin-GK/MaxFusion
 - In the Limitation and Future work section, will a no-training approach be possible? For example, "MaxFusion"-like approach but with the proposed deep integration of image restoration and multi-modal fusion.
limitations: limitations: The authors discussed limitation section in supplementary A.4 -- Particularly, Table S1 shows number of parameters and runtime. This is highly appreciated.
paper_summary: paper_summary: This paper addresses two primary challenges in multimodal image fusion: the mixed degradation of modalities and the insufficient salience of target objects. It proposes two methods to tackle these challenges: feature-level fusion diffusion and the re-modulation of fusion rules in target areas using a zero-shot segmentation model. They implement adequate experiments for evaluation, and the results demonstrate this method's advanced performance across various aspects, including the visual and semantic.
strength_weakness: strength_weakness: - This paper mentioned that the diffusion model is pre-trained to enable the denoising network to have the degradation removal function. However, details about the construction of the data used to train the diffusion model are missing. They need to describe this process to make the overall approach clearer.
- This paper focuses on multimodal image fusion, being reflected in the title. In the main text, the proposed method is evaluated in two scenarios: infrared and visible image fusion and medical image fusion. In the supplementary materials, they further provide experiments on polarization image fusion. I am curious whether the applicable scenarios of the proposed method can be further expanded, such as the typical fusion of near-infrared and visible bands.
- The experiments on polarization image fusion only provide visual results, and it would be better to add a quantitative evaluation.
- I noticed that the proposed method separates the chrominance component and the brightness component, and then performs de-degradation on them separately. An explanation of why this operation is needed should be given. Perhaps an ablation experiment could more intuitively show the effect of this operation.
- There are some minor typos, such as potential misspellings of dataset names in Tables 1 and 2. In addition, there seems to be a lack of underline on AG's second place.
questions: questions: 1. How were the degradation condition data constructed, were paired supervised datasets used or synthetic datasets?
2. Has there been an attempt to evaluate the fusion on the fusion of near-infrared and visible bands?
3. Could you provide the quantitative results of polarization image fusion?
4. The separation of chrominance and brightness requires more explanation.
limitations: limitations: Limitations and broader impacts have been included.
paper_summary: paper_summary: This work focuses on the topic of multi-modal image fusion. Two innovations enhance the performance of the fusion. One is the clever integration of information fusion into the diffusion process. This coupling way enables the fusion function to resist degradation. The other is the introduction of a text-based fusion remodulation strategy. This changes the limitation of previous fusion methods that could only use fixed mappings, allowing for the dynamic adjustment of the fused image based on specific requirements. This remodulation also enhances semantic attributes, improving the scores of the semantic segmentation task.
strength_weakness: strength_weakness: 1. On page 5, line 174, the source data used for fusion contains degradation, [{Xb,Y}|N]. My question is, in Equations (9) and (10), where do the clean {Xb,Y} used to guide the fusion come from? Is there a multi-modal dataset that contains paired degraded and clean data? The paper seems to lack an explanation for this. 
2. The forward process of the diffusion model involves T steps of noise addition, while the reverse process consists of T steps of iterative sampling. Is the Z0 obtained in equation (8) a hypothetical Z0 derived from the diffusion relation at each sampling, or is it the Z0 after completing the full T steps of sampling? This determines the object of the constraints in the loss functions (9) and (10). It would be better to provide a detailed discussion on this. 
3. Only after the T steps of sampling can the data without degradation be obtained. So why can Z_{t-1}^b in equation (7) be considered free from degradation N? 
4. It's understandable that using textual modulation to control the desired targets of interest can enhance semantic attributes. My question is whether these enhanced semantic attributes can be generalized. In other words, can it also be effective in other high-level visual tasks besides semantic segmentation? 
5. Typo: The Zt on the left side of equation (8) seems to have a missing superscript b.
questions: questions: Please answer the question raised in Weaknesses.
limitations: limitations: Yes, there are discussions about the limitations and potential negative societal impacts.
paper_summary: paper_summary: A new paradigm of multi-modal image fusion named Text-DiFuse is introduced, based on the diffusion model. The paradigm embeds a mechanism for aggregating feature-level multi-modal image information into the diffusion process of degrading multi-modal images, addressing the optimization gap between "degradation removal" and "multi-modal information fusion". Additionally, a zero-shot model is introduced to modulate the fusion strategy based on user-input target text, enhancing the saliency of the target of interest. The conducted experiments suggest significant improvements in both human visual perception and advanced computer vision tasks.
strength_weakness: strength_weakness: 1)	After the diffusion model is effectively trained, the sampling process can follow different step intervals. The information fusion in this method is integrated into the diffusion process, but the article does not seem to specify the sampling interval at which the results are obtained. Also, this article does not discuss the impact of the sampling interval on the fusion performance.
2)	The presentation is slightly unclear. For example, from Equation 2 to Equation 6, both the features and the images carry the condition N that represents the degradation. Why does equation 7 no longer include N? Why can it be considered that the degradation has been removed at this point?
3)	In Table 2 and Figure 4, some existing image restoration methods are cascaded in front of the fusion method to promote fairness in comparison, such as low-light enhancement (CLIP-LIT), denoising (SDAP), and white balance (AWB) algorithms. 
Please explain the choice of the order in which they are connected in series, i.e. why low light enhancement first, then denoising, and finally white balance. 
4)	Modulating the salience of targets of interest in the fusion process through language is novel. Intuitively, I think the improvement in semantic properties brought about by this modulation is widespread. Currently, the effectiveness of language modulation has only been verified in the semantic segmentation scenario. It is recommended to provide an evaluation in the object detection scenario to further verify its role.
questions: questions: Please refer to the weaknesses part.
limitations: limitations: The authors have analyzed the limitations and potential negative impact.

Comments: title: To author's response
comment: We truly appreciate your efforts in improving our paper. If this paper is accepted, we plan to incorporate the following revisions into the camera-ready version in response to your recommendations:

1. Explain the source of clean data for constructing loss functions: We adopt a two-step strategy to alleviate this challenge of unavailable data. The core idea of the two-step strategy is to relax the high data requirements by pre-training a generative model with limited available data. Then, this pre-trained model allows for the production of data that is not available in reality. 

2. Explain the source of $Z_0$: We customize Eq. (8) based on the diffusion relationship, so we can derive the corresponding fake final fused image $Z_0$ from the results of any sampling step and apply the corresponding fusion constraints. We also consider using the results after all sampling as Z0 in loss functions (9) and (10). The quantitative results are reported below, demonstrating the advantages of our method compared to the full sampling strategy. 
| $Z_0$ | EN | AG| SD| SCD| VIF|
| ---- | ---- | ---- | ---- | ---- | ---- |
| Full Sampling  | 5.93 | 1.74 | 23.99 | 1.26 | 0.63 |
| Ours | **7.08** | **3.31** | **47.44** | **1.44** | **0.76** |

3. Explain the degradation $N$: The degradation symbol in Eqs. (2)-(6) is not the same thing as the noise in the diffusion process. The former refers to the degradation from the source images, specifically including improper lighting, color distortion, and random noise. In contrast, the latter comes from the Gaussian noise assumption of diffusion theory itself. 

4. Verify textual modulation in object detection: We implement application experiments on the object detection task to verify the generalization of semantic attributes. Experimental results indicate that textual modulation indeed enhances semantic information, thereby benefiting downstream tasks.  In the future, we will explore controllable semantic decision paradigms based on text integration, achieving various interesting functions, such as text-assisted, text-specified, and text-deception decisions. 

We greatly appreciate your positive feedback on our work. If you have any further questions or concerns, please feel free to contact us.
We thank Reviewer DZyq for the insightful and valuable feedback. We have included new experimental results to demonstrate the necessity of brightness-chrominance separation in our method. Additionally, we have conducted a comparison with InstructIR+Fusion to further highlight the advanced performance of our method. Furthermore, a no-training version of our methods has been explored. We hope we have addressed all of your concerns. If you have any additional questions, please let us know.
We truly appreciate your efforts in improving our paper. If this paper is accepted, we plan to incorporate the following revisions into the camera-ready version in response to your recommendations:

1. Supplement data construction details: We use existing supervised datasets (MIT-Adobe FiveK Dataset, Rendered WB dataset) and additionally simulate a portion of the data to meet the requirements of mixed degradation (such as improper lighting, color distortion, and noise). These data constitute the complete dataset for training our diffusion model, driving our model's learning for compound degradation removal.

2. Add additional application scenarios: We extend our proposed model to the near-infrared and visible image fusion task. The experimental results continued to demonstrate the advantages of our method.

3. Supplement quantitative evaluation of polarization image fusion: We supplement the quantitative assessment of the polarization image fusion, showing that our method achieves the best scores in four of all five metrics.

4. Discuss brightness-chrominance separation: We discuss the two reasons why our method does not directly handle three-channel images, including the nonlinear amplification of interference and the impact of fusion loss functions.

We greatly appreciate your positive feedback on our work. If you have any further questions or concerns, please feel free to contact us.
Thank you very much for your positive feedback on our paper. The textual modulation enhances generalized semantic attributes, illustrating that abstract text encompasses rich semantic information and can aid in improving machine vision perception. This observation can inspire the design of methods for various high-level visual tasks. In the future, we will explore controllable semantic decision paradigms based on text integration, achieving various interesting functions, such as text-assisted, text-specified, and text-deception decisions. Furthermore, implicit integration of information restoration and fusion is indeed more attractive than explicit concatenation. This is not only because of its advanced performance but also due to its ability to handle multiple tasks with a single set of parameters. We commit that if this paper is accepted, all clarifications provided in the rebuttal will be incorporated into the camera-ready version.
Thanks for the effort to clarify my questions in the provided rebuttal. Using the two-step strategy to address the limitation of data unavailability is clever, and I'm pleased to see the general semantic attributes brought by textual modulation. Therefore, I'm inclined to accept this paper. In the camera-ready version, please include the provided clarifications about the source of Z0 in loss functions (9) and (10) and the degradation N.
The response is clear and my concerns are addressed. In particular, the effectiveness of semantic attribute improvement is verified in the object detection scenario. I think this observation is inspiring. I also see that the authors perform an additional comparison by using an all-in-one image enhancement algorithm InstructIR, still showing this work's effectiveness. Consequently, I plan to keep my original rating and recommend accepting this paper.
OK. Your reply solved my doubts.