Conditional Controllable Image Fusion

Authors: bing  cao
xingxin  xu
pengfei  zhu
qilong  wang
qinghua  hu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-22 22:19:28

Abstract: Image fusion aims to integrate complementary information from multiple input images acquired through various sources to synthesize a new fused image. Existing methods usually employ distinct constraint designs tailored to specific scenes, forming fixed fusion paradigms. However, this data-driven fusion approach is challenging to deploy in varying scenarios, especially in rapidly changing environments. To address this issue, we propose a conditional controllable fusion (CCF) framework for general image fusion tasks without specific training. Due to the dynamic differences of different samples, our CCF employs specific fusion constraints for each individual in practice. Given the powerful generative capabilities of the denoising diffusion model, we first inject the specific constraints into the pre-trained DDPM as adaptive fusion conditions. The appropriate conditions are dynamically selected to ensure the fusion process remains responsive to the specific requirements in each reverse diffusion stage. Thus, CCF enables conditionally calibrating the fused images step by step. Extensive experiments validate our effectiveness in general fusion tasks across diverse scenarios against the competing methods without additional training. The code is publicly available.

Main paper: 
 introduction: 
 Image fusion aims at integrating complementary information from multi-source images, fusing a new composite image containing richer details [1] . It has been applied in various scenarios that single image contains incomplete information, such as multi-modal fusion (MMF) [2, 3] , multiexposure fusion (MEF) [4, 5] , multi-focus fusion (MFF) [6] , and remote sensing fusion [2] . The fused image inherits the strengths of both modalities, resulting in a composite with enhanced visual effects [7] . These fusion tasks have diverse downstream applications in computer vision, including object detection [8] [9] [10] , semantic segmentation [11, 12] , and medical diagnosis [13] because the comprehensive representation of images with multi-scene information contributes to the improved performance of applications.Recently, numerous image fusion methods [14] [15] [16] [17] have been proposed, such as traditional fusion methods [18] , CNN-based fusion methods [19, 20] and GAN-based methods [21] . While these methods produce acceptable fused images in certain scenarios, they are also accompanied by significant drawbacks and limitations: (i) They are often tailored for specific scenarios or individual tasks, limiting their adaptability across diverse applications; (ii) These methods necessitate training and consume substantial computational resources, posing limitations in terms of time and resource requirements. Lately, denoising diffusion probabilistic models (DDPM) have emerged as an iterative The distinct process of sampling has different favor of the conditions. The crucial role that diverse conditions play in controlling various image generation processes. Throughout the diffusion sampling, different conditions are dynamically selected to best suit the generation requirement at each stage. generation framework, showcasing impressive capabilities in unconditional generation. Inspiringly, numerous researchers explored its controllable aspects. ILVR [22] proposed iterative latent variable refinement with a reference image to control image translation. Some recent works [23, 24] employed the diffusion model for image fusion, which fuses images in fixed fusion paradigms (fixed fusion conditions) by using its inherent reconstruction capacity. However, these approaches are not qualified for sample-customized fusion with dynamic conditions. At present, general image fusion with controllable diffusion models is still a challenging problem, warranting further exploration.In this paper, we propose a diffusion-based controllable conditional image fusion (CCF) framework, which controls the fusion process by adaptively selecting optimization conditions. We construct a condition bank of generally used conditions, categorizing them into basic, enhanced, and task-specific conditions. CCF dynamically assigns fusion conditions from the condition bank and continuously injects them into the sampling process of diffusion. To enable flexible integrated conditions, we further propose a sampling-adaptive condition selection (SCS) mechanism that tailors condition selection at different denoising steps. The iterative refinements of the sampling are based on the pre-trained diffusion model without additional training. It is worth noting that the estimated fused images are conditionally controllable during the iterative denoising process. The diffusion process seamlessly integrates these conditions during the sampling process, decreasing potential impacts. As illustrated in Fig. 1 , the generation process emphasizes different aspects at various sampling steps.In the initial stages, the condition selection is influenced by random noise, resulting in a random selection. During the intermediate stages, there is a shift towards content components. In the final stage, the emphasis moves to generating and selecting texture details. These various conditional factors contribute to different aspects of fusion results and demonstrate the necessity and effectiveness of introducing specific conditions in different stages. To the best of our knowledge, we for the first time propose a conditional controllable framework for image fusion. The main contributions are summarized as follows:• We propose a pioneering conditional controllable image fusion (CCF) framework with a condition bank, achieving controllability in various image fusion scenarios and facilitating the capability of dynamic controllable image fusion.• We propose a sampling-adaptive condition selection mechanism to subtly integrate the condition bank into denoising steps, allowing adaptive condition selection on the fly without additional training and ensuring the dynamic adaptability of the fusion process.• Extensive experiments on various fusion tasks have confirmed our superior fusion performance against the competing methods. Furthermore, our approach qualifies for interactive manipulation of the fusion results, demonstrating our applicability and efficacy. 
related work: 
 Image fusion focuses on producing a unified image that amalgamates complementary information sourced from multiple source images [25] .Specialized. Focus on specialized tasks such as VIF, several early approaches [26] [27] [28] relied on CNNs to address challenges across various scenarios. GTF [29] defined the objective of image fusion as preserving both intensity information in infrared images and gradient information in visible images.Besides that, researchers started artificially incorporating prior knowledge to aid in the fusion process. CDDFuse [30] introduced the concept of high and low-frequency decomposition with dual-branch as prior information. Diverging from approaches tailored to single scenarios, numerous methods are now exploring the development of a unified fusion framework.DDFM [24] represents the pioneering training-free method that employs a diffusion model for multi-modal image fusion.Generalized. Not limited to specialized applications, researchers aim to extend its use to generalized tasks. U2Fusion [31] introduced a unified framework capable of adaptively preserving information and facilitating joint training across various task scenarios. Additionally, SwinFusion [32] proposed a cross-domain distance learning method that has been extended to form a unified framework encompassing diverse task scenarios. Defusion [33] employs self-supervised learning techniques to decompose images and subsequently adaptively fuse them. TC-MoA [34] proposed a novel task-customized mixture of adapters for generating image fusion with a unified model, enabling adaptive prompting for various fusion tasks.Nevertheless, these methods cannot control image fusion for adaptation to different scenarios. Therefore, we propose a method that enables image control, manipulating the fused image through existing conditions on our condition bank. 
preliminary: 
 Denosing diffusion probabilistic models (DDPM) is a class of likelihood-based models that shows remarkable performance [35] with a stable training objective in unconditional image generation. The diffusion process entails incrementally introducing Gaussian noise to the data until it reaches a state of random noise. For a clean sample x 0 ∼ q(x 0 ) each step within the diffusion process constitutes a Markov Chain, encompassing a total of T steps, relying on the data derived from the preceding step. Gaussian noise is added as follows:EQUATIONwhere {β t } T t=1 is the variance schedule of each diffusion step which is fixed and predefined. The generative process learns the inverse of the DDPM forward (diffusion) process, sampling from a distribution by reversing a gradual denoising process. We can directly sample x t at any t step based on the original data x t ∼ q(x t |x 0 ) and via the reparameterization, it can be redefined:EQUATIONwhere defined α t := 1 -β t and ᾱ := t i=1 α i . The diffusion process introduces noise to the data, whereas the inverse process represents a denoising procedure called sampling. In particular, stats with a noise x T ∼ N (0, I), the diffusion model learns to produce slightly less-noisy sample x T -1 , the process can be formulate by:EQUATIONUtilizing the properties of Markov chains, decomposing µ θ and Σ θ , the process of generation is expressed as:EQUATIONwhere,σ 2 θ (t) = Σ θ (t) = (1-αt)(1-ᾱt-1), ϵ θ signifies the output of a neural network, commonly a U-Net. This neural network predicts the noise ϵ θ at each step, which is utilized for the denoising procedure. It can be observed that variance is a fixed quantity, because of diffusion process parameters being constant, whereas the mean is a function dependent on x 0 and x t . However, the stochastic process poses challenges in controlling the generative process.Task-specific  
method: 
 Leveraging the reconstruction capability of unconditional DDPM, we introduced a new controllable conditional image fusion (CCF) framework. Our approach accomplishes dynamically controllable image fusion via progressive condition embedding. In particular, we introduced a condition bank that regulates the incorporation of fusion information using conditions. It allows for combining the dynamic selection of multiple conditions to achieve sampling-adaptive fusion effects. As shown in Fig. 2 , we illustrate our CCF framework in detail with visible-infrared image fusion (VIF). The goal is to generate a fused image f ∈ R H×W ×N from visible v ∈ R H×W ×N and infrared i ∈ R H×W ×N images, where H, W and N denote height, width, and channel numbers, respectively.Firstly, we provide the notation for the model formulation. For each sampling instance, a pre-trained DDPM represents unconditional transition p θ (x t-1 |x t ). Our method facilitates the inclusion of conditional c during the sampling step of unconditional transformation, without no additional training. For this purpose, we sample images from the conditional distribution p(x 0 |c) given condition c:EQUATIONEach transition p θ (x t-1 |x t , c) of the generative process depends on the condition c. From the property of the forward process that latent variable x t can be sampled from x 0 in closed-form, denoised data x 0 can be approximated with model prediction ϵ θ (x t , t):EQUATIONTo compute p(x t |c), we can derive it from the Stochastic Differential Equation (SDE) [36] . For brevity, x 0|t is abbreviated as x 0 , and the expression is given by:EQUATIONClassifier Guidance [37] can be intuitively elucidated via the score function, which logarithmically decomposes the conditional generation probability using Bayes' theorem:EQUATIONFigure 3 : Qualitative comparisons of our CCF and the competing methods on VIF fusion task of LLVIP dataset. Diffusion posterior sampling can yield a more favorable generative trajectory, especially in noisy settings, [38] estimation log p(c|x t ) as follows:EQUATIONFurther, as demonstrated in [39] , we can express the score function as:EQUATION)Here, A(•) can be linear or nonlinear operation. We represent ||C -A(x 0 )|| as δ C . Now, the objective is to obtain x0 and incorporate the condition into it. We can minimize δ C to regulate the sampling within the diffusion process. In the following section, we will provide a detailed explanation of how to build a condition bank and how to select conditions.We empirically construct a condition bank and divide the image constraints into three categories: basic fusion conditions, enhanced fusion conditions, and task-specific fusion conditions. Basic fusion conditions are utilized throughout the entire sampling process, while enhanced fusion conditions are dynamically selected. Task-specific fusion conditions are manually optional, tailored to specific tasks, and may possess unique attributes that can be customized for various task scenarios. All conditions can be part of the enhanced condition set, enabling dynamic selection. The condition bank presented in this paper includes some common conditions, but additional conditions can be explored and utilized in other scenarios.In the above formulation, each conditional Markov transition with the given condition c is shown in Eq. 5. In particular, we constructed a condition bank that allows us to select required conditions C = {c 1 , c 2 , ..., c n }, subsequently integrating them into the unconditional DDPM for executing conditional image fusion. Let C represent a condition bank comprising a series of conditions. The function δ C represents the difference between the source images with the given condition. In every sampling step t, the difference function δ ci can be minimized using gradient descent. These conditions help regulate the image information within each modality involved in the fusion process.Basic Conditions. As shown in Fig. 2 , basic conditions are essential to select for a basic generation. The basic conditions aim to synthesize a foundational fused image, offering an initial, coarse representation. The fused image serves as a primary fusion output, capturing essential features from the source images, though it may suffer from detail loss or texture blur. Notably, different scenarios may require adjustments to the basic condition, as the specific requirements of each fusion task, such as clarity, contrast, and other priorities, can influence its selection. Tailoring the basic condition to align with the unique demands of each task thus ensures an effective fusion process.Enhanced Conditions. Besides basic conditions, we added enhancement conditions for refining the image generation process. The condition bank contains a variety of enhanced conditions, inspired by various Image Quality Assessments (IQA) such as SSIM, and standard deviation(SD). These conditions can be integrated into the CCF generation process to improve the quality of the generated images. The enhanced conditions can be selected with SCS algorithm, allowing different steps of the diffusion sampling process to be optimized with different conditions. This targeted approach ensures that each phase of the image generation is adapted to the specific requirements of that stage, resulting in higher quality fused images.Task-specific Conditions. The purpose of image fusion is to facilitate various downstream tasks. To meet the specific requirements of these tasks, we offer task-specific conditions. These conditions can be manually added to ensure that the fusion process is tailored to suit the downstream applications better. For example, a detection condition can be introduced by utilizing the feature extracted by a detection network F = D(x). The detection condition is formulated as ||F (x), F (M )|| 2 , where X ∈ {x 0 } m i and M is the set of m modalities. Other task-specific conditions can be similarly tailored to optimize the fusion process for different tasks. By integrating these task-specific conditions, the image fusion process can be precisely aligned with the demands of various applications, enhancing the effectiveness and utility of the generated images.During the diffusion sampling process, it is crucial to focus on generating distinct aspects of the image. To address this, we designed an algorithm that dynamically selects the appropriate condition from the condition bank to fit each sampling stage. This selection process can be denoted asC opt = Top k {Gate(C)}.The C opt is the selected condition, and Gate is the gate of selection. We hypothesize that rapidly changing conditions during the sampling process should be prioritized as they indicate greater significance at that generation stage. Inspired by multi-task learning [40] , the gate of conditions can be calculated using the following formula:EQUATIONThe ω i (t -1) represents the condition gradient from the previous step, and the ▽ω i is calculated as:EQUATIONwhere,G W = ||ω i (t)L i (t)|| 2 , L i (t)represents the condition gradient at step t, and the value can be calculated using a gradient descent algorithm in minimize δ ci . The α i (t) is defined as:EQUATIONwhere, L i = Li(t) Li(t-1) , θ is a hyper-parameter. By incorporating this SCS algorithm, we can efficiently choose the most relevant condition for each step in the diffusion process, thereby enhancing the quality of the conditional image fusion. 
experiments: 
 Datasets. We conduct experiments in three image fusion scenarios: multi-modal, multi-focus, and multi-exposure image fusion. For multi-modal image fusion task, we conducted experiments on the LLVIP [41] dataset and referred to the test set outlined in Zhu et al. [34] . For MEF and MFF, our testing procedure followed the test setting in MFFW dataset [42] and MEFB dataset [43] , respectively. Additionally, we test our method on the TNO dataset and Harvard medical dataset to assess our method's performance within the multi-modal fusion domain, detailed in App. B and H. Implementation Details. Our method utilizes a pre-trained diffusion model as our foundational model [44] . This model was directly applied without any subsequent fine-tuning for specific task requirements during our experiments. The experiments are conducted on Huawei Atlas 800 Training Server with CANN and NVIDIA RTX 3090 GPU. Experimental settings are shown in App. A.Evaluation Metrics. We evaluated the fusion results in both quantitative and qualitative. Qualitative evaluation primarily hinges on subjective visual assessments conducted by individuals. We expect that the fused image will exhibit rich texture details and abundant color saturability. Objective evaluation primarily focuses on measuring the quality assessments of individual fused images and their deviations from the source images. For different task scenarios, the different evaluation metrics used, specifically, we employ six metrics including Structural Similarity (SSIM), Mean squared error (MSE), correlation coefficient (CC), peak signal-to-noise ratio (PSNR), modified fusion artifacts measure (Nabf). In the MFF and MEF tasks, considering the different task scenarios, we employ standard deviation (SD), average gradient (AG), spatial frequency (SF), and sum of the correlations of differences (SCD) for evaluation metrics. For multi-modal image fusion, we compare our method with the state-of-the-art methods: Swin-Fusion [32] , DIVFusion [45] , MUFusion [46] , CDDFuse [30] , DDFM [24] , Text-IF [47] , and TC-MoA [34] and on the LLVIP dataset. More datasets and comparison methods are shown in App. B. Note that our method, like DDFM, does not require additional tuning. Qualitative Comparisons. Furthermore, the incorporation of the basic condition and enhanced conditions enables effective preservation of the background and texture. This comparison underscores our model's efficacy in image fusion, resulting in outstanding visual outcomes. As shown in Fig. 3 , our method showcases superior visual quality compared to other approaches. Specifically, our method excels in preserving intricate texture details well lid in low light (Fig. 3 red box). Although TC-MoA and MUFusion approach our method in retaining details, they exhibit visible artifacts, blur, and low contrast-characteristics absent in CCF (Fig. 3 , green box). CCF exhibits the highest contrast, the clearest details, and the most information content, further highlighting its superiority in preserving texture details. Its excellent detail retention and clear background generation further demonstrate the effectiveness of our proposed method.For multi-focus image fusion, we compare our CCF with five general image fusion methods: U2Fusion [31] , DeFusion [33] , DDFM [24] , Text-IF [47] , and TC-MoA [34] .Quantitative Comparisons. We employ four quantitative metrics to evaluate our model, as shown in Table 2 (left). Our method significantly outperforms the comparison methods, achieving SOTA across all metrics. Specifically, the SD is 11.19 higher than the suboptimal value, indicating higher contrast and clearer images. The SCD is 0.11 higher than the suboptimal value, suggesting a lower error between source images and fused images. The AG and SF also rank first demonstrating the retention of more texture details. These results showcase that our method effectively preserves details from the source images and produces high-quality fused images.Qualitative Comparisons. As illustrated in Fig. 4 , our proposed method demonstrates outstanding visual performance, particularly in preserving intricate details. We have carefully selected specific conditions that allow our approach to effectively handle the blurring caused by multi-focus scenes while retaining the original image's lighting and color information. In comparison, other DDPMbased methods such as DDFM are unable to achieve the same level of effectiveness as our approach.In Fig. 4 (red), our method excels in preserving the details of the watch hand. The closest result to ours is U2Fusion, but it loses texture and color fidelity, appearing blurry, in Fig. 4 (green). In short, our method performs well in maintaining both color and authentic details.For multi-exposure image fusion, we compare our model with five general image fusion methods, i.e., U2Fusion [31] , DeFusion [33] , DDFM [24] , Text-IF [47] , and TC-MoA [34] .Quantitative Comparisons. As demonstrated in Table 2 (right), our method achieved SOTA on the MEF index, analogous to the results observed for the MFF task. Notably, the metrics SD, AG, and SF signify the highest image quality, while SCD exhibits the highest correlation. Each of these metrics attained state-of-the-art performance levels, underscoring the efficacy of our approach. This consistent performance across multiple metrics illustrates the robustness and versatility of our method in enhancing image quality and fidelity.Qualitative Comparisons. Fig. 4 demonstrates the excellent visual performance of our method.Our approach effectively addresses the issue of overexposure while preserving crucial details. In contrast, DDFM, which relies on finding the middle value of two images, struggles to maintain texture details. Similarly, Text-IF tends to result in higher average brightness, which can lead to content loss in overexposed scenes. Defusion and TC-MoA exhibit similar visual performance, with more blurred edges compared to our method. In comparison, our method strikes a balance between these challenges, resulting in superior visual fidelity and saturation compared to other existing methods. The task-specific conditions can be manually selected. In this section, we use the detection condition as an example. The detection model employed is YOLOv5 [48] , trained on the LLVIP dataset. We randomly select 287 images from the LLVIP test dataset to validate our method with the detection condition. Before adding the detection condition, the fused image achieved mAP.5 = 0.737, mAP.5 : .95 = 0.509, and a recall of 0.737. After incorporating the detection condition, the mAP.5 increased by 0.049 to 0.907, mAP.5 : .95 increased by 0.054 to 0.563, and recall significantly improved to 0.832. Fig. 5 visualizes several cases detected using YOLOv5. The fused images with the detection condition exhibit higher confidence (columns 1 and 2) and recall (columns 1, 2, and 3). This demonstrates that task-specific conditions can enhance the performance of fused images in downstream tasks. By integrating task-specific conditions, the fused image can be precisely aligned with the demands of various applications, enhancing the overall effectiveness and utility of the generated images. More detail shows in App. F.Numerous ablation experiments were conducted to assess the effectiveness of different components of our model. The aforementioned four metrics were utilized to evaluate fusion performance across different experimental configurations. The quantitative results are presented in App. G.To validate the effectiveness of the condition bank, we systematically add basic and enhanced conditions individually and then verify the effectiveness of SCS. Fig. 6 illustrates a gradual variation in performance metrics with the progressive addition of conditions.In Fig. 6 (a), using only the basic condition results in image fusion. When all enhanced conditions are injected together without SCS, the metrics all reduced (Fig. 6 (b)), likely due to the conflict between different conditions, as evidenced by the messy lines and random noise in Fig. 6(d ). This demonstrates that injecting conditions unevenly without DCS leads to suboptimal results. After introducing DCS, both the metrics and visual results (Fig. 6 (e)) are well-balanced, with conditions being injected as needed during the generation process. This indicates the effectiveness of the condition bank and DCS in dynamically and appropriately selecting the conditions.Our method enables the dynamic selection of conditions, allowing for the adaptive customization of conditions for each sample. As illustrated in Fig. 1 , the statistics show the condition selection at various diffusion sampling stages. Initially, the process is stochastic due to random noise (around 0 to 1 4 T steps). As denoising progresses, the content of the image starts to be dominant in fusion, and conditions like MSE, SSIM, and SD are most frequently chosen (around 1 4 T to 3 4 T steps). As the process continues, and the fusion model tends to generate more texture details, conditions like EI and edge become more dominant (around 3 4 T to T steps). SSIM also remains selected to constrain the structure, whereas the frequency of SD selection decreases. This demonstrates the crucial role of dynamically selecting conditions during the diffusion sampling process. Our CCF adaptively decomposes diverse conflict fusion conditions into different denoising steps, which is significantly compatible with the reconstruction preferences of different steps in the denoising model. This dynamic fusion paradigm ensures appropriate condition selection at each stage of the sampling. 
conclusion: 
 In this paper, we introduce a learning-free approach for conditional controllable image fusion (CCF), utilizing a condition bank to regulate joint information with a pre-trained DDPM. We capitalize on the remarkable reconstruction abilities of DDPM and integrate them into the sampling steps. Sample-adaptive condition selection facilitates fusion in dynamic scenarios. Varied fusion images can personalize their conditions to emphasize different aspects. Empirical findings demonstrate that CCF surpasses the competing methods in achieving superior performance for general image fusion tasks. In the future, we will further explore automatic manners to distinguish basic and enhanced conditions to reduce empirical intervention, thereby enabling more robust and reliable image fusion. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We provide that in App. I.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: All the generated images are carefully checked.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have checked the related requirements.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Our method not only excels in individual settings to generate customized images but also demonstrates robust performance in general settings. In this section, we elaborate on numerous experiments for image fusion tasks to demonstrate the superiority of our method. For different tasks, the basic conditions vary. Specifically, in the VIF task, MSE [24, 46, 31, 34] serves as the basic condition, where c MSE can be expressed as ||MSE(i, x 0 ) + MSE(v, x 0 )||. In contrast, for the MEF and MFF tasks, to achieve clearer and higher fidelity images and inspired by [29, 6, 49, 50] , the basic conditions include MSE, frequency and edge conditions, as detailed in the App. C. To ensure convenience and fairness, we select eight enhanced conditions from the condition bank: SSIM, MSE, Edge, Lowfrequency, High-frequency, Spatial Frequency, Edge Intensity, and Standard Deviation while setting k = 3 across all tasks.In our evaluation using the TNO dataset, we referred to the test set outlined in the work by Tang et al. [51] We evaluated the fusion results in both quantitative and qualitative. Qualitative evaluation primarily hinges on subjective visual assessments conducted by individuals. We expect that the fused image will exhibit rich texture details and abundant color saturability. For different task scenarios, the different evaluation metrics used, specifically, we employ six metrics including standard deviation (SD), entropy (EN), spatial frequency (SF), sum of the correlations of differences (SCD), Structural Similarity (SSIM), and edge intensity (EI). In the MFF and MEF tasks, considering the different task scenarios, we employ SD, AG, SF, SCD, and MSE for evaluation metrics. For multi-modal image fusion, we compare our method with four task-specific methods: DenseFuse [52] , RFN-Nest [53] , UMF-CMGR [54] , YDTR [55] , and three general approaches U2Fusion [31] , DeFusion [33] and DDFM [24] on the LLVIP dataset and TNO dataset.Quantitative Comparisons. We employ 6 quantitative metrics to evaluate our model, as shown in Table 3 . Our method demonstrated exceptional performance across various evaluation metrics. On the TNO dataset, our method achieves the best result on the SD, EN, and SCD indicators. The result shows suboptimal performance in SF and is close to the best values. Notably, our method does not necessitate turning and holds its ground against methods requiring training. Compared with the learning-free DDFM and DeFusion, our method shows better results. This demonstrates the excellent performance of our model, achieving high performance across almost all indicators in a tuning-free model.Qualitative Comparisons. The incorporation of both basic and enhanced conditions enables effective preservation of background and texture. This comparison underscores our model's efficacy in image fusion, resulting in outstanding visual outcomes. As shown in Fig. 7 , our method demonstrates superior visual quality compared to other approaches. Specifically, it excels in preserving intricate texture details and color fidelity, such as license plate numbers (highlighted in the red box in Fig. 7 ). The DDFM and DeFusion approaches retain fewer texture details. As depicted in Fig. 7 , CCF exhibits the highest contrast, the clearest details, and the most comprehensive information content, further highlighting its superiority in preserving texture details. Its excellent detail retention ability and clear background generation further prove the effectiveness of our proposed method.High-frequency It's commonly understood that the high-frequency information in both modalities is distinctive to each modality. For instance, texture and detailed information are specific to visible images, while thermal radiation information pertains to infrared images. Specifically, we utilizes wavelet transform to extract high-frequency information from the image. For example, 2D discrete wavelet transformation with Haar wavelets to transform the input image (IM) into four sub-bands can be expressed as: whereEQUATIONLL k , HL k , LH k , HH k ∈ R H 2 k × W 2 k ×c , k ∈ [1, K],signify the averages of the input image and the high-frequency details in the vertical, horizontal, and diagonal directions, respectively, at the first-level wavelet decomposition. Denote that the high-frequency details H F (IM) from IM. So we can see the high-frequency condition below:EQUATIONwhere | • | stands for the absolute operation and F (•, •) can represent a variety of customized functions. In this paper, specifically, we employ the max function. The restored average coefficient and the high-frequency coefficient at scale k are correspondingly converted into the output at scale k -1 by employing W -1 the 2D inverse discrete wavelet transform:EQUATION)Low-Frequency Condition. Low-frequency components generally encapsulate information common to both modes. For simplicity, we directly utilize the low-frequency information obtained via wavelet transform. However, each mode may possess slightly varying amounts of low-frequency information.Our conditions allow for the adjustment of the low-frequency L F fusion ratio with hyperparameters, i.e.,EQUATIONEQUATIONEdge Condition. We aim at the fused image to retain the more intricate texture details from both infrared and visible images. To achieve this, we integrated edge conditions into the set of conditions to enhance the edge texture in the fused image. The condition is defined as :EQUATIONwhere ∇ indicates the Sobel gradient operator, which measures the texture detail information of an image.MSE Condition. We incorporate the information from the original image into the fused image as a supplementary content condition. Essentially, we utilize the image reconstruction capability of DDPM to supplement lost details and enhance the informational content pertaining to the target in the fused image, i.e.EQUATIONwhere Ψ and Ψ -1 represent the downsample and upsample operator, respectively, while y can refer to either the infrared or visible image.Totally, these conditions can be combined to govern the generation of the final fused image:EQUATIOND More Details of CCF Algorithm 1 CCF Input: : i,v Output: :EQUATIONz ∼ N (0, I)5:x 0 = xt- √ 1-ᾱtϵθ(xt-1|xt) √ ᾱt 6:for k=N,..., 1 do7: x0 ← x 0 -λ ▽ δ c k 8:end for 9: end for In the sampling process, the neural network architecture is similar to PixelCNN++ [57] , consisting of a U-Net backbone with group normalization. The diffusion step t is defined by incorporating the Transformer [58] sinusoidal position embedding into each residual block. As shown in Fig. 8 , six feature map resolutions are utilized, with two convolutional residual blocks per resolution level. Additionally, self-attention blocks are incorporated at the 16×16 resolution between the convolutional blocks.Representing a process of reverse diffusion (sampling), starting from a standard normal distribution N (0, I), T sampling steps are performed to generate the final fused image. However, the generation with unconditional DDPM [35] is uncontrollable. Therefore, conditions selected from the condition bank we constructed are introduced to control the sampling process. More precisely, the conditions can rectify each step's estimation of x 0 . Finally, after the T step, the final fused image can be generated. The algorithm of CCF is presented in Algorithm 1.We follow the recent works [30, 24, 46] , and adopt the eight most commonly used constraints as conditions in this paper. However, as shown in Table 4 , our approach is not restricted to these constraints; fewer conditions can be randomly selected (here we select SSIM, MSE, Edge, and SD), and additional enhanced conditions [31, 34] (Correlation Coefficient (CC), Multi-Scale Structural Similarity (MS-SSIM), the Sum of the Correlations of Differences (SCD), and the Visual Information Fidelity (VIF)) can also be incorporated. Additionally, we evaluate the runtime across different numbers of conditions to assess their impact on efficiency. While adding more conditions slightly improves performance, it also increases inference runtime.Fig. 9 illustrates the selection of enhanced conditions at each denoising step in the same location under both daylight and nighttime scenarios, representing a typical rapidly changing environment. It demonstrates that CCF can adapt to these changing environments by selecting different conditions at various denoising steps to respond effectively to dynamic environmental characteristics.The task-specific conditions are incorporated to guide the fusion process throughout the denoising procedure. For instance, we take the Euclidean distance of features extracted by the object detection model as the detection condition. In our experiments, we employ YOLOv5, which is pretrained on the visible modality, to extract features from both the estimated x 0 at each step and the visible image. The Euclidean distance is minimized in the inverse diffusion process iteratively. Consequently, the final fused image progressively integrates the object-specific information, enhancing the fusion performance. We provide additional visualizations of the detection conditions, as shown in Fig. 10 , directly fused images appear to have a lot of missed detection in raw 1-4 and the detection scores are generally higher. The last row of Fig. 10 demonstrates a reduction in false-positive boxes when the detection condition is applied.Table 5 also supports this, indicating that metrics such as mAP.5, mAP.5:95, and recall have increased remarkably, showing that our method can customize conditions to fit the downstream task effectively. By tailoring conditions to specific tasks, our approach effectively improves the applicability of fused images for particular applications, as evidenced by the improved detection metrics. We further explored the use of different levels of features obtained with YoloV5 as detection conditions. Specifically, we evaluate in three scales and each of the scales is evaluated individually, and all features combined involve using all three scale features simultaneously, averaged, to add task-specific perception to enhance task performance. Furthermore, Table 5 indicates that there is no significant difference between using different features. The constraint primarily enhances the implicit expressive ability of the image, enabling it to better adapt to downstream tasks.The quantitative results of the ablation study compare different scenarios: without DDPM, with only basic conditions, with enhanced conditions but without SCS, and the complete CCF results.With the reconstruction capability of DDPM, the fusion performance significantly improved after the conditions were integrated. Some metrics for the enhanced conditions without SCS show a decline due to conflicts among the various focusing aspect conditions. However, after incorporating SCS, most metrics improve, demonstrating that SCS can enhance the image fusion process, producing high-quality images. Medical image fusion (MIF) [59] experiments were conducted to validate the efficacy of our proposed method using the Harvard Medical Image Dataset [60] . We visualized fused medical images, focusing on MRI-SPECT, MRI-CT, and MRI-PET fusion, with the results presented in Fig. 11 . Observing our fused images, it is evident that the preservation of both the skull's shape and intensity is notably well-maintained. Within the brain region, the CCF method effectively combines intricate details and structures from the two modalities.Even though the proposed CCF model achieves superior performance over existing methods and demonstrates advanced generalization with dynamic condition selection, there are still some potential limitations. We provide a condition bank, but it needs to be constructed empirically. Most of the conditions are inspired by Image Quality Assessment (IQA), but classifying these conditions is challenging due to the complexity of IQA. Therefore, it is necessary to propose a method to automatically classify the conditions, reducing empirical intervention. Additionally, our method relies on a pre-trained diffusion model, which limits its efficiency and makes the generation process timeconsuming. Exploring more effective sampling processes would be valuable to improve efficiency and further enhance the model's performance. For potential social impact, it is difficult to ensure the effectiveness of selected conditions for all scenarios, which may be risky in high-risk scenarios such as medical imaging and autonomous driving.Answer Justification: Our research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper proposes a method that enables dynamic conditional controllable fusion for image pairs with reported excellent performance.

The authors provided satisfactory responses and convinced
- Reviewer spMU to increase the score to Weak Accept
- Reviewer PorP to increase the score to Borderline Accept
- Reviewer y2va to increase the score to Accept
- while Reviewer ghie maintains Borderline Accept.

Thus, for this work there is an acceptance consensus.

After reading the paper and carefully checking the reviews and the authors' responses the ACs agree with the reviews that the present work makes important contributions and is of interest for the community.

The authors are invited to further refine their paper for the camera ready by including (part of) information/details from their responses to the reviewers' comments.

Reviews: review: paper_summary: paper_summary: This paper proposes a Conditional Controllable Fusion framework called CCF, effectively addressing the issue that existing data-driven fusion methods struggle to adapt to all scenarios. The authors conducted extensive experiments to demonstrate the effectiveness of the CCF. This manuscript is standardized and the writing is fluent.
strength_weakness: strength_weakness: 1. How was the Selection frequency map in Figure 1 obtained? Is it based on real data or simulated?
2. It is not clear exactly what iterative refinement of sampling refers to. Why does sampling need to be iteratively refined? The sampling iteration is derived from the diffusion model DDPM and how is it related to the proposed CCF?
3. What are the advantages of using the DDPM for controlled image fusion? In other words, how do diffusion models contribute to the adaptability and controllability of image fusion?
4. Multiple sign ambiguities. Including but not limited to: 1) $c$ denotes both the channels and the given condition in Eq. (5). 2) What does the $f_{\theta}$ refer to? 3) What is the $s_{\theta}$?
5. How are task-specific conditions incorporated into DDPM? Lack of specific implementation details.
6. The theoretical discussion of adaptive customization of conditions for each sample is limited to sections 4.1 and 4.2, lacking specific customization procedures or visual case studies. Consequently, the credibility of this contribution in terms of condition customization is compromised.
7. The authors chose 8 enhanced conditions, i.e., SSIM, Content, Edge, Low-frequency, High-frequency, Spatial Frequency, Edge Intensity, and Standard Deviation enhancements. However, their selection lacks a strong foundation or rationale.
questions: questions: 1. Could you kindly elucidate the methodology employed to generate the Selection frequency map depicted in Figure 1? Real or simulated?
2. Why Use DDPM to achieve conditional image fusion?
3. Can you give an example to explain the specific process of condition selection in the process of sampling iteration?
limitations: limitations: Please refer to "Weakness".
paper_summary: paper_summary: This paper proposes a diffusion-based image fusion method with adaptive fusion conditions. It aims to solve the drawback of existing method, i.e., the application of distinct constraint designs tailored to specific scenes. This method builds a condition bank with basic, enhanced, and task-specific conditions. Then, it employs specific fusion constraints based on these conditions for each individual in practice. The proposed method is tested on various image fusion tasks, including multi-modal, multi-exposure, and multi-focus image fusion.
strength_weakness: strength_weakness: 1. The basic and enhanced conditions are some widely used image fusion constraints, and the task-specific condition is based on the feature extracted by a downstream network. Thus, the main contribution lies in the design of the gate of conditions. However, the reason why is the gate defined in this way and the principle behind this definition not fully explained in detail.
2. It focuses on the problem that such data-driven fusion methods are hardly applicable to all scenarios, especially in rapidly changing environments and source images with dynamic differences. However, the experiment fails to show the advantages in these scenarios.
3. This method states that multiple conditions can be considered, but the actual considerations are limited, such as the SSIM-based enhanced condition and detection-based task-specific condition.
4. The article basically describes the process of the method, but the motivation, principle, and details of the method are not clear and explicit enough.
questions: questions: 1. What does $s_\theta(x_t,t)$ in Eq. (10) represent? And what is the form of $C$ in Eq. (10)? As mentioned in the multiple types of conditions, some conditions are in pixel level, such as MSE, SSIM, feature similarity, while some is denoted with the characteristic, such as SD. The form is not unified, and the characteristic-based condition may have some effect on the content.
2. In the specific conditions, how to build the constraint between F(x_0) and F(V,I), as the first term takes a single image as input while the last term take the two source images as the input.
3. For the task-specific condition, the extracted features with the task network is used for constraint. Why not use the final performance of the task for constraint?
limitations: limitations: 1. The first and second contributions are essentially the same.
2. The conditions discussed in the Appendix are almost different basic conditions. And the results of MSE are a bit strange.
3. It lacks the experiment on challenging and dynamic scenarios.
paper_summary: paper_summary: This paper proposes a novel Controllable Condition Fusion (CCF) framework that utilizes a pre-trained diffusion model to achieve dynamic and adaptive condition selection without requiring specific training for general image fusion tasks. The authors presented a conditional bank conducted by various conditions to fit diverse scenarios. The conditions can be dynamically and adaptively selected according to the diffusion step, allowing CCF to conditionally calibrate the fused images step by step for each individual sample. Experimental results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: (1)This paper introduces a controllable condition image fusion model. However, this ‘condition’ is not the common guidance condition, such as sketch, location map, pose image, etc, but evaluation metrics, such as SSIM, Edge Intensity et al. Therefore, suggesting that authors give a clear explanation or definition of the condition in Section Introduction.
(2)An important comparison is missing. From the first impression, the proposed method is to leverage the reconstruction capability of DDPM. However, the article did not demonstrate that the fusion process utilized the reconstruction capability of DDPM, it is important to prove it systematically.
(3)One of the core innovations in this paper is the Conditional Bank, which contains three types of conditions: basic fusion conditions, enhanced fusion conditions, and task-specific fusion conditions. However, there lacks a clear definition of three conditions to identify the difference between these conditions. 
(4)The paper contains some small mistakes in symbol, such as Line 170 theta, the lack of a subscript for eq 6 \epsilon and missing the parentheses. Additionally, there is indistinct use of "x0|t" and "x0" in Line 134. Overall, the work is very readable, but the problems with the writing should be corrected.
questions: questions: Please refer to the weaknesses above.
limitations: limitations: The authors have discussed the limitations of this work. And I don’t think there are any direct negative societal implications. Other limitations and opportunities for improvement are addressed in my responses to previous questions.
paper_summary: paper_summary: The paper proposes a controllable conditional image fusion method. This method enables dynamically controllable fusion for each image pairs. The core idea is to empirically construct a conditional bank and dynamically select different control conditions during the diffusion fusion process. The method is suitable for image fusion tasks of different modalities and can also perform controlled fusion given various downstream tasks, such as object detection.
strength_weakness: strength_weakness: - The section Sampling-adaptive Condition Selection (SCS) is mainly derived from "GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks", which is cited in the paper, but some of the formulas are not clearly expressed. For example, in Eq 11, Gate(C) has C on the left, but the condition C is not contained on the right. Another example is Line 170, Li(t) = E[wi(t)Li(t)]? Additionally, there are some typos, like Line 170, theta.  
     Therefore, understanding the routing process without reading the GradNorm paper might be difficult.

- There are some issues with the writing and formatting, such as the order of appearance of Tab1 and Tab2, and the captions for the tables do not specify the task for each tab (e.g., Multi-Focus, Multi-Modal, etc.) in the quantitative comparison results. The captions for the images also have this issue and need improvement.

- There are too many empirical conditions, which may require different settings under various circumstances.
questions: questions: - In Fig1, the terms “Random”, “Content”, and “Detail” in the center of the diagram need to be clearly explained.

- In Fig2, Top-K Routing is not contained in the main text.

- In Fig3, the enlarged section does not seem to be as good as methods like Swinfusion and Text-IF.

- Line 193, “the feature extracted by a detection network F = D(x)”—what scale/level of feature serves as a good guide? Are there any selection criteria?

- For basic conditions, are the high-frequency information of both modality images (for example,  infrared and visible images) extracted?

- With so many conditions, low-frequency doesn’t seem as important as shown in Fig.1.  Additionally, are there any other empirical conditions with potential?

- In Fig8, the authors mention the network structure, but in Line 493, they state that they used “a pre-trained diffusion model.” So, this network structure should be derived from the paper on the pre-trained model? If so, a citation needs to be provided in the diagram.
limitations: limitations: - The conditional bank is manually designed, and different experiments likely require different fine-tuning. This limits the potential of the proposed method.
- It is necessary to specify the runtime, as time is usually a key metric for diffusion model-based methods.

Comments: title: Thank you for the detailed response.
Great thanks for the positive reply!
comment: Thank you for your positive feedback! We are very grateful that the reviewer recognizes our responses and work. Your insightful comments have greatly improved our work and inspired us to research more. If you have any further questions, please let us know. We would be more than happy to clarify more about our paper and discuss it further with you.
Great thanks for your support! We sincerely appreciate your constructive comments and will carefully revise our paper accordingly.
Thank you for the rebuttal. After reviewing the other reviewers' comments and your response, some of my concerns and questions have been addressed. I lean to increasing my final score to 5.
Thank you for the detailed response. My concerns have been well addressed. Accordingly, I have raised my rating and highly recommend the authors include those discussions on adaptive conditions in the revised version.
Dear Reviewer PorP,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission, and we thank a lot for your constructive comments on our paper. Given that the discussion period is around the corner, would you please kindly let us know if our response has addressed your concerns? If you have any further questions, please let us know. We would be more than happy to clarify more about our paper and discuss it further with you.

Best regards,

Authors of Submission 35
We sincerely thank the reviewer for the positive reply. Meanwhile, we are very grateful that the reviewer recognizes our responses and work, while increasing the final rating to 'accept'. Great thanks again!
Thanks for your responses. My concerns have been well solved. Based on the novel design of conditional controllable fusion framework and sufficient experiments, I tend to increase my score to 7.