Test-Time Dynamic Image Fusion

Authors: bing  cao
yinan  xia
yi  ding
changqing  zhang
qinghua  hu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-22 22:13:25

Abstract: The inherent challenge of image fusion lies in capturing the correlation of multi-source images and comprehensively integrating effective information from different sources. Most existing techniques fail to perform dynamic image fusion while notably lacking theoretical guarantees, leading to potential deployment risks in this field. Is it possible to conduct dynamic image fusion with a clear theoretical justification? In this paper, we give our solution from a generalization perspective. We proceed to reveal the generalized form of image fusion and derive a new test-time dynamic image fusion paradigm. It provably reduces the upper bound of generalization error. Specifically, we decompose the fused image into multiple components corresponding to its source data. The decomposed components represent the effective information from the source data, thus the gap between them reflects the \textit{Relative Dominability} (RD) of the uni-source data in constructing the fusion image. Theoretically, we prove that the key to reducing generalization error hinges on the negative correlation between the RD-based fusion weight and the uni-source reconstruction loss. Intuitively, RD dynamically highlights the dominant regions of each source and can be naturally converted to the corresponding fusion weight, achieving robust results. Extensive experiments and discussions with in-depth analysis on multiple benchmarks confirm our findings and superiority. Our code is available at https://github.com/Yinan-Xia/TTD.

Main paper: 
 introduction: 
 Image fusion jointly integrates complementary information from multiple sources, aiming to generate informative and high-quality fused images. With superior scene representation and enhanced visual perception, image fusion significantly benefits downstream vision tasks [1, 2] . Typically, image fusion can be categorized into multi-modal, multi-exposure, and multi-focus image fusion tasks. Multi-modal image fusion encompasses Visible-Infrared image Fusion (VIF) and Medical Image Fusion (MIF). For VIF [3] [4] [5] , infrared images effectively highlight thermal targets especially under extreme conditions, while visible images provide texture details and ambient lighting. For MIF [6, 7] , different medical imaging modalities emphasize various focal areas, enhancing diagnostic capabilities. Multi-exposure image Fusion (MEF) [8] [9] [10] bridges the gap between high dynamic range (HDR) natural scenes and low dynamic range (LDR) pictures, ensuring better detail preservation in varying lighting conditions. Multi-Focus image Fusion (MFF) [11] [12] [13] aims to produce all-in-focus images by combining multiple images focused at different depths.Numerous image fusion methods have been introduced, which can be mainly grouped into traditional techniques and deep learning approaches. Traditional image fusion methods, such as multi-scale decomposition-based models [14, 15] and sparse representation-based methods [16] , rely on mathematical transformations to fuse images in the transform domain [17] . In contrast, deep learning-based methods employ data-driven schemes to fuse multi-source images, including convolutional neural network (CNN) based methods [11, 18] , generative adversarial network (GAN) based methods [19, 4] , and transformer-based methods [20] . The effectiveness of image fusion algorithms hinges on two critical factors: feature extraction [21] and feature fusion [22] . The aforementioned methods strive to achieve high-quality fused images by learning effective uni-source or multi-source feature representations through complex network structures or feature decomposition schemes. However, they often overlook the complexity of the real world, which necessitates dynamic feature fusion.Recently, some works have highlighted the importance of dynamism in image fusion. For instance, [23] pioneered the combination of image fusion with a Mixture of Experts (MoE), dynamically extracting effective and comprehensive information from the respective modalities. [24] utilized task-specific routing networks to extract task-specific information from different sources with dynamic adapters. Despite their empirically superior fusion performance, these dynamic fusion rules mainly rely on heuristic designs, lacking theoretical guarantees and interpretability. Moreover, they potentially lead to unstable and unreliable fusion results, especially in complex scenarios.To address these issues, we reveal the generalized form of image fusion and propose a new Test-Time Dynamic(TTD) image fusion paradigm with a theoretical guarantee. Given that the fused image integrates comprehensive information from different sources, it can be obtained by weighting the effective representation of each uni-source. By revisiting the relationship between fusion weights and image fusion losses from the perspective of generalization error [25] , we decompose the fused image into multiple uni-source components and formulate the generalization error upper bound of image fusion. Based on generalization theory, we for the first time prove that dynamic image fusion is superior to static image fusion. The key to enhancing generalization lies in the negative correlation between fusion weight and uni-source component reconstruction loss. As fusion models are trained to extract complementary information from each source, the decomposed components represent the effective information from the source data. Thus, the fusion components can be estimated by source data with the fusion model, the losses of which represent the deficiencies of the source in constructing fusion images. Accordingly, we derive a pixel-level Relative Dominablity (RD) as the dynamic fusion weight, which theoretically enhances the generalization of the image fusion model and dynamically highlights the changing dominant regions of different sources as shown in Fig. 1 . Extensive experiments on multiple datasets and diverse image fusion tasks demonstrate our superiority. Overall, our contributions can be summarized as follows:• This paper first theoretically proves the superiority of dynamic image fusion over static image fusion and provides the generalization error upper bound of image fusion by decomposing the fusion image into uni-source components provably. The proposed generalization theory reveals that the key to reducing the upper bound lies in the negative covariance between the fusion weight and uni-source reconstruction loss.• We proposed a simple but effective test-time dynamic fusion paradigm based on the generalization theory. By taking the uni-source's Relative Dominability as the dynamic fusion weight, we theoretically enhance the generalization of the image fusion model and dynamically emphasize the dominant regions of each source. Notably, our method does not require additional training, fine-tuning, and extra parameters.• We conduct extensive experiments on multi-modal, multi-exposure, and multi-focus datasets.The superior performance across diverse metrics demonstrates the effectiveness and applicability of our approach. Moreover, an additional exploration of the gradient in constructing fusion weight demonstrates the reasonability of our theory and its expandability. 
related works: 
 Image Fusion aims to integrate complementary information of diverse source images. For instance, [26] utilize autoencoders to extract multi-source features and fuse them using a designed strategy. GAN-based methods [19] and transformer-based methods [20] also achieved significant progress. [27] introduced the denoising diffusion probabilistic model (DDPM) to image fusion. [28] and [29] achieve considerable fusion performance by decomposing image features into high-frequency and low-frequency components. In addition to these static image fusion methods, [23] used a Mixture of Experts (MoE) to dynamically assign fusion weights, while [24] utilized dynamic adapters to prompt various fusion tasks within a unified model. These approaches mainly focus on obtaining promising feature representations. Although some existing works have explored dynamic image fusion, the lack of theoretical guarantees may result in instability and unreliability in practice.Multimodal Dynamic Learning Although dynamic fusion is not fully studied in existing image fusion works, numerous methods have leveraged multimodal dynamic learning at the decision level [30] . For example, Han et al. [31] assigned dynamic credible fusion weights to each modality at the decision level for robust evidence fusion. Xue et al. [32] employs a Mixture of Experts to integrate the decisions of multiple experts, Zhang et al. [33] combined decision level fusion weight with uncertainty to conduct a credible fusion. Despite the wide exploration of dynamic fusion at the decision level, there is still insufficient research on dynamic fusion at the feature level with theoretical guarantees. In this paper, we focus on the dynamic nature of image fusion, theoretically prove that dynamic fusion is superior to static fusion, and propose a provable feature-level dynamic fusion strategy.The detailed pipeline for inference is shown in Fig. 6 (c ). In stage 1 (black dashed line), we feed each unisource image individually into the frozen encoder and decoder to acquire the respective decomposed uni-source components. Then, we calculate the RDs according to the colored line in Eq. ( 6) (c). In stage 2 (solid line), we feed multi-source images into the encoder and get their corresponding features. Then, we fuse features by multiplying the RDs to the respective features and adding them up. Finally, the fused feature is fed into the decoder for the final fusion results. 
method: 
 Given EQUATIONAdditionally, we define the loss function for image fusion tasks, where ∥ • ∥ represents any distance norm. The loss function is given by:EQUATIONGeneralization Error Upper Bound. In machine learning, the concept of Generalization Error Upper Bound refers to the theoretical limit on a model's performance when applied to unseen data (D) [34] . A smaller upper bound indicates better-expected performance on data from an unknown distribution. For image fusion tasks, the Generalization Error (GError) of a fusion model f can be defined as: Considering the GError of image fusion model [35] [36] [37] , ℓ(I F , x) can be further deduced asEQUATIONℓ(I F , x) ≤ M m=1 ∥ M i=1 ω (i) • D(E (i) (x (i) )) -x (m) ∥. Therefore, the fused image I F is decomposed into M uni-source components {D E (i) x (i) |i = 1..M }.Based on Eq. ( 3) and ( 8), we have: Theorem 3.1 (Decomposition of Generalization Error). The GError for multi-source image fusion model f can be decomposed into a linear combination of each uni-source component reconstruction loss under the condition that M m=1 ω (m) = 1, the detailed proof is given in Appendix A.1:EQUATIONLet ℓ (m) = ∥D E (m) x (m) -x (m) ∥, which represents the reconstruction loss between a unisource component and its corresponding uni-source image. The term Cov(ω (m) , ℓ (m) ) denotes the covariance between ω (m) and ℓ (m) . The essence of reducing generalization error lies in achieving the lowest possible fusion loss. By leveraging the triangular inequality properties of distance norms within the fusion loss, we can deduce that the GError is bounded by the covariance term and the distance between each uni-source component and its source image. It is noteworthy that f (x (m) ) remains constant during the test phase, emphasizing that the pivotal factor in reducing Generalization Error Upper Bound (GEB) lies in the covariance between ω (m) and ℓ (m) .Superiority of Dynamic Image Fusion over Static Image Fusion. Most existing image fusion approaches reduce GEB by minimizing ℓ (m) , indicating an effort to enhance the quality of uni-source feature representations. However, they often overlook the intrinsic significance of fusion weight ω (m) . Fusion strategies employed in static image fusion encompass methods such as maximum, minimum, addition, ℓ 1 -norm, etc. Nevertheless, none of these fusion weights exhibit a correlation with uni-source reconstruction loss, i.e. Cov(ω(m)static , ℓ (m) ) = 0. During the test phase, ℓ (m) remains constant. If we have: Cov(ω (m) dynamic , ℓ (m) ) < 0 for all source images, we can derive the conclusion:EQUATIONThis indicates that for a well-trained image fusion model, a dynamic fusion strategy can bring better generalization than a static fusion strategy.Relative Dominablity. Recalling the Eq. ( 4), the negative correlation between fusion weight and the reconstruction loss ℓ (m) provably reduces the generalization error upper bound. Therefore, we introduce a pixel-level Relative Dominablity (RD) as the fusion weight for each source, which exhibits a negative correlation with the reconstruction loss of the corresponding fusion component. Since fusion models are trained to extract complementary information from each source, the decomposed components of fusion images represent the effective information from the source data. Thus, the uni-source components can be estimated from source data using the fusion model, with the losses representing the deficiencies of the source in constructing fusion images. For instance, in a given region, the larger the pixel-wise fusion loss between the reconstructed component and its corresponding uni-source image, the smaller its contribution to image fusion. Intuitively, using RD as the dynamic fusion weight can capture the dominance of each source in image fusion and enhance its advantages in constructing fusion images. Theoretically, according to Thm. 3.1, negatively correlated with the pixel-wise fusion loss, RD effectively demonstrates the dominance of each source. Notably, considering the relative nature of multi-source image fusion, the sum of the RDs of different sources for the same pixel should be one. Consequently, by establishing a negative correlation with the loss and implementing normalization, we can obtain the Relative Dominablity of each source for a certain sample as follows:EQUATIONIn addition, we present the algorithm and test pipeline of our dynamic fusion strategy in Appendix B.1. 
experimental setup: 
 Datasets. We evaluate our proposed method on four image fusion tasks: Visible-Infrared Fusion (VIF), Medical Image Fusion (MIF), Multi-Exposure Fusion (MEF), and Multi-Focus Fusion (MFF).• VIF: For VIF tasks, we conduct experiments on two datasets: LLVIP [38] and MSRS [17] . For LLVIP datasets, we randomly select 70 samples from the test set for evaluation. • MIF: We conduct experiments on the Harvard Medical Image Dataset, following the test setting in [29] . • MEF: Following the setting in [24] , we verified the performance of our method on MEFB [39] dataset. • MFF: For the MFF task, we evaluate our method on MFI-WHU datasets [40] , following the test protocol in [24] . As a test-time adaption approach, TTD performs adaptive fusion solely during testing, without additional training and training data.Competing Methods. For VIF and MIF tasks, we evaluated 12 state-of-the-art methods, encompassing both DenseFuse [26] , CDDFuse [29] , U2Fusion [22] , DDFM [27] , DeFusion [41] , PIAFusion [17] , DIVFusion [42] , MUFusion [43] , IFCNN [18] , and SwinFuse [20] , and TC-MoA [24] . For MEF and MFF tasks, we compared our methods with general image fusion methods and task-specific image fusion methods. Notably, among these methods, only DDFM is training-free, and other methods are all pre-trained on VIF datasets. In experiments, we apply our TTD to CDDFuse (CDDFu-sion+TTD), PIAFusion (PIAFusion+TTD), and IFCNN (IFCNN+TTD), separately. Our experiments are conducted on Huawei Atlas 800 Training Server with CANN and NVIDIA RTX A6000 GPU.Metrics. We selected several evaluation metrics from three aspects [44] , including • information theory: entropy (EN) [45] , cross entropy (CE), the sum of the correlations of differences (SCD) [46] ,• image feature: standard deviation (SD), average gradient (AG) [47] , edge intensity (EI) and spatial frequency (SF) [48] , and • structural similarity: structural similarity (SSIM) [49] .Visible-Infrared Fusion. Tab. 1 reports the performance of competing approaches and TTDapplied methods on LLVIP and MSRS datasets for 7 metrics. Notably, by applying our TTD, the previous methods have improved on most of the indicators. Also, our TTD strategy outperforms other traditional static methods, training-free method DDFM, and data-driven dynamic strategy TC-MoA, achieving the SoTA performance on most metrics. Moreover, with particularly high values in SD, AG, EI, and SF, our TTD ensures that fusion images maintain exceptional contrast and detailed texture, highlighting its efficacy in preserving quality. The outstanding performance on EN and SCD indicates our fusion results embed more information and contain abundant edge information from the source images. Although our approach is a test time adaptation strategy that does not require training, the designed dynamic weights based on theoretical principles have led to outstanding fusion performance, achieving SoTA results on VIF tasks.Medical Image Fusion. We report the comparison results on three MIF scenarios: MRI-CT, MRI-PET, and MRI-SPECT. As depicted in Tab. 3, 6 and Tab. 7 in Appendix C.3, our method yields competitive performance on seven evaluation metrics. Specifically, our TTD enhances EN, AG, and SSIM, indicating ample gradient information and structural details in the fusion results. The significant improvements in SD, EI, and SF highlight our high definition and texture quality compared with the competing methods, making it exceptionally competitive.Multi-Exposure and Multi-Focus Image Fusion. In the comparisons on MEF and MFF tasks, we applied our TTD to the general fusion method IFCNN. We compared it with other general fusion methods and task-specific fusion methods. As depicted in Tab. 2, TTD outperforms existing general fusion methods and task-specific methods in terms of SD, EI, AG, and SF. This indicates that TTD produces fused images with clear, abundant edges and exceptional sharpness. Furthermore, the superiority in EN and CE suggests that our TTD enables the baseline to preserve more advantages from different sources. Visible-Infrared Fusion. As shown in Fig. 3 and Fig. 8 in Appendix C.1, compared with existing methods on the LLVIP dataset, our TTD effectively combines comprehensive information from different sources, leading to a significant visual performance. Specifically, the fusion result not only preserves the texture details and edge information of visible images but also incorporates high-quality thermal imaging contrast of infrared images. Additionally, as mentioned in the qualitative analysis, our fusion images exhibit high fidelity and clear contrasts, showing consistent superiority in terms of image quality. These experimental results demonstrate the effectiveness of our TTD.Medical Image Fusion. For the MIF task, we present qualitative comparisons of the MRI-PET fusion. As shown in Fig. 3 and Fig. 8 in Appendix C.1, it is clear that fusion images generated by our method preserve a substantial amount of structural information. Notably, our method maintains a significant portion of excellent soft tissue contrast details from MRI images and combines the quantitative physiologic information of PET images. With our TTD, the overall structural details and sharpness of the fused image are significantly enhanced. Moreover, in the regions where the high-intensity color areas of the PET image overlap with the structural information of the MRI, the detailed information from the original images is well preserved and highlighted in the fused image.Multi-Exposure and Multi-Focus Image Fusion. We also provide a comparison of fusion results for the MEF and MFF tasks in Fig. 4 and Fig. 8 in Appendix C.1. Notably, our TTD significantly enhances the clarity and sharpness of texture details. Obvously, after applying our TTD, the fused images on the MEF task exhibit higher clarity in both the foreground and background. For the MFF task, our method accurately utilizes the effective regions from both underexposed and overexposed images. Compared to other methods, our fusion results achieve more precise exposure and rich texture details, such as the cars in the garage and the textures on the walls. Additionally, our fusion method retains high-fidelity colors that are closer to the original images.Apart from the comparisons with the existing methods, we provided more ablated comparisons with baselines in Fig. 9 of Appendix C.2.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide detailed experiment setup in Sec. 4.1 and implementation in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
is negative correlation help?: 
 The negative correlation between RD and ℓ (m) is derived from Eq. ( 4) to reduce the generalization error of the image fusion model. To further validate the effectiveness of the theoretical guarantee, we compare it with a contrast setting: using a new fusion weight, which is positively correlated (PC) to ℓ (m) , to perform image fusion.As shown in the correlation comparison in Fig. 5 (b), loss-positive-correlated weights (yellow line), that conflict with our theory, lead to a decreased performance compared with static fusion (green line). As a comparison, the results of the loss-negative-correlated fusion strategy (red line), exhibit superior performance compared with both static image fusion and positively correlated fusion strategy. These experiments verify that the proposed negative correlation setting can explicitly reduce the generalization error, demonstrating the reasonability of the proposed TTD image fusion model.In this paper, we introduce the pixel-level Relative Dominablity, which indicates the advantages of each source. Treating the Relative Dominablity as dynamic fusion weight, TTD achieves an adaptive and interpretable image fusion. We provide visualizations of each source's pixel-level Relative Dominability obtained using CDD+TTD for VIF and MIF tasks, and IFCNN+TTD for MEF and MFF tasks.Visible-Infrared Fusion. As shown in Fig. 1 , it can be observed that Relative Dominablity (RD) accurately reflects the dominance of each source: in visible images, well-lit and properly exposed bright areas contain abundant brightness information, and areas like digits and characters exhibit rich texture details. In contrast, infrared images provide thermal imaging information for areas and objects in shadow that visible images cannot capture due to visual obstacles. The proposed RD effectively captures the advantageous regions of different source images and assigns larger weights to these regions, thereby achieving more reliable fusion results.Medical Image Fusion. We visualize RDs in the MRI-PET dataset. Similar findings are also apparent in the MIF task. In PET images, bright regions indicate areas of malignant cell proliferation, while MRI contains more structural information. As shown in Fig. 1 , the RDs of PET stand out in the bright areas while MRI's highlights the structural information. Guided by RD, TTD emphasizes potential lesion areas while preserving the structural information of these areas effectively.Multi-Exposure and Multi-Focus Image Fusion. For MEF and MFF tasks, the ideal outcome is that the fused images contain properly exposed or precisely focused regions from each uni-source.As shown in the visualized RD map in Fig. 1 , our TTD can effectively capture the dominant areas in different sources and assign higher RD values, i.e. dynamic fusion weight, to these regions.Downstream Tasks. To validate the effectiveness of our RD on downstream tasks, we compared our TTD with the baseline on an object detection task. Detailed results are given in Appendix C.5. Overall, with the integration of our TTD, the baseline model gains the ability to perceive dominant information dynamically. Therefore, this interpretable plug-and-play test-time dynamic adaptation fusion strategy can further improve the performance of the existing state-of-the-art methods. This further validates the effectiveness of RD in our TTD.In our TTD, the proposed pixel-level fusion weight is computed by pixel-level loss. However, some numeric losses are limited in directly obtaining the pixel-level weights, making it hard to integrate with TTD flexibly. To overcome this dilemma, we extend our TTD to a more general form and construct gradient-based Relative Dominability through any fusion losses for a more fine-grained and robust image fusion.Recalling our optimization objective of the generalization error bound, we aim for a negative correlation between the weights and losses of the same modality, i.e., establishing a correlation between losses and weights. Inspired by this positive correlation between loss value and the absolute value of its gradient for features with any convex loss function, our TTD can be further extended to a gradient-based dynamic fusion weight.Specifically, we first calculate the absolute value of gradients |G (m) | ∈ R H×W ×C of each uni-source feature. As a test-time adaption approach, TTD does not update the network parameters, meaning that the unimodal feature space remains fixed for the same baseline. For the same task scenarios, the feature patterns tend to be similar. Therefore, we can empirically select the gradient channels that well represent the advantage areas to compute RDs. Also, as illustrated in Fig. 5 (a), gradient maps of some channels (such as the 44th and the 13th channels) lack significant useful information and fail to capture advantageous regions in the original images. Therefore, we select the gradient map |g (m) | ∈ R H×W among C channels that best represent the dominance of the uni-source empirically. By replacing ℓ (m) , we can obtain the RD and the dynamic fusion weight as follows:ω (m) = RD (m) = Sof tmax(e -|g| (m)).We have also conducted comparisons of loss-based TTD, full gradient-based TTD, and best gradientbased TTD on IFCNN. For the best gradient-based TTD, we choose the 58-th gradient map to obtain the fusion weight. The results of gradient-based RD in Fig. 5 (b) demonstrate that full gradient (yellow line) may bring wrong or useless dominance information to fusion weights, leading to worse performance compared with loss-based TTD (green line). However, by selecting the empirically best gradient map (red line), our TTD provides more fine-grained dominance information compared to the global loss maps, achieving more detailed dynamic fusion with better performance.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The source code is provided in https://github.com/Yinan-Xia/TTD. 
conclusion: 
 Image fusion aims to integrate effective information from multiple sources. Despite numerous methods being proposed, research on dynamic fusion and its theoretical guarantees remains significantly lacking. To address these issues, we derive from a generalized form of image fusion and introduce a new Test-Time Dynamic (TTD) image fusion paradigm with a theoretical guarantee. From the perspective of generalization error, we reveal that reducing generalization error hinges on the negative correlation between the fusion weight and the uni-source component reconstruction loss. Here the uni-source components are decomposed from fusion images, reflecting the effective information of the corresponding source image in constructing fusion images. Accordingly, we propose a pixel-level Relative Dominablity (RD) as the dynamic fusion weight, which theoretically enhances the generalization of the image fusion model and dynamically highlights the changing dominant regions of different sources. Comprehensive experiments with in-depth analysis validate our superiority. We believe the proposed TTD paradigm is an inspirational development that can benefit the community and address the theoretical gap in image fusion research.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide all the implementation details in the Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have checked the related requirements.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No new assets are provided.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not study human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not conduct experiments human participants.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Proof of Theorem 3.1Proof. By leveraging the properties of convex functions, the loss function can be derived to the following inequality when M i=1 ω (i) = 1:EQUATIONTherefore, the fused image IF can be decomposed to M uni-source components. By taking the expectation on both sides of the inequality, we can derive the generalization error GError(f ) of the model on the unknown distribution D.EQUATIONFor simplicity, we use ℓ x(i) com, x (j) to denote ∥D(E (i) (x (i) )) -x (j) ∥.Consequently, leveraging the triangle inequality for norms and the fact that M m=1 ω (m) = 1, we have: EQUATIONHere we report the algorithm of the whole dynamic fusion strategy in Algorithm 1. To accomplish our TTD, we initially feed each uni-source image individually into the fusion network to acquire the respective uni-source components. Then we compute the pixel-wise loss between the uni-source components and their corresponding uni-source images. Finally, utilizing Eq. ( 6) we obtain the dynamic fusion weight and apply dynamic fusion accordingly.Input: x = x (m) |m = 1, 2, ..., M , Output: IF 1: for each m ∈ [1, M ] do 2: ℓ (m) = ∥D(E (m) (x (m) )) -x (m) ∥ 3: end for 4: for each m ∈ [1, M ] do 5: ω (m) = Sof tmax(e -ℓ (m) ) 6: end for 7: IF = D M m=1 ω (m) • E (m) x (m) 8: return F B.RD is adaptable to the noise condition. We simulate a noisy situation in which the visible image quality is affected by contrast perturbation. As shown in Fig. 7 , with the corruption severity level increasing, the dominant regions of visible modality are gradually reduced, while the unchanged infrared modality gains an increasing RD. Our RD effectively perceives the dominance changes. As time progresses from day to night, an intuitive observation is that the dominance of visible images gradually decreases, while the dominance of infrared images increases.RD is adaptable to different data qualities. a) The quality of images also changes with illumination. As shown in Fig. 8 , we visualized the RDs of the samples at different times in the same scenario. As it changes from day to night, the dominance of visible images gradually decreases, while the dominance of infrared images increases.b) Furthermore, to simulate the malfunction of sensors in a real scenario, we masked the infrared image randomly. As shown in Fig. 6 (a)(b), the RD of the region being masked is apparently smaller than the surrounding area, while that of the same region in the infrared image is relatively greater.As shown in the ablation study results of our TTD on four tasks in Fig. 9 , our TTD can highlight the advantageous regions in four tasks, improve contrast, and preserve detail compared with baselines. For example, in the VIF task, our method enhances the details of people and the shadow textures of trees in the fused images compared to the baseline. In the MIF task, our method maintains the bright information from PET while strengthening the texture details from MRI in the overlapping regions. In the MEF and MFF tasks, the fused images produced by our method have stronger texture details and edge information, as well as higher clarity and color fidelity, compared to the baseline fused images. Our TTD is a simple but effective method with a straightforward structure, and we analyze the effectiveness of the TTD from different aspects. We have summarized these ablated analyses here:(i) ablation study on different baselines: see Sec. 4.2, Tab. 1, and Tab.(ii) ablation study on the correlation between weight and loss: see Sec. 5.1 and Fig. 5 .(iii) ablation study on the ways to obtain weight: see Sec. 5.3 and Fig. 5 .(iv) ablation study on different forms of fusion weights. We compared different forms of fusion weight: w = 0.5 (baseline),w = Sof tmax(-ℓ), w = Sof tmax(Sigmoid(-ℓ)), Sof tmax(e -ℓ ) over IFCNN on the LLVIP dataset, results are given in Tab. 4, it shows that forms of fusion can be flexible to achieve the negative correlation between weight and reconstruction loss.(v) ablation study on the normalization of the weights: we compared three forms of normalization over IFCNN on the LLVIP, results are given in Tab. 5, indicating that as a premise of the generalization theory (see Thm. 3.1), the normalization of the weights is necessary and the ways to normalize have little impact on our method.Overall, we performed complete ablation analyses to validate the effectiveness of TTD (i), the necessity of the negative correlation between fusion weight and reconstruction loss (ii), the expandability of ways to obtain fusion weight (iii), the flexibility in the form of weights (iv), the significance of normalization (v). Here we provide the quantitative comparison results on MRI-CT, MRI-PET, and MRI-SPECT datasets in Tab. 3, 6, and 7. Our method yields competitive performance on seven evaluation metrics on the three MIF datasets.In Tab. 1, we have applied TTD to various baselines with different capabilities and all achieved consistent enhancement, TTD can even further improve the performance when combined with current state-of-the-art methods. To further validate that our TTD is effective on models with different performances, we conducted additional experiments to apply TTD on models with varying performance levels by adding random Gaussian noise to the pre-trained model (IFCNN) parameters. The results on the LLVIP dataset are given in Tab. 8, showing that the performance of the baseline decreases with increasing noise added to it. As a comparison, our TTD effectively improves all these baselines' performance, indicating the effectiveness and generalizability of our TTD on various baselines with different performances.First, we train the detection model with visible images from the LLVIP dataset. Then we employed our TTD on IFCNN and compared its performance with the baseline on the object detection task. As illustrated in Fig. 10 , the detection results of the baseline fused images exhibit missing detection for hard-recognized fast-moving blurred objects. In contrast, after applying our TTD, all objects were accurately detected. Additionally, the fused images obtained using our TTD achieved higher performance in detection tasks compared to the baseline. our DIF shows improvements over the baseline in P, R, and mAP.5:.95 metrics.The inference time of TTD is dependent on the inference time of the baseline. Since TTD is executed in two stages: in the first stage, we calculate the uni-source reconstruction loss and then compute the fusion weights; in the second stage, we perform the fusion based on the weights. As baselines perform the static fusion, the inference time of TTD is approximately double that of the baseline. We measured the average processing time per image on the test set of the LLVIP dataset. The results of the inference time are given in Tab. 9.As a test-time dynamic image fusion method, the performance of our TTD significantly depends on the performance of the baseline models. In the future, we will try to employ the dynamic fusion mechanism in the optimization of baseline models to guide fusion or design a more effective network, further improving fusion performance. Moreover, in the gradient-based TTD, we select the best gradient empirically, a more adaptive selection approach should be explored in the future. As for the potential social impact, our method performs multi-sensor information fusion, which can be applied to drones, cameras, etc., but it is hard to guarantee the effectiveness of baseline models, which may be risky in high-risk scenarios such as medical imaging.• The proofs can either appear in main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The final rating for this paper is two borderline accrpt and two accept. Overall, the reviewers' opinions on the paper were consistent and generally positive. Moreover, most reviewers indicated that the authors had addressed their previous concerns in the rebuttal. Additionally, two reviewers raised their scores after the rebuttal phase. In summary, the overall evaluation of the work is positive, and the reviewers found the authors' responses to be proactive and effective. Therefore, I am inclined to accept this work.

Reviews: review: paper_summary: paper_summary: - This paper tries to solve the image fusion task, where multi-source images are provided and one needs to extract and integrate effective information from them.
- The paper demonstrates its effectiveness on four different tasks: VIF, MIF, MEF, and MFF.
- The paper proposes a test-time dynamic image fusion method with theoretical justification.
- This paper theoretically proves the superiority of dynamic image fusion over static image fusion, and provides a generalization error upper bound.
- By using the relative domainability of each source as the dynamic fusion weight, it is able to theoretically improve the generalization of the image fusion model and dynamically emphasize the dominant regions of each source.
- This method theoretically and empirically demonstrates superiority over static fusion methods through extensive experiments on various datasets, including visible-infrared, medical, multi-exposure, and multi-focus image fusion tasks.
strength_weakness: strength_weakness: - The adaptation method heavily relies on the performance of the baseline model, which can be ineffective when the model performance is poor.
- The improvement over the non-adaptive baseline is minor.
questions: questions: -
limitations: limitations: -
paper_summary: paper_summary: This paper proposes a theoretically guaranteed new paradigm for test-time dynamic image fusion, which exploits the negative correlation between the fusion weights and the single-source reconstruction loss to reduce the upper bound of the generalization error. Extensive experiments demonstrate its effectiveness on a variety of image fusion tasks.
strength_weakness: strength_weakness: 1. Authors does not mention the change in model inference efficiency. 
2. The paper assumes that the decoder of the fusion model is a CNN model, which introduces certain limitations. If the model were a Transformer or Diffusion model, would TDD still be effective?
3. For specific fusion tasks, it is recommended to compare specific methods rather than generalized methods, e.g., multifocus image fusion tasks should compare multifocus image fusion methods.
4. Ablation lacks enough persuasiveness.
5. Table I and II are written inconsistently e.g. TDD and Ours.
questions: questions: See Weaknesses.
limitations: limitations: See Weaknesses.
paper_summary: paper_summary: This paper proposes a theoretical justification of image fusion from a generalization perspective and reduces the upper bound of generalization error by decomposing the fused image into multiple components corresponding to its source data. A new test-time dynamic image fusion paradigm TTD is further proposed with the finding that the negative correlation between fusion weight and the uni-source reconstruction loss is the key to reducing the generalization loss. Extensive experiments and discussions confirm the theory and superiority.
strength_weakness: strength_weakness: The details presentation and explanation about test-time adaption are not very clear.
questions: questions: 1. Is the idea that adding up every uni-source data linearly to get fused image reasonable enough? Are there any information only found with several data neglected in the whole process?
2. Mathematical formulas in Appendix A.1 are a little confused. More parentheses ought to be used to present clear explanations about the scopes of every mathematical symbols.
3. Is the TTD applied to every combination from sources? Most test-time adaption methods only need few data to fine-tune the model. But this paper seems to apply TTD to all source data.
limitations: limitations: This paper presents a new perspective to analysis the generalization error of the image fusion task, the details presentation and explanation are not very clear, mathematical formulas are little confused and more explanation about inference process should be provided.
paper_summary: paper_summary: This paper introduces a method for dynamically adjusting fusion weights for image pixels based on their relative dominability, calculated using pixel-wise reconstruction losses. This approach aims to minimize generalization error by considering the correlation between fusion weights and reconstruction losses.
strength_weakness: strength_weakness: 1. The claim that RD accurately captures the dominant regions of different sources without solid empirical justification for various scenarios might be overreaching. It is unclear how RD performs under various noise conditions or with sources of different qualities.

2. In Eq. (4), the fusion weights $w$ are important as they determine the contribution of each source to the total loss. However, there is no clear definition for normalizing these weights.

3.  In Eq. (8), the paper lacks discussion on the initialization of $w^(m)$  and how it affects convergence and stability during the dynamic adjustment process.

4. Figure 2 is not clear. How are the feature maps of different layers fused, and what is the impact of each of them on the overall performance of the proposed model?
questions: questions: Please see the weaknesses!
limitations: limitations: Not very adequate. The limitation is general.

Comments: title: Reminder for review
Reminder for review
comment: Dear Reviewer gbUk, I have noticed that you have not yet responded to the authors' rebuttal. I kindly urge you to engage in a discussion with the authors at your earliest convenience to help advance the review process.
Dear Reviewer gBb1, I have noticed that you have not yet responded to the authors' rebuttal. I kindly urge you to engage in a discussion with the authors at your earliest convenience to help advance the review process.
Thanks a lot for your reply. We are delighted to have addressed your concerns. We also appreciate your insightful comments, which have greatly improved our work and inspired us to research more.
The response from the author has addressed my comments. I increased my score.
Thanks a lot for your positive feedback. Your insightful comments have greatly improved our work. We sincerely appreciate your support.
Thanks for the rebuttal. The authors have addressed my concern. I intend to increase my score.