Test-Time Dynamic Image Fusion
Authors: Bing Cao
Yinan Xia
Yi Ding
Changqing Zhang
Qinghua Hu
Venue: NeurIPS 2024 poster (2024)
Keywords: Dynamic image fusion
generalization error
negative correlation
Primary area: interpretability_and_explainability
TL;DR: Dynamic image fusion is better than static image fusion.

Abstract: The inherent challenge of image fusion lies in capturing the correlation of multi-source images and comprehensively integrating effective information from different sources. Most existing techniques fail to perform dynamic image fusion while notably lacking theoretical guarantees, leading to potential deployment risks in this field. Is it possible to conduct dynamic image fusion with a clear theoretical justification? In this paper, we give our solution from a generalization perspective. We proceed to reveal the generalized form of image fusion and derive a new test-time dynamic image fusion paradigm. It provably reduces the upper bound of generalization error. Specifically, we decompose the fused image into multiple components corresponding to its source data. The decomposed components represent the effective information from the source data, thus the gap between them reflects the \textit{Relative Dominability} (RD) of the uni-source data in constructing the fusion image. Theoretically, we prove that the key to reducing generalization error hinges on the negative correlation between the RD-based fusion weight and the uni-source reconstruction loss. Intuitively, RD dynamically highlights the dominant regions of each source and can be naturally converted to the corresponding fusion weight, achieving robust results. Extensive experiments and discussions with in-depth analysis on multiple benchmarks confirm our findings and superiority. Our code is available at https://github.com/Yinan-Xia/TTD.

Dynamic Image Fusion
Bing Cao 1.2
Yinan Xia
Yi Ding'
Changqing Zhang
Qinghua Hu!
'College of Intelligence and Computing, Tianjin University, Tianjin, China "Tanjin Key Lab of Machine Learning, Tianjin, China
{caobing, xyn, ding-yi0731, zhangchangçing, huqinghua) edu, cn
Abstract
The inherent challenge of image fusion lies in capturing the correlation of multi- source images and comprehensively integrating effective information from different sources. Most existing techniques fail to perform dynamic image fusion while notably lacking theoretical guarantees, leading to potential deployment risks in this field. Is it possible to conduct dynamic image fusion with clear theoretical justification? In this paper, we give our solution from a generalization perspee- tive. We proceed to reveal the generalized form of image fusion and derive a new test-time dynamic image fusion paradigm. It provably reduces the upper bound generalization error. Specifically, ponents corresponding to its source data. The decomposed components represent the effective information from the source data, thus the gap between them reflects the Relative Dominability (RD) the uni-source data in constructing the fusion image. Theoretically, prove that the key to reducing generalization error hinges on the negative correlation between the RD-based fusion weight and the uni-source reconstruction loss. Intuitively, RD dynamically highlights the dominant regions each source and can be naturally converted to the corresponding fusion weight, achieving robust results. Extensive experiments and diseussions with in-depth analysis on multiple benchmarks confirm our findings and superiority. Our code is available
decompose the fused image into multiple com-
Yinan- Xia/TTD.
Introduction
Image fusion jointly integrates complementary information from multiple sources, aiming to generate informative and high-quality fused images. With superior scene representation and enhanced visual perception, image fusion significantly benefits downstream vision tasks [ 2] Typically, image fusion can be categorized into multi-modal, multi-exposure,, and multi-focus image fusion tasks. Multi-modal image fusion encompasses Visible-Infraree image Fusion (VIF) and Medical Image Fusion (MIF). For VIF 3.5) infrared images effectively highlight thermal targets especially under extreme conditions, while visible images provide texture details and ambient lighting. For MIF [6] 7), different medical imaging modalities emphasize various focal areas, enhancing diagnostic capabilities. Multi-exposure image Fusion (MEF) [8-10 bridges the gap between high dynamic range (HDR) natural scenes and low dynamic range (LDR) pictures, ensuring better detail preservation varying lighting conditions. Multi-Focus image Fusion (MFF) [11 aims to produce all-in-focus images by combining multiple images focused at different depths.
Numerous image fusion methods have been introduced, which can be mainly grouped into traditional techniques and deep learning approaches. Traditional image fusion methods, such multi-scale decomposition-ben based models [14] [5] and sparse representation-beee based methods [16], rely on mathe- matical transformations fuse images in the transform domain [77. contrast, deep learning- based
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Visible
Infrared
RD
MRI
RD
PET
RD
(a) VIF
(b) MIF
Foreground Focus
Background Focus
Overexposure
RD
Underexposure
RD
RD
RD
(c) MEF
(d) MFF
We visualized the Relative Dominablity (RD) of each source on four tasks, which effectively highlights the dominance of uni-source in image fusion.
Figure 1:
methods employ data-driven schemes to fuse multi-source images, including convolutional neural network (CNN) based methods [8], generative adversarial network (GAN) based methods [19] 4] and transformer-based methods 20 The effectiveness of image fusion algorithms hinges on two critical factors: feature extraction 21 and feature fusion [22]. The aforementioned methods strive to achieve high-quality fused images by learning effective uni-source multi-source feature representa- tions through complex network structures or feature decomposition schemes. However, they often overlook the complexity of the real world, which necessitates dynamic feature fusion.
Recently, some works have highlighted the importance of dynamism in image fusion. For instance, 23 pioneered the combination of image fusion with Mixture of Experts (MoE), dynamically extracting effective and comprehensive information from the respective modalities. [24] utilized task-specific routing networks to extract task-specific information from different sourees with dy- namic adapters. Despite their empirically superior fusion performance, these dynamic fusion rules mainly rely heuristic designs, lacking theoretical guarantees and interpretability. Moreover, they potentially lead to unstable and unreliable fusion results, especially in complex scenarios.
To address these issues, reveal the generalized form of image fusion and propose new Test- Test-Time Dynamic(TTD) image fusion paradigm with a theoretical guarantce. Given that the fused image integrates comprehensive information from different sources, it can be obtained by weighting the effective representation each uni-source. By revisiting the relationship between fusion weights and image fusion losses from the perspective of generalization error 225 decompose the fused image into multiple uni-source components and formulate the generalization error upper bound of image fusion. Based on generalization theory, we for the first time prove that dynamic image fusion is superior to static image fusion. The key to enhancing generalization lies in the negative correlation between fusion weight and uni-source component reconstruction loss. As fusion models trained to extract complementary information from each source. the decomposed components represent the effective information from the source data. Thus, the fusion components can be estimated by source data with the fusion model, the losses of which represent the deficiencies of the source in constructing fusion images. Accordingly, we derive pixel-level Relative Dominablity (RD) as the dynamic fusion weight, which theoretically enhances the generalization of the image fusion model and dynamically highlights the changing dominant regions of different sources shown in Fig. Extensive experiments on multiple datasets and diverse image fusion tasks demonstrate our superiority. Overall, our contributions can be summarized as follows:
This paper first theoretically proves the superiority of dynamic image fusion over static image fusion and provides the generalization eтror upper bound of image fusion by decomposing the fusion image into uni-source components provably. The proposed generalization theory reveals that the key to reducing the upper bound lies in the negative covariance between the fusion weight and uni-source reconstruction loss.
We proposed simple but effective test-time dynamic fusion paradigm based on the gener- alization theory. By taking the uni-source's 's Relative Dominability as the dynamic fusion weight, we theoretically enhance the generalization of the image fusion model and dynami- cally emphasize the dominant regions of each source. Notably, our method does not require additional training, fine-tuning, and extra parameters.

---

We conduct extensive experiments on modli.-odall, multi- exposure, and multi multi-focus datasets. The superior performance across diverse metrics demonstrates the effectiveness and applica- bility of our approach. Moreover, an additional exploration of the gradient in constructing fusion weight demonstrates the reasonability of our theory and its expandability.
2
Related Works
Image Fusion aims to integrate complementary information of diverse source images. For instance,
26 utilize autoencoders to extract multi-source features and fuse them using a designed strategy.
GAN- -bdddddd methods [19] and transformer- methods (20] also achieved significant progress.
27 introduced the denoising diffusion probabilistic model (DDPM) to image fusion. [28] and [29]
achieve considerable fusion performance by decomposing image features into high-frequency and
low-frequency components. In addition to these static image fusion methods, 23 used a Mixture of
Experts (MoE) to dynamically assign fusion weights, while 24 utilized dynamic adapters to prompt various fusion tasks within unified model. These approaches mainly focus on obtaining promising feature representations. Although some existing works have explored dynamic image fusion, the lack of theoretical guarantees may result instability and unreliability practice.
Multimodal Dynamic Learning Although dynamic fusion is not fully studied in existing image fusion works, numerous methods have leveraged multimodal dynamic learning at the decision level [30]. For example, Han et al. [1 assigned dynamic credible fusion weights to each modality the decision level for robust evidence fusion. Xue al. [32 employs a Mixture of Experts to integrate the decisions of multiple experts, Zhang et al. [33 combined decision level fusion weight
with uncertainty to conduct a credible fusion. Despite the wide exploration of dynamic fusion the decision level, there is still insufficient research on dynamic fusion at the feature level with theoretical guarantees. this paper, we focus on the dynamic nature image fusion, theoretically prove that dynamic fusion is superior to static fusion, and propose provable feature-level dynamic fusion strategy.
3 Method
Given the data from M sources for image fusion, the input samples are denoted as {x (m) m = 1,, where (m)) represents the input from the m-th source. Let f be the image fusion model, comprising both encoders and decoders. Define E = (E(m) m = 2...... M) as the set of encoders within the image fusion network, where (-) (-) is the encoder for the m--hh source. In early fusion, the encoders E are constant mapping functions, meaning that multi-source images are combined at the image level. Let D--) denote the decoder in the image fusion network, and let (m)(m) E IR # | m = 11......44 .M} represent the set of image fusion weights. Consequently, the fused image Ip can be expressed as:
M D( D (m)) Elml (x(()))
1F
m=11
Additionally, we define the loss function for image fusion tasks, where || |||| represents any distance norm. The loss function is given by:
M
[(F
Σ I x (m)))
m==1
Generalization Error Upper Bound. In machine learning, the concept of Generalization Error Upper Bound refers the theoretical limit model's performance when applied to unseen data (D) 33]. A smaller upper bound indicates better-expected performance on data from an unknown distribution. For image fusion tasks, the Generalization Error (GError) of fusion model f can be defined as:
GError(f) = E Ez- [(=f((((())
(3)
3

---

11 x
Visible Component 1
x (o))
18
Decompose
Generalization Error Upper Bound
Cow (I (x.(o))
- -0000,, win))
RD
Fused Image 1
(ID( (x((e))
-060 whin)
+
Decompose
RD
11 - x (6))
Infrared Component FF
x (6r)
The framework of our TTD. Deriving from the generalization theory, decompose fused images into uni-source components and find the key to reducing generalization error upper bound is the negative correlation between the fusion weight and reconstruction loss. Accordingly, propose pixel-wise Relative Dominablity (RD) for each source, which is negatively correlation with the reconstruction loss and highlights the dominant regions of uni-source in constructing fusion images.
Figure 2:
Considering the GError of image fusion model 35 37
e[Ip, 2) can be further deduced as [[IF, 2) S Ei=1 (+) ((E (xx)0) _((m)] Therefore, the fused image Ip is decomposed into M
M
M
uni-source components (D (EG) (1) 1...). Based Eq. (3) and (8). have:
Theorem 3.1 (Decomposition of Generalization Error). The GError for multi-source image fusion
modelf can be decomposed into linear combination of each uni-souree component reconstruction loss under the condition that E (m) the detailed proof given in Appendix A.I
.M
M
GError (f) = EEND
Σ
((w (x((0)) (m)))ll)
m=
E [[M 1) (() (2( (mm)|| + (M 1) Σ
M
Cor(wim), || D (im) (m)) -x(m)]))
(4)
1011 source componont
Let ((m) = ID (E(m) (I(()) =.) which represents the reconstruction loss between a uni- source component and its corresponding uni-source image. The term (")( g(m)) denotes the covariance between (m) and g(m), The essence of reducing generalization error lies in achieving the lowest possible fusion loss. By leveraging the triangular inequality properties of distance norms deduce that the GError is bounded by the covariance term and the distance between each uni-source component and its source image. It is noteworthy that f (m)( remains constant during the test phase, emphasizing that the pivotal factor in reducing Generalization Error Upper Bound (GEB) lies in the covariance between (m) and g(m).
within the fusion loss,
Superiority of Dynamic Image Fusion over Static Image Fusion. Most existing image fusion approaches reduce GEB by minimizing g(m), indicating an effort to enhance the quality of sni-source feature representations. However, they often overlook the intrinsic significance of fusion weight (m)) Fusion strategies employed in static image fusion encompass methods such maximum, minimum, addition, -norm, etc. Nevertheless, none of these fusion weights exhibit a correlation with uni-source reconstruction loss, i.e. Comt g(m)) = O. During the test phase, g(m) remains constant. If we have: Cov(w (oooaaammeee g(ma) < 0 for all source images, we can derive the conclusion:
(m)
GEBaynamic GEB static-
(5)

---

This indicates that for a well-trained image fusion model, dynamic fusion strategy can bring better generalization than static fusion strategy.
Relative Dominablity. Recalling the Eq. the negative correlation between fusion weight and the reconstruction loss ((m) provably reduces the generalization error upper bound. Therefore, we introduce pixel-level Relative Dominablity (RD) the fusion weight for each source, which exhibits a negative correlation with the reconstruction loss of the corresponding fusion component. Since fusion models are trained to extract complementary information from each source, the de- composed components of fusion images represent the effective information from the source data. Thus, the uni-source components can be estimated from source data using the fusion model, with the losses representing the deficiencies of the source in constructing fusion images. For instance, in given region, the larger the pixel-wise fusion loss between the reconstructed component and its corresponding uni-source image, the smaller its contribution to image fusion. Intuitively, using RD as the dynamic fusion weight can capture the dominance of each source in image fusion and enhance its advantages in constructing fusion images. Theoretically, according to Thm. 3.1 negatively correlated with the pixel-wise fusion loss, RD effectively demonstrates the dominance of each source. Notably, considering the relative nature of multi-souree source image fusion, the sum of the RDs of different sources for the same pixel should be one. Consequently, by establishing a negative correlation with the loss and implementing normalization, we can obtain the Relative Dominablity of each source for certain sample as follows:
(()) (m)) Softmaz(e {(m)
In addition, we present the algorithm and test pipeline of our dynamic fusion strategy in Appendix B. B.I
4 Experiments
4.1 Experimental Setup
Datasets.
We evaluate our proposed method on four image fusion tasks: Visible-Infrared Fusion (VIF), Medical Image Fusion (MIF), Multi-Exposure Fusion (MEF), and Multi-Focus Fusion (MFF).
o VIF: For VIF tasks, we conduct experiments on two datasets: LLVIP 3S and MSRS [7]. For
LLVIP datasets, we randomly select 70 samples from the test set for evaluation. MIF: We conduct
experiments on the Harvard Medical Image Dataset, following the test setting in 29
MEF:
Following the setting in (24), we verified the performance of our method MEFB
39 dataset.
MFF: For the MFF task, we evaluate our method on MFI-WHU datasets [40], following the test
protocol in 24. As test-time adaption approach, TTD performs adaptive fusion solely during testing, without additional training and training data. Competing Methods. For VIF and MIF tasks, we evaluated 12 state-of-the art methods, encompass- ing both DenseFuse (26), CDDFuse (29, U2Fusion (22). DDFM (27), DeFusion [41], PIA FlAFusoon
[17. DIVFusion 42). MUFusion 43,, IFCNN [18) and SwinFuse (20]. and TC-MoA 24 For MEF and MFF tasks, we compared our methods with general image fusion methods and task-specific image fusion methods. Notably, among these methods, only DDFM is training-free, and other methods are all pre-trained on VIF datasets. experiments, we apply our TTD to CDDFuse (CDDFu- sion+TTD), PIAFusion (PIAFusion+TTD) and IFCNN (IFCNN+TTD), separately. Our experiments are conducted on Huawei Atlas 800 Training Server with CANN and NVIDIA RTX A6000 GPU.
including o information theory: entropy (EN) 45), cross entropy (CE), the sum of the correlations of differences (SCD) [46] image feature: standard deviation (SD), average gradient (AG) [47], edge intensity (EI) and spatial frequency (SF) and o structural similarity: structural similarity (SSIM) 49
Metrics. We selected several evaluation metrics from three aspects 44
4.2 Quantitative Comparisons
Visible-Infrared Fusion. Tab. I reports the performance of competing approaches and TTD- applied methods on LLVIP and MSRS datasets for 7 metrics. Notably, by applying our TTD, the previous methods have improved on most of the indicators. Also, our TTD strategy outperforms other traditional static methods, training-free method DDFM, and data-driven dynamic strategy TC-MoA, achieving the SoTA performance on most metrics. Moreover, with particularly high values in SD, AG, EI, and SF, our TTD ensures that fusion images maintain exceptional contrast and detailed texture, highlighting its efficacy in preserving quality. The outstanding performance on EN and SCD indicates
5

---

Table 1: Quantitative performance comparison of different fusion strategies on visible-infrared datasets. The 'TTD' suffix and gray background indicates our method is applied to this baseline. The red and blue represent the best and second-best result respectively. The bold indicates the baseline w/ TTD performance better than that w/o TTD. We used A to illustrate the amount of improvement our TTD method achieved compared to the baseline.
LLVIP
Dataset
MSRS Dataset
Method
ENT
SD1
AGT
Hit
SPT
SCDT
CE
ENT
SDT
AG1
HlT
SFT
SCD1
CEU
Densefuse
26
6.83
33.98
3.62
8.80
12.27
1.24
8.13
5.93
23.55
2.05
5.42
6.02
1.25
7.75
U2Fusion
6.64
35.83
4.13
10.56
13.70
1.27
9.14
5.21
22.67
2.51
6.70
8.06
1.15
12.54
DePusion
41
7:21
42.91
3.80
9.76
11.99
1.21
7.83
6.38
35.43
2.64
7.12
8.15
1.27
7.55
SwinFuse
20
5.84
40.95
3.58
9.02
15.38
1.27
8.51
4.24
29.72
1.93
5.12
9.47
1.03
8.93
MUFusion
43
7.29
50.09
4.96
13.38
13.29
1.38
7.57
6.09
31.81
3.46
9.54
9.77
1.33
6.87
DDFM 27
6.31
32.31
3.25
7.93
11.71
1.07
8. 8.66
5.76
22.94
2.01
5.28
6.44
1.22
7.56
TC-MoA
7.40
48.92
2.76
7.47
9.78
1.40
7.83
6.49
3.12
8.99
10.77
1.33
7.12
IFCNN 18
6.95
37.75
5.18
13. 13
18 18
1.32
7.82
6.07
26 99
3.44
8.99
10.77
1.33
7.12
IFCNN+TTD
6.98
38.99
5.48
13.92
19.
19.40
1.34
7.79
6.09
28
3.58
9.39
11.46
1.35
7.10
Improve
20.03
2.1.21
2:0.30
2.0.79
2.1.2
2.0.02
2.0.03
2.0.02
2.1.0
2.0.14
2.0.10
2.0.69
2.0.02
2.0.02
CDDFuse 29
7.36
50.90
4.99
12.68
18.26
1.62
7.79
6.71
43.38
3.78
10.08
11.57
1.00
6.92
CDDRusc+TTD
7.34
53.88
83.88
5.54
14.07
20
17
1.38
7.83
6.61
13.78
3.97
10.
54
67
1.58
6.95
Improve
70.02
72.98
70.55
1,339
A1.91
v0.04
/.0.
v0.07
7.040
1.0.9
/.0.46
11.1
v0.02
v0.03
PlAFusion 17
7.39
52.12
5.77
19.59
1.59
7.72
6.64
45.34
3.95
10.57
12.12
1.70
PIAFusion TTD
7.42
56.
14
5.90
15.15
20 20.29
1.64
7.65
6.62
48 26
4.18
11.18
12.99
1.52
6.92
Improve
20.04
244.02
2.0.13
2.0.34
2.0.70
4.0.0
2.0.07
70.02
2.2.92
2.0.23
2.0.1
2.0.87
30.18
2.0.11
Table 2: Quantitative comparison on MFI-WHU dataset in MFF task and MEFB dataset in MEF task.
MEFB Dataset AGT
MFI-WHU Datasel Elf
Method
SDf
ET1
ENT
SET
CEI
Method
SDf
ENT
AG1
SF1
CEJ
PMG1
50
62.36
13.33
7.25
5.35
18.60
9.57
PMGI 600
44.64
11.03
7.10
4.20
11.36
10.00
U2Fusion
22
52.27
12.06
6.93
1.65
15.37
13.66
U2Pusion
22
51.38
18.21
7.32
7.03
19.10
7.95
DeFusion
11
46.85
10.51
6.78
4.07
13.48
13.35
DeFusion
41
50.78
12.23
7.29
4.65
12.56
7.42
TC MoA
24
48.91
12.13
7.06
4.77
15.56
11.91
McA
24
53.35
17.31
7.36
6.83
20.56
7.40
Deepfuse
48.29
8.74
6.97
9.90
12.33
DRPI. 12
53.88
18.93
7-38
7.66
23. 66
7.38
DEM 51
52.35
13.60
7.32
5.46
18.85
11.68
ECNN 1
53.79
18.77
7.38
7.59
23.51
7.38
DSIFT EF 52
50.65
12.54
7.36
5.00
17.18
12.47
GCF 53
53.78
18.81
7.38
7.60
23.56
7.38
MEFAW
[44]
48.31
12.41
7.22
4.95
16.86
11.74
GFDF 55
53.72
18.70
7.38
7.55
23.38
7.39
MEFCNN 56
51.29
12.23
7.24
4.88
16.89
13.15
MADCNN 57
53.85
18.82
7.38
7.59
23.37
7.38
MEPOpt
49.21
14.06
7.18
5.63
19.27
12.09
PCANer 59
53.73
18.61
7.37
7.51
23.33
7.38
GAL.Fusion M
50.39
9.53
6.95
3.78
13.04
13.12
SESF 61]
53.85
18.75
7.38
7.57
23.57
7.38
HoLoCo
52.83
12.27
7.19
4.65
13.56
13.63
TF [3]
53.71
18.72
7.38
7.57
23.41
7.38
IFCNN LB
51.78
14.48
7.06
5.85
20.41
11.60
IFCNN
54.02
18.71
7.38
7.55
22.85
7.40
IFCNN-TID
52.86
15.94
7.10
6.38
21.99
11.41
IFCNN+TTD
54.22
19.10
7.39
7.70
23.51
7.40
Impoove
A1.08
4.1.4
2.0.04
2.0.53
A1.58
2.0.19
Improved
2.0.20
2.0.39
2.0.01
2.0.15
2.0.66
2.0.0
our fusion results embed more information and contain abundant edge information from the source images. Although our approach is a test time adaptation strategy that does not require training, the designed dynamic weights based on theoretical principles have led to outstanding fusion performance, achieving SoTA results on VIF tasks.
Medical Image Fusion. We report the comparison results three MIF scenarios: MRI-CT, MRI- PET, and MRI-SPECT. As depicted Tab. 3 and Tab. 7 Appendix [C.3 our method yields competitive performance on seven evaluation metrics. Specifically, our TTD enhances EN, AG, and SSIM, indicating ample gradient information and structural details in the fusion results. The significant improvements in SD, EI, and SF highlight our high definition and texture quality compared with the competing methods, making it exceptionally competitive.
Multi-Exposure and Multi-Focus Image Fusion. In the comparisons on MEF and MFF tasks, we applied our TTD to the general fusion method IFCNN. We compared it with other general fusion methods and task-specifie fusion methods. As depicted in Tab. [ TTD outperforms existing general fusion methods and task-specific methods in terms of SD, EI, AG, and SF. This indicates that TTD produces fused images with clear, abundant edges and exceptional sharpness. Furthermore, the superiority in EN and CE suggests that our TTD enables the baseline to preserve more advantages from different sources.

---

Visible Image
Infrared Irmage
Dense fuse
U2Fusion
MRI
PET
Densefuse
U2Fusion
DeFusion
MUFusion
DDFM
TC-MoA
DeFusion
MUFusion
DDFM
TC-MoA
IFCNN
CDDFuse
PIAFusion
IFCNN
CDDFuse
PIAFusion
Ours
Figure 3: (a) On the VIF task, our TTD produces fused images that retain more multi-source information compared with existing approaches. (b) On the MIF task, our method improves the contrast of the fused image and preserves more details from the source image.
Source
Source
PMG1
U2Fusion
Source
Souree
PMGI
U2Fusion
DeFusion
Tc. MoA
IFCNN
Ours
DeFusion
TC-MoA
IFCNN
Ours
Figure 4: The comparison fusion results on MEF and MFF tasks. (a) On the MFF task, our method retains the color and clarity of the original image better. (b) On the MEF task, our TTD ensures better detail preservation in varying lighting conditions.
4.3 Qualitative Comparisons
Visible-Infrared Fusion. As shown in Fig. and Fig. 8 in Appendix C.1
compared with existing methods on the LLVIP dataset, our TTD effectively combines comprehensive information from different sources, leading to significant visual performance. Specifically, the fusion result not only preserves the texture details and edge information of visible images but also incorporates high-quality thermal imaging contrast of infrared images. Additionally, mentioned in the qualitative analysis, our fusion images exhibit high fidelity and clear contrasts, showing consistent superiority in terms of image quality. These experimental results demonstrate the effectiveness of our TTD.
Medical Image Fusion. For the MIF task, we present qualitative comparisons of the MRI-PET fusion. As shown in Fig. and Fig. & in Appendix C.I it is clear that fusion images generated by our method preserve substantial amount of structural information. Notably, our method maintains significant portion of excellent soft tissue contrast details from MRI images and combines the quantitative physiologic information of PET images. With our TTD, the overall structural details and sharpness of the fused image are significantly enhanced. Moreover, in the regions where the high-intensity color areas the PET image overlap with the struetural information of the MRI, the detailed information from the original images is well preserved and highlighted in the fused image.

---

Multi-ENs Multi-Exposure and Multi-Focus Image Fusion. We also provide a comparison of fusion results for the MEF and MFF tasks in Fig. and Fig.8inn Appendix C.I. Notably, our TTD significantly enhances the clarity and sharpness of texture details. Obviously, after applying our TTD, the fused images on the MEF task exhibit higher clarity both the foreground and background. For the MFF task, our method accurately utilizes the effective regions from both underexposed and overexposed images. Compared to other methods, our fusion results achieve more precise exposure and rich texture details, such the cars in the garage and the textures on the walls. Additionally, our fusion method retains high-fidelity colors that are closer to the original images.
Apart from the comparisons with the existing methods, we provided more ablated comparisons with baselines in Fig. of Appendix C.2
5 Discussion
5.1 Is Negative Correlation Help?
The negative correlation between RD and g(m) is derived from Eq. reduce the generalization error of the image fusion model. To further validate the effectiveness of the theoretical guarantee, we compare with a contrast setting: using new fusion weight, which is positively correlated (PC) to g(m), to perform image fusion.
As shown in the correlation comparison in Fig. (b), cuss-positive-corelateee weights (yellow line), that conflict with our theory, lead to decreased performance compared with static fusion (green line). As comparison, the results of the superior performance compared with both static image fusion and positively correlated fusion strategy. These experiments verify that the proposed negative correlation setting can explicitly reduce the generalization error, demonstrating the reasonability of the proposed TTD image fusion model.
-eerrllve- fusion strategy (red line), exhibit
5.2 Relative Dominability
In this paper, we introduce the pixel-level Relative Dominablity, which indicates the advantages of each source. Treating the Relative Dominablity dynamic fusion weight, TTD achieves adaptive and interpretable image fusion. We provide visualizations of each source's pixel-level Relative Dominability obtained using CDD+TTD for VIF and MIF tasks, and IFCNN+TTD for MEF and MFF tasks.
Inffaredd Fusion. As shown in Fig. it can be observed that Relative Dominablity (RD) accurately reflects the dominance of each source: in visible images, well-lit and properly exposed bright areas contain abundant brightness information, and areas like digits and characters exhibit rich texture details. In contrast, infrared images provide thermal imaging information for areas and objects in shadow that visible images cannot capture due visual obstacles. The proposed RD effectively captures the advantageous regions of different source images and assigns larger weights to these regions, thereby achieving more reliable fusion results.
Medical Image Fusion. We visualize RDs in the MRI-PET dataset. Similar findings are also apparent in the MIF task. In PET images, bright regions indicate areas of malignant cell proliferation, while MRI contains more structural information. As shown in Fig. [ the RDs of PET stand out in the bright areas while MRI's highlights the structural information. Guided by RD, TTD emphasizes potential lesion areas while preserving the structural information of these areas effectively.
Multi-Expone Expti-Exppsuuee and Multi-Focus Image Fusion. For MEF and MFF tasks. the ideal outcome is that the fused images contain properly exposed or precisely focused regions from each uni-source. As shown in the visualized RD map in Fig. [ our TTD can effectively capture the dominant areas in different sources and assign higher RD values, i.e. dynamic fusion weight, to these regions.
Downstream Tasks. validate the effectiveness of our RD on downstream tasks, we compared our TTD with the baseline object detection task. Detailed results are given in Appendix C.5
8

---

13
Visible Image
Visible Channel 44
Visible Channel 13
Visible Channel 58
Exploration of Gradicnt-basce an
11s
Infrored Image
Infrared Charnel 44
Infrared Channel 13
Infrared Charnel 58
Comparison of the Correlation
(a) The visualization of obtained by gradient maps different channels (b) radar charts of experiments discussion
Figure 5: (a) The visualization of RDs obtained by gradient maps of different channels. The 44th gradient map provides wrong dominance information, and the 13th gradient map offers insignificant information, while the 58th gradient map performs the proper advantages of the two source images. (b) The radar chart of the gradient- based RD experiment (upper) and the validation of the negative correlation (below).
Overall, with the integration of our TTD, the baseline model gains the ability to perceive dominant information dynamically. Therefore, this interpretable plug-and-play test-time dynamic adaptation fusion strategy can further improve the performance of the existing state-of-the- art methods. This further validates the effectiveness of RD in our TTD.
5.3 Gradient- Relative Dominability
In our TTD, the proposed pixel-level fusion weight is computed by pixel-level loss. However, some numeric losses are limited in directly obtaining the pixel-level weights, making it hard to integrate with TTD flexibly. To overeome this dilemma, we extend our TTD to a more general form and construct gradient-basee Relative Dominability through any fusion losses for a more fine-grained and robust image fusion.
Recalling our optimization objective of the generalization error bound, we aim for a negative correlation between the weights and losses of the same modality, i.e., establishing correlation between losses and weights. Inspired by this positive correlation between loss value and the absolute value of its gradient for features with any convex loss function, our TTD can be further extended to gradient-based dynamic fusion weight.
Specifically, we first calculate the absolute value of gradients [G(m)] E HxXxx of each uni-source feature. As test-time adaption approach, TTD does not update the network parameters, meaning that the unimodal feature space remains fixed for the same baseline. For the same task scenarios, the can empirically select the gradient channels that
feature patterns tend to be similar. Therefore, well represent the advantage areas to compute RDs. Also, as illustrated Fig. 5 (a), gradient maps of some channels (such as the 44th and the 13th channels) lack significant useful information and fail to capture advantageous regions in the original images. Therefore, we select the gradient map Ig [m)] E H xW among channels that best represent the dominance of the uni-source empirically. By replacing ((m), we can obtain the RD and the dynamic fusion weight as follows:
w (m) RD (m) = Sofftmaa(e (1))
We have also conducted comparisons of bass-based TTD, full gradient- based TTD, and best gradient- based TTD on IFCNN. For the best gradient-based TTD, we choose the 58-th gradient map obtain the fusion weight. The results of gradient- based RD in Fig. [5 (b) demonstrate that full gradient (yellow line) may bring wrong or useless dominance information to fusion weights, leading to worse performance compared with loss-based TTD (green line). However, by selecting the empirically best gradient map (red line), our TTD provides more fine- tinennained dominance information compared the global loss maps, achieving more detailed dynamic fusion with better performance.

---

6 Conclusion
Image fusion aims to integrate effective information from multiple sources, Despite numerous meth- ods being proposed, research on dynamic fusion and its theoretical guarantees remains significantly lacking. To address these issues, we derive from a generalized form of image fusion and introduce Test-Time Dynamic (TTD) image fusion paradigm with theoretical guarantee. From the perspective of generalization error, we reveal that reducing generalization error hinges on the negative correlation between the fusion weight and the uni-souree component reconstruction loss. Here the uni-source components are decomposed from fusion images, reflecting the effective information of the corresponding source image in constructing fusion images. Accordingly, propose pixel-level Relative Dominablity (RD) as the dynamic fusion weight, which theoretically enhances the gener- alization of the image fusion model and dynamically highlights the changing dominant regions of different sources. Comprehensive experiments with in-depth analysis validate our superiority. We believe the proposed TTD paradigm is an inspirational development that can benefit the community and address the theoretical gap in image fusion research.
Acknowledgements
This work was sponsored by the National Natural Science Foundation of China (No.s U23B2049, 62476198, 62376193, 62106171, 61925602), and CCF-Baidu Open Fund. This work was also sponsored by CAAI-CANN Open Fund, developed on Openl Community, Yinan Xia and Yi Ding contributed equally to this work. The authors thank anonymous peer reviewers for their helpful suggestions.
References
[1] Robert T Collins, Alan Lipton, Takeo Kanade, Hironobu Fujiyoshi, David Duggins, Yanghai Tsin, David Tolliver, Nobuyoshi Enomoto, Osamu Hasegawa, Peter Burt, et al. A system for vidco surveillance and monitoring, VSAM final report, 2000(1-68): 1688)11 2000.
[2] Li-Jia Li, Richard Socher, and Li Fei-Fei. Towards total scene understanding: Classification, annotation and segmentation in automatic framework. In 2009 JEEE Conference Computer Vision and Pattern Recognition, pages 2036-2043. IEEE, 2009.
[3] Jiayi Ma, Yong Ma. and Chang Li. Infrared and visible image fusion methods and applications: A survey. Information fusion, 45:153- 15:153-178, 178, 2019.
Jinyuan Liu, Xin Fan, Zhanbo Huang. Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and multi-scenario multi modality benehmark to fuse infrared and visible for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 5802 581111111 2022.
[5] Jinyuan Liu, Xin Fan, Ji Jiang, Risheng Liu, and Zhongxuan Luo. Learning deep multi-scale feature ensemble and edge-attention guidance for image fusion. IEFF Transactions Circuits and Systems for Video Technology, 32(1):105-1 119, 2021.
[6] Alex pappachen James and Belur Dasarathy. Medical image fusion: A survey of the state of the art. Information fusion, 19:4-19, 2014.
[7] Jiao Du, Weisheng Li, Ke Lu, and Bin Xiao. An overview of multi-modee medical image fusion. Newro- computing, 215:3 20, 2016.
[8] Kede Ma, Kai Zeng, and Zhou Wang. Perceptual quality assessment for multi-exposurre image fusion. IEEE Transactions on Image Processing, 24(11):3345-335 2015. doi: 10. 00.10/9912200152222222
[9] K Ram Prabhakar, V Sai Srikar, and R Venkatesh Babu. Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs. In Proceedings of the IEEE international conference on computer vision, pages 47 44222, 2017.
[10] Jinyuan Liu, Jingjie Shang. Risheng Liu, and Xin Fan. Attention Attention-guided global- adversarial learning for preservin multi-exposure image fusion. IEEE Transactions on Circuits and Systems for Video Technology. 32(8):5026 50400 2022.
10

---

[11] Yu Liu, Xun Chen, Hu Peng. and Zengfu Wang. Multi-focus image fusion with a deep convolutional neural network. Information Fusion, 191-077 2017.
[12] Jinxing Li, Xiaobao Guo, Guangming Lu, Bob Zhang, Yong Xu, Feng Wu, and David Zhang. Drpl: Deep regression pair learning for multi-focus image fusion. IEEF Transactions on Image Processing, 29: 4816-4831, 2020.
[13] Mostafa Amin-Naji, Ali Aghagolzadeh, and Mehdi Ezoji. Ensemble of cnn for multi-focus image fusion. Information fusion, 51:201-214 2019.
[14] Zhong Zhang and Rick Blum. A categorization of pettissall-ddcomposiiiiiie based image fusion schemes with performance study for digital camera application. Proceedings of the IEEE, 87(8):1315- 1999.
[15] Jun Chen, Xuejiao Li, Linbo Luo, Xiaoguang Mei, and Jiayi Ma. Infrared and visible image fusion based on target-enhaneed multiscale transform decomposition. Information Sciences, 508:64-78, 2020.
[16] Qiang Zhang. Yi Liu, Rick S Blum, Jungong Han, and Dacheng Tao. Sparse representation based multi- sensor image fusion for multi-focus and multi multi-modality images: A review. Information Fusion, 40:57-75. 2018.
[17] Linfeng Tang. Jiteng Yuan, Hao Zhang, Xingyu Jiang, and Jiayi Ma. Piafusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 83:79-92, 2022.
[18] Yu Zhang, Yu Liu, Peng Sun, Han Yan, Xiaolin Zhao, and Li Zhang. Ifenn: A general image fusion framework based on convolutional neural network. Information Fusion, 54:99-118, 18, 2020.
[19] Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. Fusiongan: A generative adversarial network for infrared and visible image fusion. Information fusion, 48:1414:1100 2019.
[20] Zhishe Wang, Yanlin Chen, Wenyu Shao, Hui Li, and Lei Zhang. Swinfuse: A residual swin transformer fusion network for infrared and visible images. IEEE Transactions on Instrumentation and Measurement, 71:1-12, 2022.
[21] Mark Nixon and Alberto Aguado. Feature extraction and image processing for computer vision. Academic press, 2019.
[22] Han Xu, Jiayi Ma, Junjun Jiang, Xiaojic Guo, and Haibin Ling, U2fusion: A unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 (4(1):502-518, 2020.
[23] Bing Cao, Yiming Sun, Pengfei Zhu, and Qinghua Hu. Multi-modal gated mixture of local-to-gee to-global experts for dynamic image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23555 33555-23564, 2023.
[24] Pengfei Zhu, Yang Sun, Bing Cao, and Qinghua Hu. Task-customized mixture of adapters for general image fusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 7099-7108, 2024.
[25] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
[26] Hui Li and Xiao-Jun Wu. Densefuse: A fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 28(5):6614-2633, May 2019.
[27] Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Luc Van Gool. Ddfm:: denoising diffusion model for multi-modality image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8082-8093, 2023.
[28] Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Pengfei Li, and Jiangshe Zhang. Didfuse: Deep image decomposition for infrared and visible image fusion. arXiv preprint arXiv:2003.0922 2020.
[29] Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Shuang Xu, Zudi Lin, Radu Timofte, and Luc Van Gool. Cddfuse: Correlatonn-rive dual-branch feature decomposition for multi-modality image fusion. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5906-5916, June 2023.
[30] Berk Gokberk and Lale Akarun. Comparative analysis of decision- fusion algorithms for 3d face recognition. In 18th International Conference on Pattern Recognition (ICPR 066) volume 3, pages 1018-101 1021. IEEE, 2006.
11

---

[31] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion. IEEE transactions on pattern analysis and machine intelligence, 45(2): 2551-2566, 2022.
[32] Zihui Xue and Radu Marculescu. Dynamic multimodal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2574-2583, 2023.
[33] Qingyang Zhang. Haitao Wu, Changging Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning, pages 41 41759-41769. PMLR, 2023.
[34] Partha Niyogi and Federico Girosi. On the relationship between generalization emror hypothesis complexity, and sample complexity for radial basis functions. Neural Computation, 8(4):819882, 1996.
[35] Yuchen Zhang, Percy Liang, and Martin J Wainwright. Convexified convolutional neural networks. In International Conference on Machine Learning, pages 4044 4053. PMLR, 2017.
[36] Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over- parameterized deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelligence.
volume 34, pages 3349-3356, 3356, 2020.
[37] Gábor Lugosi and Gergely Neu. Generalization bounds via convex analysis. In Conference on Learning Theory, pages 3524-3544 3546. PMLR, 2022.
[38] Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. Livip: A visible-infrared paired dataset for low-light vision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3496-3504, 2021.
[39] Xingchen Zhang. Benchmarking and comparing multi mxpos-ee image fusion algorithms. Information Fusion, page 111-131, Oct 2021. doi: 10.1016/J.inffus.. www.iinfuu..22...0... 20..002.005 URL http:: //dx doi. org/10 1016/j inf fus. 20220022220 02.005
[40] Hao Zhang, Zhuliang Lc, Zhenfeng Shao, Han Xu, and Jiayi Ma. Mff-gan: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Information Fusion. page 40-53, Feb 2021. doi: 10.10166/.inffuus... 2020.08.022. URL /tttp://xx... doi 10r 1016/
j jiinfuua 2020.0200000 18.08..000 022
[41] Pengwei Liang. Junjun Jiang, Xianming Liu, and Jiayi Ma. Fusion from decomposition: A self-supervised decomposition approach for image fusion. In European Conference Computer Vision, pages 719-735. Springer, 2022.
[42] Linfeng Tang, Xinyu Xiang. Hao Zhang, Meiqi Gong, and Jiayi Ma. Divfusion: Darkness- infrared and visible image fusion. Information Fusion, 91:477-499 -4777749,,22 2023.
[43] Chunyang Cheng, Tianyang Xu, and Xiao-Jun Wu. Mufusion: A general unsupervised image fusion network based on memory unit. Information Fusion, 92:80 900,99222220 2023.
[44] Fang Xu, Jinghong Liu, Yueming Song, Hui Sun, and Xuan Wang. Multi-exposure image fusion techniques: A comprehensive review. Remote Sensing. 14(3):771, 2022.
[45] Wesley Roberts, Jan A Van Aardt, and Fethi Babikker Ahmed, Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing, 2(1): 023522, 2008
[46] V. Aslantas and E. Bendes. A new image quality metric for image fusion: The sum of the correlations of differences. AEU International Journal of Electronics and Communications, page 1890-1880 1896. Dec 2015. doi: (0.10166j.aeue.20155 09400. URL/http: hotpt::::p/dddoooooooooooo
doi. orw.://............o0/......0... 1016/J aeue. 2015. 09.15.09.00 00.
[47] Guangmang Cui, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scal image decomposition. Optics Communications, page 199-209, Apr 2015. doi: 1001166.pptcom.20144....4. 12.002 URL batt:://dxx doi
org/10. 1016/ optcom..ppmmmmmmm..mmm. 2014. 12. 032
[48] Ahmet M Eskicioglu and Paul S Fisher. Image quality measures and their performance. IEEE Transactions communications, (((1)))29599296) 1995.
[49] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, page 600-612 Apr 2004. doi: 10.1 109/tip. 2003.819861. URL /tdx: /tw./xxxd.iooooo//ooo/////..o aeeilorg/100 1109/tip...ppp 2003 819861
12

---

[50]
Hao Zhang, Han Xu, Yang Xiao, Xiaojie Guo, and Jiayi Ma. Rethinking the image fusion: A fast unified image fusion network based proportional maintenance of gradient and intensity. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 12797-12804, 2020.
[51] Qiantong Wang, Weihai Chen, Xingming Wu, and Zhengguo Li. Detail-enhanced multi-scale exposure fusion yuv color space. IEEE Transactions on Circuits and Systems for Video Technology. 30(8): 2418- 2429,,242,2222 2019.
[52] Liu and Zengfu Wang. Dense sift for ghost- free multi- exposure fusion. Journal of Visual Communication and Image Representation, 31:208 224, 2015.
[53] Han Xu, Fan Fan. Hao Zhang, Zhuliang Le, and Huang. A deep model for multi-focus image fusion based on gradients and connected regions. IEEE Access, 8:26316 263277 2020.
[54] Sang-hoop hoon Lee, Jae Sung Park, and Nam Ik Cho. exposue image fusion based on the adaptive weights reflecting the relative pixel intensity and global gradient. In 2018 25th IEEE international conference image processing (ICIP), pages 1737-1741. IEEE, 2018.
[55] Xiaohua Qiu, Min Li, Liqiong Zhang, and Xianjie Yuan. Guided filter-based multi-focus image fusion through focus region detection. Signal Processing: Image Communication, 72:35 0::35-46.25 2019.
[56] Hui Li and Lei Zhang. Multi-exposure fusion with emn features. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 1723-1727. IEEE, 2018.
[57] Rui Lai, Yongxue Li, Juntao Guan, and Ai Xiong. Multi-scale visual attention deep convolutional neural network for multi-focus image fusion. IEEE Access, 1:115485-118 114399, 2019.
[58] Kede Ma, Zhengfang Duanmu, Hojatollah Yeganeh, and Zhou Wang. Multi-exposure image fusion by optimizing structural similarity index. IEEE Transactions on Computational Imaging, 4(1):60-72, 2017.
[59] Xu Song and Xiao-Jun Wu. Multi-focus image fusion with pea filters of peanet. In Multimodal Pattern Recognition of Social Signals in Human- Caatuun--omuter--rr Computer-Interuction:: Interaction: 5th IAPR TC 9 Workshop. MPRSS 2018, Beijing, China, August 20, 2018, Revised Selected Papers 5, pages 1-17. Springer, 2019.
[60] Jia Lei, Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang, and Nikola K Kasabov. Galfusion: Multi- exposure image fusion via global local aggregation learning network. IEEE Transactions on Instrumen- tation and Measurement. 72:1 15, 2023.
[61] Boyuan Ma. Yu Zhu, Xiang Yin, Xiaojuan Ban, Haiyou Huang, and Michele Mukeshimana. Sesf-fuse: An unsupervised deep model for multi-focus image fusion. Neural Computing and Applications, 33: 5793-5804, 2021.
[62] Jinyuan Liu, Guanyao Wu, Junsheng Luan. Zhiying Jiang, Risheng Liu, and Fan. Holoco: Holistic and local contrastive learning network for multi-exposure image fusion. Information Fusion, 95:237-249, 2023.
[63] Jinlei Ma. Zhiqiang Zhou, Bo Wang. Lingjuan Miao, and Hua Zong. Multi-focus image fusion using boosted random walks-based algorithm with scale focus maps. Neurocomputing, 335:9-201 2019.
13

---

Appendix
A Proof
A.1 Proof of Theorem 3.1
Proof. By leveraging the properties of convex functions, the loss function be derived the following inequality when Ei= w(t) = 1:
(Ir, =
Σ Ip x
m3)
D(w (x())) - -x( (m
M
w D(E - x(
m=1
1=1
M
M
( |I
(z(1)) 2(m)
Therefore, the fused image Tr can decomposed to uni-source components, By taking the expectation on both sides the inequality, we can derive the generalization error GError(]) of the model on the unknown distribution D.
IID(E - -Z ( =(11))
GError(S) S BEWD
Σ
Σ
w
(9)
For simplicity, we use (1)
(66)) to denote |ID( (z.1)) - 10 Consequently, leveraging the triangle
inequality for norms and fact that D. (m) 1, we have:
M
M
GFirro(f))
BIND
(())
cocom mm))
M (m) f (Ilsm) x (m)
M
Σ
E
Σ
w..
(2o00ma al(m)
m=1
i= 1=1,8mm
M
M
M
EZND
(1
Σ
wi
(2 Foom
(o)) + (M 1) 1) (
(Jia0ms ( xf(m)
M
m=1
$=1,5
EawD (m))
BunD [e (aloonn slmm) }
+ Cov (wim)) (m) x alm)))
EZND
Σ
x
ccom xlm)) + (2(c0m (260m) ml))
(M 1)
Σ w6t)
M
1=,,s
j= 1.
Ex-D [e (ilood) x slm)) }
+ Cov (w) ee (stond x cm()))
M
Σ RaND p (srindd x x()))
Σ
2 (oloms s(m))
+ Σ RanD {e (I.lml) ad())
M
1m1.fm
Σ
Cov e e x (()) (m))
Σ
Coo (wl ( )) EunD
2M
(m) coons
((())
M
Σ
(clben amm))
M
M
m=1
(=1, i=1,imm
consiant
M Cov (m) || ((m) ( (m))
(m)))
(10)
m=1
mmb-source componess
14

---

Table 3: Quantitative comparison on MRI-CT dataset medical image fusion task.
MRI-CT Dataset
Method
ENT
AG1
Elf
SPT
SSIM
CEL
Densefuse
26
4.51
57.06
4.73
12.19
19.37
1.49
4.95
U2Fusion
4.87
6.23
16.20
22.48
0.57
17.45
DeFusion
41
4.60
66.17
5.38
14.22
21.55
1.49
4.63
SwinFuse
20
3.94
72.61
7.04
17.66
35.96
1.45
4.71
MUFusion
45
4.74
79.41
6.44
17.62
21.04
1.38
4,91
DDFM 27
4.59
62.55
5.48
14.03
23.77
1.47
4.75
TC-MoA
3.37
78.62
7.01
26.18
1.42
5.20
IFCNN 18
4.62
61.98
7.86
19.92
31.06
1.49
4.72
IFCNN+TTD
4.66
66.14
8.27
21.11
32.63
1.50
4.63
Improve
20.04
A44.6
20.41
1.199
A1557
2.0.1
2.0.9
CDD 29
4.80
88.18
8.30
20.81
34.32
1.46
4.77
CDD+TTD
4.81
88.12
9.21
23.08
$7.50
1.43
4.81
Improve
A0.01
70.06
7.0.91
2.2.27
A.3.8
70.03
v0.04
PIAFusion
17
4.99
79.98
8.30
21.49
31.42
0.99
6.31
PIAFusion+TTD
4.85
82.09
8.43
21.90
32.45
1.38
5.76
Improve
V0.14
222.1
2.0.13
230.41
A1.03
2.0.39
2.0.55
B Implementation Details
B.1
Algorithm
To accomplish TTD, imitially feed each uni-source image individually into fusion network acquire the respective uni-source components. Then we compute the pixel-wise loss between the uni-source components and their corresponding uni-source images. Finally, utilizing Eq. (6) we obtain the dynamic fusion weight and apply dynamic fusion accordingly.
Here report the algorithm of the whole dynamic fusion strategy in Algorithm |
Algorithm algorithm of dynamic fusion strategy
Input: (m()) Im 1.2.... M
Output:
for each m E [1, M do
2: g(mm) = D (( (z(t)()))) x [m)]
end
4: for each m E M1 do
5: ((()) Softmar(e elon) )
6: end for
7: 1f D Dmm= (m) Blm) (a (m))
8: return
B.2 The Pipeline of TTD
The detailed pipeline for inference shown in Fig. (c). In stage (black dashed line), feed each uni- source image individually into the frozen encoder and decoder to acquire respective decomposed uni-source components. Then, we calculate the RDs according to the colored line in Eq. (6) (c). In stage (solid line), we feed multi-source images into the encoder and get their corresponding features. Then, fuse features by multiplying the RDs the respective features and adding them up. Finally, the fused feature is fed into the decoder for the final fusion results.
C Experimental Results
C.1 Visualization Relative Dominability
noise condition. We simulate a noisy situation which visible image quality affected by contrast perturbation. As shown in Fig. with the corruption severity level increasing, the dominant regions of visible modality gradually reduced, while the unchanged infrared modality gains an increasing RD. Our RD effectively perceives the dominance changes.
RD adaptable
15

---

Re 2 c0
infiured
infrared weight
infrared
inffured weight
Suge?
visible (a) uni-source images
visible weight
visible weight (mask) (b) masked uni-source images
visible
inference pipeline ofT
Figure 6: (a)(b) Visualization of RDs in mask condition. We randomly masked uni-source data. The RD of the region being masked is apparently smaller than the surrounding area, while that of the same region in the infrared image relatively greater. (c) The pipeline of TTD
bigh
The contrast of visible image
Figure 7: Visualization of RDs with varying contrast visible images. With the corruption severity level (contrast perturbation) increasing, the dominant regions visible modality are gradually reduced. Our RD effectively perceives the changes on visible modality in the visualizations, while the unchanged infrared modality gains an increasing RD.
Infrared
Visible
Timeline
Figure 8: Visualizations of RD maps at different times within the same scenario in the LLVIP dataset. As time progresses from day to night, intuitive observation is that the dominance of visible images gradually decreases, while the dominance of infrared images increases.
RD is adaptable to different data qualities. a) The quality of images also changes with illumination. As shown in Fig. we visualized the RDs of the samples at different times in the same scenario. As it changes from day to night, the dominance of visible images gradually decreases, while the dominance of infrared images increases.
b) Furthermore, simulate the malfunction of sensors real scenario, masked the infrared image randomly. As shown F (a),),, the RD region being masked is apparently smaller than the surrounding area, while that of the same region in the infrared image is relatively greater.
C.2 Ablation Study
As shown in the ablation study results of our TTD on four tasks in Fig. our TTD can highlight the advantageous regions in four tasks, improve contrast, and preserve detail compared with baselines. For example, in the VIF task, method enhances the details of people and the shadow textures of trees in the fused images compared the baseline. texture details from MRI in the overlapping regions. the MEF and MFF tasks, the fused images produced by our method have stronger texture details and edge information, well as higher clarity and color fidelity. compared to the baseline fused images.
MIF task, our method maintains the bright information from PET while strengthening the
16

---

Infrared
Visible
PIA
PIA+TTD
PET
PID
CDD
CDD-TTD
IPCNN
JFCNNTTT
CDD
CDD-TTD
IECNN
IPONN-TTD
YORE
YONEY
YDNEY
YOREY
YONE
YDNEY
Underesposure
Overexposure
IFONN
IFCNN-DIF
Foreground Focus
Background Focus
IFONN
IFCNN-TTD
Figure 9: The ablation study TTD on four tasks.
Table Ablation study on different forms of fusion weights on LLVIP dataset.
Forms of weight
EN
SDt
AGT
Elt
SFT
SSIM
CF.
0.5
6.95
37.75
5.18
13.13
18.18
1.32
7.82
So Softmaz(-f)
6.97
38.41
5.24
13.31
18.31
1.35
7.81
2 Softmar Sigmoid(
6.97
38.48
5.36
13.60
18.87
1.33
7.80
O Softmar(e
6.98
38.99
5.48
13.92
19.40
1.34
7.79
Table 5: Ablation study on the normalization of the weights on LLVIP dataset.
Forms normalization
ENT
SDf
AG
Elt
SFT
SSIM
CE
baseline
6.95
37.75
5.18
13.13
18.18
1.32
7.82
w/o norm
29.84
4.60
11.56
16.56
0.95
8.80
Proportional Norm
6.97
38.41
5.24
13.31
18.31
1.34
7.80
softmax(ours)
6.98
38.99
5.48
13.92
19.40
1.34
7.79
is simple but effective method with a straightforward structure, we analyze the effectiveness the TTD from different aspects. We have summarized these ablated analyses here:
(i) ablation study on different baselines: see Sec. Tab. I and Tab. 2
(ii) ablation study on the correlation between weight and loss: see See. 5.1 and Fig.
(iii) ablation study on the ways to obtain weight: see Sec. 5.3 and Fig.
(iv) ablation study on different forms of fusion weights. We compared different forms fusion weight: = 0.5 (baselinc),ur w = Softmex(-f) w So Softmaz( (Sigiiiddddda ()). Softmar(e over IFCNN on the LLVIP dataset, results given in Tab. 4 it shows that forms of fusion can be flexible to achieve the negative correlation between weight and reconstruction loss.
(v) ablation study the normalization of the weights: compared three forms of normalization on the LLVIP, results are given Tab. [ indicating that as premise the generalization theory (see Thm. B.I the normalization of the weights is necessary and the ways to normalize have little impact on our method.
IFCNN
Overall, we performed complete ablation analyses to validate the effectiveness of TTD (i), the necessity of the negative correlation between fusion weight and reconstruction loss (ii), the expandability of ways to obtain fusion weight (ii), the flexibility in the form of weights (iv), the significance of normalization (v).

---

Table 6: Quantitative comparison on MRI-PET dataset in medical image fusion task.
MRI-PET Dataset
Method
ENT
AG1
Elt
SP
SSIM
CEL
Densefuse
26
3.80
49.62
4.32
11.14
15.38
1.48
3.99
U2Fusion
4.31
5.54
19.06
0.43
19.10
DeFusion
41
4.15
63.46
5.78
15.22
21.21
1.51
3.59
SwinFuse
20
2.91
54.09
4.59
11.71
21.63
1.45
3.64
MUFusion
45
3.68
64.75
5.71
15.59
18.51
1.43
3.61
DDFM
27
3.86
54.71
5.16
13.25
18.58
1.44
3.64
TC-MoA
4.83
71.65
6.79
22.52
1.46
4.31
IFCNN 18
3.96
55.29
7.21
18.21
26.69
1.52
3.66
IFCNN+TTD
4.00
58.48
7.80
19.77
28.97
1.52
3.63
Improve
20.04
A3.19
2...9
1.55
22,28
2.0.0
2..03
CDD 29
4.28
81.33
7.68
19.74
27 27.84
1.50
3.57
CDD+TTD
4.27
83.15
8.73
22.03
32.58
1.47
3.59
Improve
v0.01
A1.82
A1105
2V2.29
A4.74
70.03
v0.02
PIAFusion
17
4.43
73.32
8,53
21.96
30.41
0.95
5,88
PIAFusion+TTD
4.33
75.79
8.75
22.59
31.65
1.40
5.39
Improve
V0.10
2.2.47
2.0.22
230.63
A1.24
2.0.45
2s0.49
Table 7: Quantitative comparison on MRI-SPECT dataset in medical image fusion task.
MRI-SPECT Dataset
Method
ENT
SD1
AG1
EI
SFT
SSIM
CE1
Densefuse
3.61
44.36
2.78
7.13
10.74
1.59
3.79
U2Fusion
20
3.89
45.30
10.18
15.67
0.43
19.17
DeFusion
41
3.77
49.80
3.37
8.77
13.32
1.60
3.53
SwinFuse
20
2.79
44.29
2.84
7.24
14.16
1.53
3.74
MUFusion
49
3.52
50.28
4.06
10.96
14.45
1.51
3.60
DDFM 27
3.69
47.74
3.32
8.99
13.18
1.56
3.70
TC-MoA
58.27
16.30
4.43
IFCNN 18
3.63
47.48
4.75
11.96
19.35
1.60
3.61
IFCNN+TTD
3.66
49.08
5.13
12.94
21.10
1.60
3.59
Improve
2.0.3
A.160
2.0.38
2.0.98
A1.75
v0.00
20.02
CDD 29
3.90
71.58
5.21
13.28
20.70
1.58
3.65
CDD+TTD
3.91
74.45
5.69
14.47
23.40
1.55
3.72
Improve
A0.01
A...7
A.0.48
A1.19
/22.0
V0.03
70.07
PIAFusion
17
4.08
61.62
5.60
14:23
21.92
0.97
6.33
PIAFusion+TTI
4.01
63.85
5.68
14.43
22.56
1.46
5,68
Improve
V0.07
22.23
20.08
2.0.20
200.4
400.99
20.45
C.3
Comparison On MIF task
Here we provide the quantitative comparison results on MRI-CT, MRI-PET, and MRI-SPECT datasets in Tab. 3 7 Our method yields competitive performance evaluation metries the three MIF datasets.
C.4 The Effectiveness of TTD on Baselines with Varying Performaces
In Tab. we have applied TTD to various baselines with different capabilities and all achieved consistent enhancement, TTD methods. further validate that our TTD is effective on models with different performances. we conducted additional experiments to apply TTD on models with varying performance levels by adding random Gaussian noise to the pre-trained model (IFCNN) parameters. The results on the LLVIP dataset are given in Tab. B showing that the performance of the baseline decreases with increasing noise added to it. As comparison,
further improve the performance when combined with current state-of-the-art
18

---

Baseline
Baseline+T11)
Haseline
Baseline+11D)
Figure 10: The comparison of detection results between IFCNN and IFCNNTTTD.
Table 8: The effectiveness of TTD on baselines with varying performances.
Noise
Method
ENT
SDT
AGt
Elt
SPT
SCD1
CE
IFCNN
6.95
37.75
5.18
13.13
18.18
1.32
7.82
0.0
IFCNN+TTD
6.98
38.99
5.48
13.92
19.40
1.34
7.79
Improve
A0.03
A1.24
A.0.0
2.0.79
A1,22
A.0.02
2.0.03
IFONN
5.00
12.60
18.10
1.28
7.89
0.1
IFCNN+TTD
6.92
38.01
5.26
13.27
19.33
1.30
7.87
Improve
A0.02
A1.13
7..26
70.67
A1.23
7.0.2
A0.02
IFCNN
6,86
36.61
4.90
12.34
17.44
1.28
8.15
0.2
IFCNN+TTD
6.89
37.95
5.16
13.01
18.36
1.31
8.11
Improve
20.03
A1.34
00.26
2.0.77
2.0.22
00.03
2.0.4
IFCNN
5.93
32.20
4.45
11.16
16.81
1.04
8.33
0.3
IFCNN+TTD
5.96
33.45
4.74
11.88
18.26
1.05
8.23
Improve
2.0.3
A1.25
2.0.29
2.0.72
41.45
A.0.1
2.0.10
IFONN
6.04
4.10
1.01
8.28
0.4
IFCNN+TTD
6.33
36. 16.81
4.69
11.89
17.48
1.11
7.99
Improve
00.29
24.63
0..59
1155
22.10
0..00
0.229
IFCNN
4.49
30.38
3.72
9.33
16.03
0.75
8.99
0.5
ITCN
4.53
35.3
4.31
10.82
18.71
0.82
8.82
Improve
2..04
24.94
2..59
11.49
2.2.69
00.07
00.17
Table 9: Per image inference time of TTD on different baselines
Method
Baselines
Buseline+TTD
CDDFuse
29
1.10
1.77
PIAFusion
2.05
4.02
IFCNN 18
0.006
0.0013
TTD effectively improves all these baselines' performance, indicating the effectiveness and generalizability of TTD various baselines with different performances.
C.5
Results on Downstream Task
First, we train the detection model with visible images from LLVIP dataset. Then we employed our TTD on IFCNN and compared its performance with the baseline on the object detection task. As illustrated in Fig. TT) the detection results of the baseline fused images exhibit missing detection for hard-recognized fast-moving blurred objects. In contrast, after applying our TTD. all objects images obtained using our TTD achieved higher performance in detection lasks compared to the baseline. our DIF shows improvements over the baseline in P. R, and mAP.5:955 metrics.
accurately detected. Additionally. the fused
D Inference Time
The inference time of TTD is dependent the inference time of the baseline. Since TTD is executed in and then compute the fusion weights;
stages: the first stage, we calculate uni-source reconstruction the second stage, we perform the fusion based on the weights. As baselines perform the static fusion, the inference time of TTD is approximately double that of the baseline. We measured the average processing time per image the test set of the LLVIP dataset. The results of the inference time are given in Tab. 9
19

---

E Limitations and Broader Impacts
As test-time dynamic image fusion method. the performance of our TTD significantly depends on the performance of the baseline models. In the future, will try employ the dynamic fusion mechanism in the optimization of bascline models to guide fusion design more effective network, further improving fusion performance. Morcover, in the gradient-based TTD, we select the best gradient empirically, a more adaptive selection approach should be explored in the future. As for the potential social impact, our method performs multi-sensor information fusion, which can be applied to drones, cameras, etc., but it is hard to guarantee the effectiveness of baseline models, which may be risky in high-risk scenarios such medical imaging.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made the abstract and introduction accurately reflect the contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation is discussed the Appendix
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide detailed assumptions and complete proofs in Appendix A
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
21

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide detailed experiment setup in Sec. 4.1 and implementation in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification:
The source code is provided in sttps://github. com/ cow.:onannnnnnnnnoooooo Yinan -iaii-innannnnnnnnnn Xia/TTD
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines http::://nips.cc/ppbb/l/ guides/CodeSubres sei.bboiiion for more details.
While we encourage the release of code and data, we understand that this might not be possihle,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ :/nniss//.///////////e:::eee cc/public/ qu..dess/oooeSSutnlesssoooooo... for more details.
22

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Detal
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide all the implementation details in the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: All the experiments and comparisons are performed with the same settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be elearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap. etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI. if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should he careful symmetric error hars would yield results that are out of range (c.g. negative error rates).
show in tables figures
error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Our experiments are conducted on NVIDIA RTX A6000 GPU and Huawei Atlas 800 Training Server with CANN, and the test time is provided in Tab 9
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
23

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics hatzpa://neurippl httpp//.....ppiiceeccccceceeeeee
Answer:
[Yes]
Justification: Our research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We state this in Appendix
E
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datascts)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
24

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: have checked the related requirements.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No assets provided.
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not study human subjects.
Guidelines:
answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Beard (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
25

---

Justification:
This paper does not conduct experiments with human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Va8VYvfFau", "reviewer": "Reviewer_gBb1", "summary": "- This paper tries to solve the image fusion task, where multi-source images are provided and one needs to extract and integrate effective information from them.\n- The paper demonstrates its effectiveness on four different tasks: VIF, MIF, MEF, and MFF.\n- The paper proposes a test-time dynamic image fusion method with theoretical justification.\n- This paper theoretically proves the superiority of dynamic image fusion over static image fusion, and provides a generalization error upper bound.\n- By using the relative domainability of each source as the dynamic fusion weight, it is able to theoretically improve the generalization of the image fusion model and dynamically emphasize the dominant regions of each source.\n- This method theoretically and empirically demonstrates superiority over static fusion methods through extensive experiments on various datasets, including visible-infrared, medical, multi-exposure, and multi-focus image fusion tasks.", "questions": "-", "limitations": "-", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is well written.\n- The method is evaluated on four different tasks.\n- The approach is fairly simple.\n- The approach is justified theoretically\n- When the baseline model is robust, it can improve the fusion performance.", "weaknesses": "- The adaptation method heavily relies on the performance of the baseline model, which can be ineffective when the model performance is poor.\n- The improvement over the non-adaptive baseline is minor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our **effectiveness on multiple tasks**, **theoretical guarantee**, and **well presentation**. We appreciate your support and constructive suggestions and address your concerns as follows.\n* Weakness 1: The adaptation method heavily relies on the performance of the baseline model, which can be ineffective when the model performance is poor.\n\nThanks for your comment. Please kindly note that our performance is related to the baseline model as we claimed in the Limitations (Line 479-Line 480), however,  it does not imply that our approach would be ineffective on weak-performance baselines. Based on the generalization theory, our TTD explicitly reduces the upper bound of the generalization error of models regardless of the baselines' performance. We have also added experiments to further clarify our effectiveness on the baseline with different performances.\n  We have elaborated on this in two facts:\n\n  (i) **Theoretically**, referring to the deduce in Sec. 3, the fused image can be regarded as a linear combination of multiple uni-source components, i.e. the uni-source reconstruction. We reveal that the model's upper bound of generalization error in the image fusion task is composed of the distance between the uni-source image and each uni-source component as well as the correlation between fusion weight and uni-source component reconstruction loss according to Eq. 4. As the model's encoder and decoder are frozen in the test time, the essence of reducing generalization error lies in the negative correlation between fusion weights and uni-source component reconstruction loss. In static image fusion (baseline), the correlation between weight and reconstruction loss is 0. In contrast, **the dynamic weight in our TTD is negatively correlated to the reconstruction loss**, achieving a reduction in the generalization error compared with the baseline and improving the baseline's performance effectively. Thus, **the key to TTD functioning is independent of the performance of the baseline theoretically**.\n\n  (ii) **Experimentally** , in our paper, we have applied TTD to various baselines with different capabilities and all achieved consistent enhancement, TTD can even further improve the performance when combined with current state-of-the-art methods. To further validate that our TTD is effective on models with different performances, we conducted additional experiments to apply TTD on models with varying performance levels by adding random Gaussian noise to the pre-trained model (IFCNN) parameters. The results on the LLVIP dataset are given in **Tab. A2** (see the global rebuttal PDF), showing that the performance of the baseline decreases with increasing noise added to it. As a comparison, **our TTD effectively improves all these baselines' performance, indicating the effectiveness and generalizability of our TTD on various baselines with different performances**.\n* Weakness 2 : The improvement over the non-adaptive baseline is minor.\n \nThanks for your comment. Please kindly note that our method is independent of the baseline's adaptability, which of TTD is reflected in its ability to dynamically adjust the fusion weights to each sample.\n\n  (i) **The adaptability of TTD**. As analyzed in Question 1, TTD can theoretically reduce the image fusion model's generalization error by constructing the negative correlation between fusion weight and uni-source component reconstruction loss. For each sample, we derive a pixel-level Relative Dominablity (RD) as the dynamic fusion weight, and RD is negatively correlated with uni-source component reconstruction loss according to Eq. 6. Experimental results in Tab. 1-5, the visualization in Fig. 1 and Fig. 6 show that RD-based dynamic fusion weight effectively captures the dominance of each source in image fusion and enhances its advantages in the fused images. Overall, **this adaptability of TTD refers to its adaptive fusion weights instead of the baseline**. \n\n  (ii) **The improvements over various baselines are not minor**. We applied our TTD on three baselines with different performances, and we also conducted extensive experiments on multi-modal, multi-exposure, and multi-focus datasets. The superior performance across diverse metrics demonstrates the effectiveness and applicability of our approach. We further perform additional experiments to combine TTD with baseline models of varying performance. As shown in **Tab. A2** (see the global rebuttal PDF), our TTD significantly improves these baselines with different performances. **This validates that our TTD effectively improves the various baselines in different scenarios as well as with different performances**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KkQOUC4ihz", "reviewer": "Reviewer_UH2o", "summary": "This paper proposes a theoretically guaranteed new paradigm for test-time dynamic image fusion, which exploits the negative correlation between the fusion weights and the single-source reconstruction loss to reduce the upper bound of the generalization error. Extensive experiments demonstrate its effectiveness on a variety of image fusion tasks.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed method is simple and effective，the experimental results are detailed and rich, and the effect is competitive compared to SOTAs.", "weaknesses": "1. Authors does not mention the change in model inference efficiency. \n2. The paper assumes that the decoder of the fusion model is a CNN model, which introduces certain limitations. If the model were a Transformer or Diffusion model, would TDD still be effective?\n3. For specific fusion tasks, it is recommended to compare specific methods rather than generalized methods, e.g., multifocus image fusion tasks should compare multifocus image fusion methods.\n4. Ablation lacks enough persuasiveness.\n5. Table I and II are written inconsistently e.g. TDD and Ours.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful and thorough comments on our paper as well as for recognizing our **theoretical guarantee**, the **simple but effective framework** of our TTD, the **detailed and rich experiments**, and the **competitive effect compared to SoTAs**. We will also make an effort to increase clarity throughout.\n* Weakness 1: Authors do not mention the change in model inference efficiency.\n\nThanks for the constructive comment. The inference time of TTD is dependent on the inference time of the baseline. Since TTD is executed in two stages: in the first stage, we calculate the uni-source reconstruction loss and then compute the fusion weights; in the second stage, we perform the fusion based on the weights. As baselines perform the static fusion, **the inference time of TTD is approximately double that of the baseline**. We measured the average processing time per image on the test set of the LLVIP dataset. The results of the inference time over multiple models are given in **Tab. A4** (see the global rebuttal PDF).\n* Weakness 2: The paper assumes that the decoder of the fusion model is a CNN model, which introduces certain limitations. If the model were a Transformer or Diffusion model, would TDD still be effective?\n\nThanks for the suggestive comment. Although we assume that the decoder of the fusion model is a CNN model, we applied TTD on both **CNN-based** (PIAFusion, IFCNN) and **Transformer-based** (CDDFuse) models, and achieved competitive results, the results are given in Tab. 1 and Tab. 2. In addition, we apply our TTD to a **diffusion-based** image fusion method (Dif-Fusion [1]). The experimental results are given as follows, demonstrating that TTD can be applied to baselines with various network structures and improve their performance on multiple metrics.\n\n| Method | EN | SD | AG | EI | SF | SCD | CE |\n| :------: | :------: |  :------: | :------: | :------: | :------: | :------: | :------: |\n| Dif-Fusion | 7.45 | 50.68 | 3.87 | 10.41 | 13.89 | **1.43** | **7.81** |\n| Dif-Fusion+TTD | **7.45** | **53.52** | **4.75** | **12.80** | **16.62** | 1.31 |  8.14 |\n    \n* Weakness 3: For specific fusion tasks, it is recommended to compare specific methods rather than generalized methods, e.g., multifocus image fusion tasks should compare multifocus image fusion methods.\n\nThanks for the constructive suggestion. We have added comparisons of our TTD with methods specifically applicable to the multi-exposure or multi-focus task, and the results in **Tab. A1** in the global rebuttal PDF shows that our method can outperform these specific methods. \n\n* Weakness 4: Ablation lacks enough persuasiveness.\n\nThanks for the valuable comment. Our TTD is a simple but effective method with a straightforward structure, and we analyze the effectiveness of the TTD from different aspects in our paper. We have summarized these ablated analyses here:\n\n  (i) Ablation study on **different baselines**: see Sec. 4.2, Tab. 1, and Tab. 2.\n\n  (ii) Ablation study on **the correlation between weight and loss**: see Sec. 5.1 and Fig. 5.\n\n  (iii) Ablation study on **the ways to obtain weight**: see Sec. 5.3 and Fig. 5.\n\n  In addition, we added more ablation experiments here: \n\n  (iv) Ablation study on **different forms of fusion weights**. We compared different forms of fusion weight: $w=0.5$ (baseline),$w = Softmax(-\\ell)$, $w = Softmax(Sigmoid(-\\ell))$, $Softmax(e^{-\\ell})$ over IFCNN on the LLVIP dataset, results are given as follows, it shows that forms of fusion can be flexible to achieve the negative correlation between weight and reconstruction loss. \n\n| Forms of weight | EN    | SD     | AG    | EI     | SF     | SCD   | CE     |\n|-------------|-------|--------|-------|--------|--------|-------|--------|\n| $w=0.5$   | 6.95 | 37.75 | 5.18  | 13.13 | 18.18 | 1.32 | 7.82  |\n| $w=Softmax(-\\ell)$    | 6.97 | 38.41  | 5.24 | 13.31 | 18.31 | **1.35** | 7.81  |\n| $w=Softmax(Sigmoid(-\\ell))$     | 6.97 | 38.48  | 5.36 | 13.60 | 18.87 | 1.33 | 7.80 |\n|$w=Softmax(e^{-\\ell})$       | **6.98** | **38.99**  | **5.48**| **13.92** | **19.40** | 1.34 | **7.79**   |\n\n  (v) Ablation study on **the normalization of the weights**: we compared three forms of normalization over IFCNN on the LLVIP dataset, results are given as follows, indicating that as a premise of the generalization theory (see Theorem 3.1), the normalization of the weights is necessary and the ways to normalize have little impact on our method.\n  \nOverall, we performed complete ablation analyses to validate the effectiveness of TTD **(i)**, the necessity of the negative correlation between fusion weight and reconstruction loss **(ii)**, the expandability of ways to obtain fusion weight **(iii)**, the flexibility in the form of weights **(iv)**, the significance of normalization **(v)**.\n\n| Method | EN | SD | AG | EI | SF | SCD | CE |\n| ------------------- | ------- | -------- | ------- | -------- | -------- | ------- | -------- |\n| baseline | 6.95 | 37.75 | 5.18 | 13.13 | 18.18 | 1.32 | 7.82 |\n| w/o norm | 6.57 | 29.84 | 4.60 | 11.56 | 16.56  | 0.95 | 8.80 |\n| Proportional Norm | 6.97 | 38.41 | 5.24 | 13.31 | 18.31 | **1.34** | 7.80 |\n| softmax(ours) | **6.98** | **38.99** | **5.48** | **13.92** | **19.40** | 1.34 | **7.79** |\n\n* Weakness 5: Table I and II are written inconsistently e.g. TDD and Ours.\n\nThanks for the detailed comments. We have modified the claims to avoid inaccurate descriptions, and we have thoroughly reviewed the entire manuscript and corrected these errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4R0lHvRVs", "reviewer": "Reviewer_gbUk", "summary": "This paper proposes a theoretical justification of image fusion from a generalization perspective and reduces the upper bound of generalization error by decomposing the fused image into multiple components corresponding to its source data. A new test-time dynamic image fusion paradigm TTD is further proposed with the finding that the negative correlation between fusion weight and the uni-source reconstruction loss is the key to reducing the generalization loss. Extensive experiments and discussions confirm the theory and superiority.", "questions": "1. Is the idea that adding up every uni-source data linearly to get fused image reasonable enough? Are there any information only found with several data neglected in the whole process?\n2. Mathematical formulas in Appendix A.1 are a little confused. More parentheses ought to be used to present clear explanations about the scopes of every mathematical symbols.\n3. Is the TTD applied to every combination from sources? Most test-time adaption methods only need few data to fine-tune the model. But this paper seems to apply TTD to all source data.", "limitations": "This paper presents a new perspective to analysis the generalization error of the image fusion task, the details presentation and explanation are not very clear, mathematical formulas are little confused and more explanation about inference process should be provided.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea that applying the test-time adaption method into image fusion task with theoretical guarantee is quite meaningful and experiments are sufficient.", "weaknesses": "The details presentation and explanation about test-time adaption are not very clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your valuable comments and appreciate your recognition of the **theoretical justification**, **meaningful and sufficient experiments** as well as the **superiority of our work**. We believe the constructive feedback will improve the paper and increase its potential impact on the community.\n* Weakness 1: The details presentation and explanation about test-time adaption are not very clear.\n\nThanks for your constructive comment. To illustrate the TTD workflow more clearly, we draw a new detailed pipeline during inference in **Fig. A3** (see the global rebuttal PDF). \n\n**In stage 1** (dashed line), we input each uni-source image individually into the frozen encoder and decoder to acquire the respective decomposed uni-source components. Then, we construct the RD which is negatively correlated to the distance between the uni-source component and its original image according to Eq.6. \n\n**In stage 2** (solid line), multiple source images are fed into the encoder at the same time and get their corresponding features. Next, we get the fused feature by multiplying the RDs as weights to their respective features and adding them up. Finally, the fused feature is input into the decoder and the fused image is obtained.\n\n* Question 1: Is the idea that adding up every uni-source data linearly to get fused image reasonable enough? Are there any information only found with several data neglected in the whole process?\n\nThanks for your valuable comment. \n\n  (i) Most recent approaches [18][19] imply **multi-source image fusion aims at integrating comprehensive information from different source data**. The key challenge lies in capturing the effective component of each uni-source data. To address this problem, we theoretically propose a method that can effectively extract single-source information. \n\n* **Theoretically**, the fused image can be regarded as a linear combination of $M$ uni-source components. We reveal that the model's upper bound of generalization error in the image fusion task is composed of the distance between the uni-source image and each uni-source component as well as the correlation between fusion weight and uni-source component reconstruction loss according to Eq. 4. As the model's encoder and decoder are frozen in the test time, **the essence of reducing generalization error lies in the negative correlation between fusion weights and uni-source component reconstruction loss**. TTD performs fusion by the dynamic weight negatively correlated to the reconstruction loss, achieving a reduction in the generalization error compared with the baseline. \n\n* **Empirically**, the fusion weight, e.g. RD (defined in Eq. 6), since fusion models are trained to extract complementary information from each source, the decomposed components of fusion images represent the effective information from the source data. Thus, the uni-source components can be estimated from source data using the fusion model, with the losses representing the deficiencies of the source in constructing fusion images. Negatively correlated to the reconstruction loss, the **RD effectively demonstrates the dominance of each source and highlights the dominant regions as a fusion weight**.\n\n\n(ii) With the neglect of other source data, the uni-source reconstruction can be regarded as the corresponding decomposed component of the fusion image,  it represents the effective information from the source data, thus the loss between it and source data reflects the Relative Dominability (RD) of the uni-source data in constructing the fusion image, i.e., we **leverage the impact of the missing modality on the loss to perceive the RD** of that source in the fusion image and use the RD as fusion weight. This aligns with the theoretical guarantee of image fusion (see (i)): the key to reducing the generalization error in image fusion tasks lies in the negative correlation between fusion weights and uni-source component reconstruction loss.\n* Question 2: Mathematical formulas in Appendix A.1 are a little confused. More parentheses ought to be used to present clear explanations about the scopes of every mathematical symbols.\n\nThanks for the valuable suggestions. Here we provide a detailed explanation of the proof in Appendix A.1. At the second equals sign in Eq. 10, the covariance term is derived from the covariance formula: $\\mathbb E[XY]=\\mathbb E[X]+\\mathbb E[Y]+Cov(X, Y)$. At the second inequality sign in Eq. 10, the expression is expanded by grouping $\\Vert D(E^{(i)}(x^{(i)})) − x^{(m)}\\Vert$ as like terms and expanding the summation notation. Then it is simplified by combining like terms using $w^{(i)}$. At the third inequality sign in Eq. 10, the property of the norm is used, i.e. $\\left\\|a\\right\\|-\\left\\|b\\right\\|\\leq\\left\\|a+b\\right\\|$, to achieve further simplification. Due to character limitations, a more detailed derivation and more easily understandable parentheses have been revised and added in Appendix A.1.\n* Question 3: Is the TTD applied to every combination from sources? Most test-time adaption methods only need a few data to fine-tune the model. But this paper seems to apply TTD to all source data.\n\nThank you for your valuable questions. Most test-time adaptation methods [20][21] require a small amount of data to fine-tune the model. Unlike traditional test-time adaptation methods, our TTD does not require fine-tuning any network parameters. For each sample, the negative correlation between the fusion weight $w$ and the uni-source reconstruction loss can reduce the generalization error upper bound according to the theoretical analysis in the answer to question 2. **TTD performs fusion by the dynamic weight negatively correlated to the reconstruction loss referring to Eq.6 through a single calculation step without any training or fine-tuning**, reducing the generalization error and achieving robust results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dX46KVFhBV", "reviewer": "Reviewer_EtNz", "summary": "This paper introduces a method for dynamically adjusting fusion weights for image pixels based on their relative dominability, calculated using pixel-wise reconstruction losses. This approach aims to minimize generalization error by considering the correlation between fusion weights and reconstruction losses.", "questions": "Please see the weaknesses!", "limitations": "Not very adequate. The limitation is general.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1- The presentation of the method is simple and clear. \n\n2- As the results show, the proposed framework achieves good results on several datasets.", "weaknesses": "1. The claim that RD accurately captures the dominant regions of different sources without solid empirical justification for various scenarios might be overreaching. It is unclear how RD performs under various noise conditions or with sources of different qualities.\n\n2. In Eq. (4), the fusion weights $w$ are important as they determine the contribution of each source to the total loss. However, there is no clear definition for normalizing these weights.\n\n3.  In Eq. (8), the paper lacks discussion on the initialization of $w^(m)$  and how it affects convergence and stability during the dynamic adjustment process.\n\n4. Figure 2 is not clear. How are the feature maps of different layers fused, and what is the impact of each of them on the overall performance of the proposed model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’d like to thank the reviewer for the valuable comments, the acknowledgment of our **good results**, and the **simple but clear presentation**. We provide detailed responses to the constructive comments.\n* Weakness 1: More explanations for RD.\n\nThanks for the valuable comments. We have added experiments under **noise conditions** and with sources of **different qualities**. In addition, our RD's effect was validated on **different methods and tasks**. These experiments and the visualization of RD empirically demonstrate that the proposed dynamic weights (RDs) based on theoretical principles are effective in capturing and emphasizing the dominant regions of different sources, leading to outstanding fusion performance.\n\n  (i) **Comparisons on different tasks over multiple baselines**.\n\n* For different scenarios, we have evaluated our TTD on four image fusion tasks: VIF (see Tab. 1), MIF (see Tab. 3-5), MEF (see Tab. 2), and MFF (see Tab. 2). \n* For different baselines, we have applied our TTD to CDDFuse (CDDFuse+TTD), PIAFusion (PIAFusion+TTD), and IFCNN (IFCNN+TTD), separately. Results are given in Tab. 1-5.\n\n\n(ii) **The visualizations of RDs in various scenarios**.\n\n* **RD is adaptable to the noise condition**. We simulate a noisy situation in which the visible image quality is affected by contrast perturbation. As shown in **Fig. A4**, with the corruption severity level increasing, the dominant regions of visible modality are gradually reduced, while the unchanged infrared modality gains an increasing RD. Our RD effectively perceives the dominance changes.\n\n* **RD is adaptable to different data qualities**. \n    a) To simulate the malfunction of sensors in a real scenario, we masked the infrared image randomly. As shown in **Fig. A2**, the RD of the region being masked is apparently smaller than the surrounding area, while that of the same region in infrared image is relatively greater. b) Furthermore, the quality of images also changes with illumination. As shown in Fig. 6 (see Appendix C.1), we visualized the RDs of the samples at different times in the same scenario. As it changes from day to night, the dominance of visible images gradually decreases, while the dominance of infrared images increases.\n* **RD is adaptable to different tasks**. We have presented visualizations of RDs obtained using CDD+TTD for VIF and MIF tasks, and IFCNN+TTD for MEF and MFF tasks, as CDDFuse is a VIF-specific method while IFCNN is a unified approach. As shown in Fig. 1, our RD reflects the dominance of each source on different tasks adaptively. \n\n\n(iii) **RD's effectiveness over varying performance baseline**. We further conduct additional experiments to combine TTD with baselines of varying performance. As shown in **Tab. A2** of global rebuttal PDF, our TTD significantly improves these baselines with different performance. This validates that our TTD effectively improves baselines in different scenarios and with different performances. \n\n* Weakness 2: More explanations for normalizing the weights.\n\nThanks for the valuable comment. In Eq. 7, we performed a softmax normalization on $w$ as the sum of $w$ being 1 is a prerequisite for deriving the upper bound of generalization error (GError). As shown in **Fig. A1** of global rebuttal PDF, we normalized the weight maps of different sources at the same positions.\n\n* Weakness 3: More explanations for the initialization.\n\nThanks for the constructive comment. The initialization of $w$, which is multiplied with unimodal features, **only participates in the uni-source reconstruction process** to adapt the uni-source features to the feature space of the baseline. If the initialization of $w$ is different from that of the baseline in the reconstruction process, **the feature distribution will deviate from the baseline's feature space**, affecting the reconstruction performance. Accordingly, we have added experiments with different initialization weights during uni-source reconstruction to explore the effect of different initializations. According to **Tab. A3** of global rebuttal PDF, we set $w$ the same as the baseline.\n\nBesides, unlike traditional test-time adaptation methods [20][21], our TTD does not require fine-tuning the network. In the fusion process, $w$ can be obtained by a single calculation step (Eq. 6). Theoretically, the fused image can be regarded as a linear combination of $M$ uni-source components. We reveal that the fusion model's upper bound of GError is composed of the distance between the uni-source image and each uni-source component as well as the correlation between fusion weight and uni-source component reconstruction loss according to Eq. 4. As the model is frozen in the test time, the essence of reducing GError lies in the negative correlation between fusion weight and uni-source component reconstruction loss. **TTD performs fusion by the dynamic weight negatively correlated to the reconstruction loss according to Eq. 6 by a single calculation step without any training or fine-tuning**, reducing the generalization error and achieving robust results.\n  \n* Weakness 4: More explanations for the framework.\n\nThanks for your comments. We draw a more detailed pipeline for inference in **Fig. A3**. \n\n**In stage 1** (dashed line), we feed each uni-source image individually into the frozen encoder and decoder to acquire the respective decomposed uni-source components. Then, we construct the RD according to Eq. 6. \n\n**In stage 2** (solid line), we feed multi-source images into the encoder and get their corresponding features. Then, we fuse features by multiplying the RDs to the respective features and adding them up. Finally, the fused feature is fed into the decoder for the final fusion results.\n\nPlease kindly note that the fusion of different sources only occurs at the fusion layer of the baseline without interactions between features from different layers. We will release all our code for reliable reproduction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear PCs, SACs, ACs, and Reviewers,

We would like to thank you for your valuable feedback and insightful reviews, which have greatly contributed to improving the paper. This is a **clear and well-written** (Reviewer gBb1, Reviewer EtNz) manuscript with a **theoretical guarantee** (Reviewer gBb1, Reviewer UH2o, Reviewer gbUk), we proposed a **effective and superior framework** (Reviewer UH2o, Reviewer gbUk), **the meaningful, detailed and sufficient experiments** on multiple tasks validate the theory and **TTD’s effectiveness and superiority** (Reviewer UH2o, Reviewer gbUk, Reviewer gBb1, Reviewer EtNz).

In our rebuttal, we addressed the following raised concerns/misunderstandings.

  * We have provided a detailed explanation and experimental validation for TTD's performance on baselines with varying performance.

  * We have provided the inference time of TTD and baselines.

  * We have validated TTD's adaptability on Dif-Fusion.

  * We have compared our TTD with MEF-specific methods and MFF-specific methods.

  * We have added more ablation experiments to verify the flexibility in the form of weights and the significance of normalization.

  * We have drawn a new detailed pipeline during inference in Fig. A3.

  * We have visualized the RDs under different noise conditions and with sources of different qualities.

  * We have conducted an ablation study on the initialization of $w$.

We hope that our responses will satisfactorily address your questions and concerns. We sincerely appreciate the time and effort you have dedicated to reviewing our submission, along with your invaluable suggestions. We believe that these clarifications and additional details strengthen our paper and address the reviewers' concerns. We understand the constraints of time and workload that reviewers and AC face, and we appreciate the effort already put into evaluating our work. If there are any additional insights, questions, or clarifications on our responses/submission that you would like to discuss with us, we would be very grateful to hear them, your feedback is invaluable for the improvement of our research.


Best regards,

Authors of Submission 29


## Reference
[1] Yue J, Fang L, Xia S, et al. Dif-fusion: Towards high color fidelity in infrared and visible image fusion with diffusion models[J]. IEEE Transactions on Image Processing, 2023.

[2] Ram Prabhakar K, Sai Srikar V, Venkatesh Babu R. Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs[C]//Proceedings of the IEEE international conference on computer vision. 2017: 4714-4722.

[3] Wang Q, Chen W, Wu X, et al. Detail-enhanced multi-scale exposure fusion in YUV color space[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(8): 2418-2429.

[4] Liu Y, Wang Z. Dense SIFT for ghost-free multi-exposure fusion[J]. Journal of Visual Communication and Image Representation, 2015, 31: 208-224.

[5] Lee S, Park J S, Cho N I. A multi-exposure image fusion based on the adaptive weights reflecting the relative pixel intensity and global gradient[C]//2018 25th IEEE international conference on image processing (ICIP). IEEE, 2018: 1737-1741.

[6] Li H, Zhang L. Multi-exposure fusion with CNN features[C]//2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018: 1723-1727.

[7] Ma K, Duanmu Z, Yeganeh H, et al. Multi-exposure image fusion by optimizing a structural similarity index[J]. IEEE Transactions on Computational Imaging, 2017, 4(1): 60-72.

[8] Lei J, Li J, Liu J, et al. GALFusion: Multi-exposure image fusion via a global–local aggregation learning network[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-15.

[9] Liu J, Wu G, Luan J, et al. HoLoCo: Holistic and local contrastive learning network for multi-exposure image fusion[J]. Information Fusion, 2023, 95: 237-249.

[10] Li J, Guo X, Lu G, et al. DRPL: Deep regression pair learning for multi-focus image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4816-4831.

[11] Amin-Naji M, Aghagolzadeh A, Ezoji M. Ensemble of CNN for multi-focus image fusion[J]. Information Fusion, 2019, 51: 201-214.

[12] Xu H, Fan F, Zhang H, et al. A deep model for multi-focus image fusion based on gradients and connected regions[J].

[13] Qiu X, Li M, Zhang L, et al. Guided filter-based multi-focus image fusion through focus region detection[J]. Signal Processing: Image Communication, 2019, 72: 35-46.

[14] LAI R U I, LI Y, GUAN J, et al. Multi-Scale Visual Attention Deep Convolutional Neural Network for Multi-Focus Image Fusion[J].

[15] Song X, Wu X J. Multi-focus image fusion with PCA filters of PCANet[C]. Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction: 5th IAPR TC 9 Workshop, MPRSS 2018, Beijing, China, August 20, 2018, Revised Selected Papers 5. Springer International Publishing, 2019: 1-17.

[16] Ma B, Zhu Y, Yin X, et al. Sesf-fuse: An unsupervised deep model for multi-focus image fusion[J]. Neural Computing and Applications, 2021, 33: 5793-5804.

[17] Ma J, Zhou Z, Wang B, et al. Multi-focus image fusion using boosted random walks-based algorithm with two-scale focus maps[J]. Neurocomputing, 2019, 335: 9-20.

[18] Xu H, Ma J, Jiang J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.

[19] Tang L, Yuan J, Zhang H, et al. PIAFusion: A progressive infrared and visible image fusion network based on illumination aware[J]. Information Fusion, 2022, 83: 79-92.

[20] Liu Y, Kothari P, Van Delft B, et al. Ttt++: When does self-supervised test-time training fail or thrive?[J]. Advances in Neural Information Processing Systems, 2021, 34: 21808-21820.

[21] Sun Y, Wang X, Liu Z, et al. Test-time training with self-supervision for generalization under distribution shifts[C]. International conference on machine learning. PMLR, 2020: 9229-9248.

Meta-review: The final rating for this paper is two borderline accrpt and two accept. Overall, the reviewers' opinions on the paper were consistent and generally positive. Moreover, most reviewers indicated that the authors had addressed their previous concerns in the rebuttal. Additionally, two reviewers raised their scores after the rebuttal phase. In summary, the overall evaluation of the work is positive, and the reviewers found the authors' responses to be proactive and effective. Therefore, I am inclined to accept this work.
Decision: Accept (poster)