Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model
Authors: Hao Zhang
Lei Cao
Jiayi Ma
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Image fusion
multi-modal fusion
text
diffusion
Primary area: machine_vision
TL;DR: Interactive Multi-modal Image Fusion

Abstract: Existing multi-modal image fusion methods fail to address the compound degradations presented in source images, resulting in fusion images plagued by noise, color bias, improper exposure, etc. Additionally, these methods often overlook the specificity of foreground objects, weakening the salience of the objects of interest within the fused images. To address these challenges, this study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse. First, this framework integrates feature-level information integration into the diffusion process, allowing adaptive degradation removal and multi-modal information fusion. This is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion. Second, by embedding the combination of the text and zero-shot location model into the diffusion fusion process, a text-controlled fusion re-modulation strategy is developed. This enables user-customized text control to improve fusion performance and highlight foreground objects in the fused images. Extensive experiments on diverse public datasets show that our Text-DiFuse achieves state-of-the-art fusion performance across various scenarios with complex degradation. Moreover, the semantic segmentation experiment validates the significant enhancement in semantic performance achieved by our text-controlled fusion re-modulation strategy. The code is publicly available at https://github.com/Leiii-Cao/Text-DiFuse.

Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model
Hao Zhang,
Lei Cao;
Jaiyi Ma
Electronic Information School
Wuhan University
Wuhan, China
(zhpersonalbox, jyma2000)@gmail..cmm whu, cwwlei www.e..wuu.e.. edu.
Abstract
Existing multi-modal image fusion methods fail to address the compound degra- dations presented in source images, resulting in fusion images plagued by noise, color bias, improper exposure, etc. Additionally. these methods often overlook the specificity of foreground objects, weakening the salience of the objects interest within the fused images. To address these challenges, this study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse First, this framework integrates feature-leve information integration into the diffusion process, allowing adaptive degradation removal and multi-modal information fusion. This is the first attempt to deeply and explicitly embed information fusion within diffusion process, effectively addressing compound degradation in image fusion. Second, by embedding the com- bination of the text and zero-shot location model into the diffusion fusion process, text-controlled fusion re-modulation strategy is developed. This enables user- customized text control to improve fusion performance and highlight foreground objects in the fused images. Extensive experiments diverse public datasets show that our Text-DiFuse achieves state-of-the-ie art fusion performance across various scenarios with complex degradation. Moreover, the semantic segmentation experi- ment validates the significant enhancement in semantic performance achieved by our text-controlled fusion modulllaiio strategy. The code is publicly available -iFuse
Introduction
Due to constraints in imaging principles and hardware technology, single-modal images fall short of accurately and comprehensively describing scenes, thereby limiting their utility subsequent tasks. Hence, image fusion technology emerges essential in this context [60 28 It aims to integrate useful information from multi-modal images, producing high-quality visual results that enhance both human and machine perception of scenes. Currently, image fusion technology has been integrated into various tasks, significantly ad advancing performance in related fields such as autonomous driving 47 intelligent security (57 25), and discase diagnosis (14).
Over recent decades, rapid advancements in deep learning have propelled significant progress in image fusion. Deep learning- based methods have surpassed traditional approaches in fusion performance by a considerable margin. the historical context, the evolution image fusion closely aligns with the
"quual Contribution
Cooresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

X=
Po Kr. (XAD
([XY)]))
Diffusion Fusion
1 12 E. (XX
Source Image
Source Imoge
Zo
2r-1
Diffusion FUDION
PaCk PMaD
y-11/0 [110]
Yo
Y-11
Y2
Vr
400%%
DDFM 2023) (Tox:-DFFuse)
Figure 1: Our proposed explicit coupling paradigm of
information fusion and diffusion.
advancements in network paradigms. Autoencoders (AE)
19
20 convolutional neural networks
31 and Transformers 32 represent a coherent axis of progress in image fusion. This phenomenon arises because there is no ground truth supervise fusion learning. Therefore, fusion performance depends heavily on the continuous enhancement of the feature expression potential of neural network paradigms (55).
(CNN) [54 49 , generative adversarial networks (GAN) 29
However, these methods falter in scenes with degradation, especially composite degradation, which we refer to as the composite degradation challenge. Essentially, current methods prioritize multi-modal information integration without considering effective information restoration from degradation [62] 26 Last few years, the emergence of diffusion models has impressed many with their remarkable performance in visual restoration tasks [8] [6]. This prompts a natural question: Can diffusion models be utilized to tackle the challenge of multi-modal image fusion in scenes with complex degradation? According the research status of diffusion model-based image fusion, two factors have hindered the implementation of this intuitive idea. First, visual restoration using diffusion models requires pairs of degraded and clean images, but in image fusion tasks, there is clean fused image as ground truth due to the unsupervised nature of the task. Second, breaking through the paradigm that explicitly couples information fusion and diffusion is yet to be achieved.
Moreover, current fusion methods fail to account for the specificity of objects in the scene (e.g.. pedestrians, vehicles), applying the same fusion rules indiseriminately to both foreground and background. This lack of differentiation, termed the under-customizzatin objects limitation, is unreasonable and may compromise the delineation of crucial objects [53] 64 [1]] Undoubtedly,
maintaining the salience of foreground objects is crucial to satisfy both human and machine interest in them. This necessitates fusion models to possess the capability of interacting with users, achieving a "what you are interested in is what you get" approach.
To address the challenges of composite degradation and under-customizztion objects in multi- modal image fusion, we propose a novel interactive the mot-lloulltedd diffusion model (Text-DiFuse). On the one hand, Text-DiFuse customizes new explicit coupling paradigm of multi-modal information fusion and diffusion, eliminating complex degradation like color casts, noise, and improper lighting, as shown in Fig. D Specifically, it first applies independent conditional diffusion to data with compounded degradation, enabling degradation removal priors to be embedded into the mncoder-decoder network. A fusion control module (FCM) is then embedded between the encoder and decoder to manage the integration of multi-modal features. This involves fusing multiple diffusion processes the feature level, continuously aggregating multi- modal information while removing degradation during T-step sampling. To our knowledge, this is the first time information fusion is deeply and explicitly embedded in the diffusion process, effectively addressing compound degradation in image fusion tasks. On the other hand, to interactively enhance focus on objects of interest during diffusion fusion, we design text-controlled fusion re-modulation strategy. This strategy incorporates text and zero-shot location model to identify the objects of interest, thereby performing secondary modulation with the built-in prior to enhance their saliency. Thus, both the visual quality and semantic attributes of the fused image are significantly improved.
image fusion framework based on
In summary, we make the following contributions:
We propose a novel explicit coupling paradigm of information fusion and diffusion, solving the compound degradation challenge in the task of multi-modal image fusion.

---

A text-controlled fusion re-modulation strategy is designed, allowing users to customize fusion rules with language to enhance the salience of objects of interest. This interactively improves the visual quality and semantic attributes of fused images.
We evaluate our Text-DiFuse on extensive datasets and verify its advantages over state-of-the-an methods in terms of degradation robustness, generalization ability, and semantic properties.
2 Related Work
Deep Multi-modal Image Fusion. As mentioned carlier, the progress in deep multi-modal image fusion is closely tied to updates in neural network paradigms. Initially, AE-based fusion meth- ods 19 20] utilize pre-trained encoders and decoders alongside hand-crafted fusion rules, resulting in performance bottlenecks. Subsequent methods introduce CNN [ 5 and Transformer [37 [58]
for end-to-end fusion guided by specific unsupervised loss, yielding improved performance. The introduction of GAN is groundbreaking due to their inherently unsupervised nature, enabling the preservation of important multi-modal features [29, [56]. However, the instability of the adversarial game often leads non-equilibriun appearances in fused images (30) Furthermore, the diffusion model is highly anticipated for solving image fusion and is used in two main ways: injecting features into CNNs for separate fusion and diffusion [52], or treating
images as conditions for implicit fusion [63]. However, both methods fail to utilize the diffusion model's 's degradation removal embeds feature fusion into the diffusion process, ensuring robustness and aggregation of multi- mollil information. Additionally, we use text combined with zero-shot location model for customuuto fusion, enhancing object salience.
capabilities and struggle with complex degradation. In contrast, our
makes top-notch in visual generation. It constructs Markov chain by progressively adding noise forward, and then estimates the underlying data distribution and uses inverse sampling to generate images. This natural property of degradation removal has made the diffusion model excel in visual restoration tasks [46] [1]. However, the practical application of the diffusion model hindered by its slow T-step continuous sampling. Recent efforts have focused enhancing sampling efficiency and sample quality [50). For example, DDIM 42 extends the original denoising diffusion probability model to non-Markovian scenarios, requiring only discrete time steps during sampling to reduce costs. Furthermore, iDDPM (35) introduces an enhanced denoising diffusion probability model, parameterizing backward variance through lincar interpolation and training with mixed objectives to acquire knowledge of backward variance. This approach increases log-liiel and accelerates sampling rates without compromising sample integrity. Therefore, method employs iDDPM to expedite sampling while upholding the quality of fused images.
Diffusion Model. The impressive performance of the diffusion model 41 [
Zero-shot Location. Establishing connections between unseen and seen categories using semantic information [18), zero-shot location models (27 33 can understand unseen images to identify
and locate designated objects. Representative zero-shot location models include GLIP
22 OWL.
VIT 33 and Grounding DINO (27). Additionally, methods like DiffSeg (40), PADing 12] and
SAM [17 achieve finer pixel-level object localization. These powerful zero-shot location techniques
provide a solid foundation for the implementation of our text-controlled fusion re-modulation.
3 Methodology
3.1 Problem Statement and Modeling
Let us formally define the research problem of this work: achieving multi-modal image fusion under degraded scenes while supporting text-controlled fusion re-modulation of objects of interest. The multi-modal image pair captured under degraded conditions is formulated [[,, Y) Ω], in which {X, Y} denotes clean multi-modal images (e.g., infrared and visible images), and Q indicates composite degradation (e.g., color casts, noise, and improper lighting). We aim to process degraded multi-modal images to obtain a clean fused image: = T[[[,, =(()), The function I' must handle two tasks: degradation removal R and information fusion F. There are two routes: concatenation (T = R+ F) and coupling (T = R F). Concatenation overlooks the intrinsie connection between degradation removal and fusion, leading to limited performance (see comparative experiments).
3

---

X' CRAARON
Encedle Decode
E-02
Ze
No
D-0?
Encode EE
Decode
Favion Control Medale
de
(Mltii-ooda images
Text:
Encode +
Decode Ep
Please hightiht
Palourin
Re-modulation
Ye
Lnybrid
Kandom Degridation
of
Noiey level enbtadding
Figure The pipeline of our Text-DiFuse. (a) The text-controlled diffusion fusion process; (b) training diffusion model for degradation removal; (c) the detailed structure of fusion control module.
Therefore, choose the coupling route and introduce the diffusion model for degradation removal. Then, the key question becomes how to integrate the diffusion model with information fusion. Notably, the diffusion operates as continuous process with multi-step sampling, making it difficult to incorporate fusion. To address this, we propose a novel explicit coupling paradigm of information fusion and diffuusio,, trained on data with compound degradation, incorporating the degradation removal prior into the encoder sye and decoder ei, During T-step reverse sampling, the multi-modal encoded features are continuously passed to the FCM for fusion, aiding in the reconstruction of the final fused image by the decoder. This essentially consolidates multiple diffusion processes into a single one, effectively integrating degradation removal and information fusion. However, the above methodology has not yet addressed the re- eemooulation of objects of interest in image fusion. To this end, further develop text-controlled fusion re-modulation strategy to highlight the objects of interest, thereby enhancing subsequent semantic decision performance. This strategy can be formulated as Z = [[[ [, K,Y } I... where I. represents the user-defined language command. Specifically, we utilize text combined with the zero-shot location module to identify and locate the objects of interest. This knowledge triggers the re-modulation of diffusion fusion to enhance the saliency of the objects with in-built contrast-enhancement prior, thereby significantly improving perceptual quality for both humans and machines.
illustrated in Fig. (a). Specifically, the diffusion model is initially
3.2 Explicit Coupling Paradigm of Information Fusion and Diffusion
Diffusion for Degradation Removal. Embedding the degradation removal prior into the encoder- decoder network of the diffusion model forms the foundation for diffusion fusion. We consider three primary types of degradation: color casts, noise, and improper lighting. They cover both nighttime and daytime negative imaging conditions and can be considered comprchensive to certain extent. In our model, we separate brightness and chrominance components, and perform independent diffusion for them. Here, we describe and represent these two diffusion processes consistently. For clean components s (brightness or chrominance), the corresponding degraded versions (2ss) involve composite degradation like color casts, noise, and improper lighting. These degraded components are fed into the encoder-decoder network as conditions. diffusion process, the original clean component denoted as 80 at step 0, is progressively added with Gaussian noise over 7 steps obtain 8T. the reverse sampling process, the encoder- decoder network is guided to estimate the mean (s R(s), t) and variance (s R(s), f) of sc-1.
shown in Fig. (b), the forward

---

progressively approaching the clean component with the conditions 92(s). Drawing upon iDDPM 35 the optimization can be defined as:
Vo |l 6 -
(1a)
Vo - c0(s, (()), 1|? - 1000, 9(s)),
(Ib)
where Va means optimization by gradient descent, e; denotes the added noise in the forward diffusion process, DKL is the regularization term based KL divergence, and q and po represent the prior and posterior probability distributions, respectively. eg is the noise predictor.
Fusion Control Module for Information Fusion. For information fusion, we design an FCM to aggregate encoded features during the diffusion process, as shown in Fig. (c). It primarily consists of convolution layers with CBAM 145 In them, integrating spatial and channel attention mechanisms helps perceive the importance multi-modal features on a wider scale, promoting rational feature fusion. To reduce the solution space, FCM generates weight coefficients for fusion rather than directly predicting fused features, allowing faster convergence multi-step sampling of the diffusion process.
Diffusion Fusion. Everything is ready, and now we can seamlessly integrate information fusion with diffusion, termed diffusion fusion. Given the degraded multi-modal image pairs [[X,, Y {}[]], we assume X is color image and Y is grayscale image. This assumption aligns with most multi-modal image fusion scenarios, such as visible and infrared image fusion, and MRI and PET image fusion. By separating components, we obtain the brightness component [X*|S2] and chrominance component [X][]]. First, using the trained diffusion model Eğ. we remove the degradation in the chrominance component [1]] obtaining a clean and reasonable chrominance component Xú.
Then, the processing of paired [(x*, {}]] involves diffusion fusion, which achieves simultancous degradation removal and information fusion. Specifically, given randomly sampled Gaussian noise 74. ~ (0, 7), [X*,, [) are regarded as the condition input to the shared encoder eqe in another diffusion model, obtaining features H " H2]
step
Ho-"", 4 H ex [[X", Y (1) 1), (T,..-00
The FCM generates weight coefficients [w,"" ,.v" } for fusing these multi-modal features:
| [2] = [(0,"", Φ H[] o (wr" ,wY).
where | [2]] is the fused feature with residual degradation step f, and @ denotes the Hadamard product. Subsequently, the fused feature is fed into the decoder s to predict the contained noise (t) at step I, and the relevant variable (t) for learning the variance:
Eo(t), (t()) = (1*/11),.). tS),.).
(4)
Then, the mean and variance of 2t_1 can be obtained according to:
at -(z VI ai eo())) no no 2 [Xx Y)
(5)
= Eo ((x", YH = exp(vo(t) log 3r + (1 - (t)) log 3).
(6)
where 3r represents the variance associated with the forward diffusion process, using the notation
Og =1 3, and ã. Io: Additionally, we parameterize the variance between 3 and 3 in
the logarithmic domain using the technique of iDDPM (35] where 3 I-ai be computed according to:
1- 13. Then, 2 can
こ = no (Z., , + (X
(7)
where z denotes the randomly sampled Gaussian noise = ~ N(0, when 1, otherwise
According to Eqs. 4 each sample will derive 28:-
23(2), = z vT ac
(8)
Va
5

---

Notably, (7, ( indicates the corresponding fake final fused image that is derived from the results of any step of sampling. Therefore, we construct constraints to guide the FCM in retaining beneficial information during the diffusion fusion process. Considering pixel intensity and gradient as two basic elements that describe images, we specify intensity loss Line and gradient loss Lgrad to emphasize the preservation of significant contrast and rich texture:
Lint =11 1 (Z. co(t)) max[[x"], |Y II.
(9)
graa =I (o) maxmmmax---- VY} ||:
(10)
where max is the maximum function, V is the Sobel gradient operator, X and Y are clean source components after diffusion. The total loss is summarized as:
Laiff fuiifnn
-inthint + Tigrad
(11)
where Tine and Tarad control the balance of these terms, set to and 0.2, respectively, After
optimization, we obtain the clean fused brightness component 2h = Zg. The purified chrominance component X is used as the fused image's ' chrominance component: ze = X Finally, stitching z" and Z yields the final fused image Z with accurate colors, minimal noise, and proper lighting. Through these designs, information fusion and diffusion have been fully and explicitly coupled, achieving multi-modal image fusion while removing compound degradation.
3.3 Text-controlled Fusion Re-modulation Strategy
The above diffusion fusion constitutes the basic version of our method. Now, we aim to expand it into a modulatable version, allowing users to re-modulate the fusion process based on personalized needs, enhancing the perception of objects of interest. Firstly, we use state-of-the-art zero-shot localization models to identify and locate objects interest based on text commands. Specifically, we introduce OWL-VIT (33) for detecting objects of interest with open-word input. Then, SAM [17] provides pixel-Ievel positioning of these objects, obtaining the mask M. Subsequently. M is fed into the re-modulation block to generate fusion modulation coefficients {xx* "**",, incorporates built-in contrast-enhancee seaanetten prior, aiming to maximize the contrast between the object area and the background in the fused image, thus improving the salience of the objects. Consequently, the multi-modal feature fusion in the diffusion process changes from Eq. (3) to:
}. This block
| / mode [ [[
/ ]10] {wi w @ (x
(12)
In non-object areas, the original distribution of diffusion fusion should maintained:
( r (1 (1- M) 2 M m
(13)
The modulated fused image enhances the saliency of objects compared to the before, making it more suitable for subsequent advanced tasks. We prove this in the re-modulation verification section.
4 Experiments
Configuration. We evaluate our method on two typical multi-modal image fusion scenarios: in- frared and visible image fusion (IVIF) and medical image fusion (MIF). For IVIF, dataset [43], with 485 training and 100 testing image pairs. For MIF, we use the Harvard medicine d with 160 training and 50 testing image pairs, covering CT-MRI, PET-MRI, and SPECT-MRI. Data augmentation like random flipping and cropping increases the training pairs to 12, 888 for IVIF and 6, 408 for MIF. Besides, generalization is evaluated on 60 pairs from LLVIP 15 and 25 pairs
the MSRS
from RoadScene (4). Competitors include 9 methods: REN-Nest (20), GANMcC B1], SDNet [53]. U2Fusion 44 TarDAL (5). DeFusion
23 LRRNet (2T) DDFM (63, and MRFS (58). Five
metrics are used: EN (39, AG 3 SD 38] SCD (2) and VIF [1] The Adam optimizer with
learning rate of 2e - is used for parameter updates. Experiments are conducted on an NVIDIA RTX
3090 GPU and a 3.80 GHz Intel 17-1070KK CPU.
https:: :/tt:s:://www med. harvard edu/ Aww.AAANLIBIIIA Bww.ee////e...eeee html

---

SPECT
MRI
PET
MRI
Source Image
Source Image 2
TarDAL
DeFusion
LRRNet
DDFM
MRFS
Ours
Figure 3: Visual comparison of image fusion methods.
Table I: Quantitative comparison of image fusion methods.
Blod: the best; underline: second best.
MSRS DataSet
Havard Medicine Dataset
Methods
EN 1
AG1 1
SD 1
SCD
VIF f
EN 1
AG f
SD
SCD 1
VIF 1
REN-Nest (InF221)
5.89
1.84
26.03
1.41
0.63
5.34
4,07
63.65
1.58
0.43
GANMcC (TIM'21 21)
6.03
1.91
25.58
1.37
0.65
5.38
5.13
55.00
1.11
0.43
SDNet (UCV'21)
4,90
2.33
16.35
0.91
0.49
5.56
6.22
46.64
0.48
0.38
U2Fusion (TPAMI'22)
5.19
2.46
24.82
1.21
0.52
5.22
6.08
53.20
1.03
0.41
TarDAL (CVPR'22)
3.30
2.04
18.52
0.63
0.15
5.66
5.13
41.94
1.12
0.18
DeFusion (ECCV'22)
6,22
2.31
32.34
1.36
0.75
4,96
4.47
55,45
0.93
0.48
LRRNet (TPAMI'23)
5.89
2.19
26.64
0.75
0.52
5.34
5.39
45.89
0.59
0.40
DDFM (3CCV'23)
5.81
2.65
24.98
1.37
0.62
5.00
5.04
63.53
1.59
0.48
MRFS (CVPR 24)
6,91
2.67
40.95
1.23
0.75
7.24
4.41
70.75
1.53
0.41
Ours (Text-DiFuse)
7.08
3.31
47.44
1.44
0.76
6.44
7.31
80.19
1.69
0.49
Comparative Experiments. We first compare the basic version of our Text-DiFuse with current state-of-the-art fusion methods, and the qualitative results are shown Fig. 3 The first two rows depict IVIF results, demonstrating our method's ability to correct color casts, restore scene information under low-light conditions, and suppress noise. The last three rows display MIF results, which show that method highlight physiological structure information while maintaining functional distribution. In contrast, competitors are unable to achieve such information recovery and still suffer significantly weakened appearance. The quantitative results in Table demonstrate method's d's advantages over other fusion techniques. For further fairness, we introduce state-of- state-of-the-art low-light enhancement (CLIP-LIT (24), denoising (SDAP (6)), and white balance (AWB (1) algorithms as the pre-per preeprocesingg steps for these competitors, with results presented Fig. 4 and Table 2 Clearly, our method still outperforms these comparative methods. This is because these added pre-processing steps for information recovery are entirely independent of information fusion, so they cannot mine habits that more conducive to modal complementarity, leading limited performance.
Generalization Evaluation. Next, we directly test the model trained on the MSRS dataset on the LLVIP and RoadScene datasets to evaluate the generalization ability of the proposed method. We select a daytime scene with overexposure and a low-light nighttime scene, and the qualitative results are shown in Fig. 5 It can be observed that our Text-DiFuse still maintains high-quality degradation removal and information fusion capabilities. In particular, has wo-way information recovery functions such as overexposure correction and low-light enhancement, producing visually satisfying
7

---

CLIS-LIT P AWT Fasson
VIS
CLI-A PAAAISSSP-P-ssii Fision
PET
MRI
Source Image
Source Image?
TarDAL
DeFusion
LRRNet
DDFM
MRES
Ours
Figure 4: Visual comparison of enhancement plus image fusion methods,
Table 2: Quantitative comparison of enhancement plus image fusion methods.
MSRS Dataset
Havard Medicine Dataset
Methods
EN
SD
SCD
VIF
EN
AG
SD
SCD
VIF
Res-Nest
6.43
2.23
27.17
1.38
5.72
4.11
71.46
1.64
GANMcC
6.25
2.06
24.55
1.31
5.80
5.28
66.37
1.19
SDNet
5.84
2.99
20.26
1.08
0.52
5.91
6.00
60.83
1.15
0.30
CLIP-LIT SDAP AWB
U2Fusion
6.55
3,55
29.08
1.32
5.68
71.59
1.56
0.32
TarDAL
5.29
4.42
25.22
1.00
0.35
6.11
4.81
36.54
0.69
0.23
DcFusion
6.31
2.07
25.52
1.16
0.59
6.08
4.27
67.77
1.38
0.35
I.RNNet
6.55
2.68
31.19
1.13
0.54
5.86
5.23
62.91
1.34
0.21
DDEM
6.39
2.43
26.40
1.16
0.60
5.70
4.48
77.40
1.64
0.35
MRFS
6.84
2.86
32.28
1,28
0.58
7.18
4.19
87.53
1.50
0.31
Ours (Text-DiFuse)
7.08
3.31
47.44
1.44
0.76
6.44
7.13
80.19
1.69
0.49
Source Image Source Image?
TarDAL
DeFusion
LRRNEt
DDFM
MRFS
Ours
Figure 5: Visual results of generalization evaluation.
fused results. By comparison, other competitors lose useful information masked by overexposure or low light. We further prove the good generalization ability of our method in Table In general, these results indicate that our
can be applied more reliably in real scenarios.
Re-modulation Verification. We verify the performance gains brought by our text-controlled fusion re-modulation strategy on the MFNet dataset [1O] We select 6 state-of-the-art RGB-T segmen- tation methods for comparison, Le., MFNet 10), FEANet 4 EGFNet [7]. CMX (59), GMNet (65) and MDRNet (48.. Besides, train SegNext 9 on infrared images, visible images, the fused

---

Table 3: Quantitative comparison of generalization ability.
LLVIP Dataset
RoadSeene Dataset
Methods
ENT
AGT
SD
SCD
VIFT
ENT 1
AG T
SD
SCD
VIF
REN-Nest
6.37
2.24
26.66
1.63
0.73
7.37
2.62
46.77
1.66
0.58
GANMcC
6.24
2.09
27.02
1.59
0.65
7.24
3.58
43.68
1.39
0.57
SDNNet
6.00
2.74
23.05
1.24
0.62
7.18
4.86
40.63
1.16
0.66
U2Fusion
5.52
2.69
21.12
1.32
0.61
7.32
4.92
43.99
1.49
0.66
TarDAL
3.85
2.59
23.05
0.22
7.35
11.84
52.30
0.47
DeFusion
2.36
29.48
0.82
6.97
2.85
35.96
0.59
LRRNet
5.67
2.28
19.49
1.06
0.57
7.19
3,55
44.01
1.47
0.58
DDFM
6.46
3.51
30.64
1.72
0.70
7.30
3.63
44.19
1.57
0.65
MRES
7.00
2.34
40.90
1.67
0.86
7.18
2.70
46.57
1.20
0.52
Ours (exu-))Fuse)
7.08
3,99
41.78
1.73
0.87
7.46
2.96
52.84
1.67
0.66
JEN
THt piart phrpe
Mork
ToIs goan boolgh
Thanic
Suse Thaios motll
Figure 6:
Visual results of re-modulation verification.
Table Quantitative verification of re-modulation on semantic segmentation.
Segmentation
Background
Curve
Stop
Color
Bump
MINNGE
RGB-1
96.26
60.95
53.4-1
43.14
22.94
9.14
0.00
18.30
23.47
36.49
FEANCE
RGB-T
87.41
62.74
29.
29.07
EGENet
RGB-T
98.01
87.84
71.12
61.08
46.48
22.
6.64
55.35
47.12
54.76
CMX-B2
RGB-T
97.39
84.23
67.12
$6.93
41.11
39.56
18.94
48.84
54.42
58.31
RGB-T
MDRNet
RGB-T
97.90
87.07
69.81
60.87
47.80
34.18
8.21
50.18
$4.98
$6.78
IR
97.79
84.89
70.73
36.29
41.94
24.13
7.60
33.91
48.64
31.39
SegNext-Base Base
Our basis
98.11
88.56
70.00
64.30
43.07
30.25
11.95
55.14
56.27
57.53
Our modulatable
98.18
72.23
44.79
33.11
13.76
56.32
$8.97
58.63
ii
11
Figure 7: Visual verification in detection scenario.
images generated by the basic version of our method, and the modulatable version respectively to achieve segmentation. It can be from Fig. 6 that different language commands derive customized fused results, which promote the completeness and accuracy of semantic segmentation while visu- ally highlighting the objects of interest. The quantitative results in Table 4 further prove that re-modulation strategy can improve the semantic attributes, achieving the best segmentation scores.
Semantic Verification Detection. We further verify the semantic gain brought by text modula- tion on the object detection task. Specifically, the MSRS dataset 43) is used, which includes pairs of infrared and visible images with two types of detection labels: person and car. Therefore, the text instruction is formulated as: "Please highlight the person and car", which guides our method to enhance the representation of these two types objects the fused image. Then, we adopt YOLO-v5 detector to perform object detection on infrared images, visible images, and fused images generated by various image fusion methods. The visual results are presented in Fig. 7 in which more complete cars and people can be detected from our fused images while showing higher class

---

Table 5: Quantitative verification detection scenario.
Detection
IR
VIS
DeFusion LRRNet DDFM
MRFS Our basis
Our modulatable
mAP@0.5
71.9
74.8
86.6
86.3
88.6
82.0
87.3 56.3
89.7
mAP@ [0.5:0.95]
48.4 47.3
60.1
58.9
59.4
53.2
60.9
IR
Source
Source 2
IV
VI
VII
VIII (Ours)
Figure 8: Visual results of ablation studies.
Table 6: Quantitative results of ablation studies.
Index
Dif.
Dint
Lrrad
FCM
EN
AGT
SDT
SCD1
VIFT
X/max
5.71
1.90
25.66
1.30
0.49
X/add
6.08
1.99
20.14
1.11
0.58
X/mean
5.60
1.49
20.71
1.15
0.56
VI V X
X/variance
5.91
1.90
27.59
0.91
0.46
V
X
6.20
2.75
33.78
1.42
0.73
S
6.67
3.25
45.60
1.43
0.76
VII
X/AE
6.37
2.26
37.48
1.42
0.69
VIII
7.08
3.31
47.44
1.44
0.76
confidence. Furthermore, we provide quantitative detection results in Table [ the highest average accuracy obtained from our fused images, demonstrating the benefits of text modulation. Overall, these results indicate that text control indeed provides significant semantic gains, benefiting downstream tasks.
It can be seen that
Ablation Studies. We conduct ablation studies to verify the effectiveness of specific designs, involving eight variants: removing FCM with using maximum rule; II: removing FCM with using addition rule; II: removing FCM with using mean rule; IV: removing FCM with using variance-
based rule 334; V: removing figrad: VI: removing Lint; VII: removing diffusion with using AE route; VIII: our full model. The visual results in Fig. 7 show that removing any of these designs results in a reduction visual satisfaction. The quantitative scores in Table also support this view. Overall, these designs in our Text-DiFuse collectively guarantee advanced fusion performance.
5 Conclusion
This paper proposes new interactive multi-modal image fusion framework based on the text- modulated diffusion model. On the one hand, it is the first to develop explicit coupling paradigm for information fusion and diffusion models, achieving the integration of multi-modal beneficial information while removing composite degradation. On the other hand, text-controlled fusion re-modulation strategy is designed. It incorporates text combined with the zero-shot location module into the diffusion fusion process, supporting users' language control enhance the perception of objects of interest. Extensive experiments demonstrate that our method achieves better performance than current methods, effectively improving the visual quality and semantic attributes of fused results.
Acknowledgement
This work was supported by NSFC (62276192).
10

---

References
[1] Mahmmoud Afifi, Marcus Brubaker, and Michael S Brown. Auto thite-balance correction for mixed- illuminant scenes. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1210-1219, 2022.
[2] V Aslantas and Bendes. new image quality metrie for image fusion: The sum of the correlations of differences. Aeu-hnternatioaal Journal of Electronics and Communications, 69(12); 12:118900118888 1896, 2015.
[3] Guangmang Cui, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition. Optics Communications, 341:199-209, 2015.
[4] Fuqin Deng, Hua Feng, Mingjian Liang, Hongmin Wang, Yong Yang. Yuan Gao, Junfeng Chen, Junjic Hu, Xiyue Guo, and Tin Lun Lam. Feanet: Feature-enhanced attention network for rgb-thermal real-time semantic segmentation. In Proceedingy of the IEEE/RSJ International Conference Intelligent Robots and Systems, pages 4467-4473, 2021.
[5] Xin Deng and Pier Luigi Dragotti. Deep convolutional neural network for multi modal image restoration and fusion. IEEE Transactions on Puttern Analysis and Machine Intelligence, 43(1))33333 333--33488, 2020.
[6] Wang Di, Liu Jinyuan, Fan Xin, and Risheng Liu. Unsupervised misaligned infrared and visible image fusion via cross moddllty image generation and registration. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 3508 3515, 2022.
[7] Shaohua Dong, Wujic Zhou, Caie Xu, and Weiqing Yan. Egfnet: Edge-aware guidance fusion network for rgb- thermal urban scene parsing. IEEE Transactions on Intelligent Transportation Systems, 25(1): 2023.
[8] Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. Proceedings of the IFFF//VF Conference on Computer Vision and Pattern Recognition, pages 9935-9946, 2023.
[9] Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in Neural Information Processing Systems, 11:1140-11500 2022.
[10] Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems, pages $108-5113 2017.
[11] Yu Han. Yunze Cai, Yin Cao, and Xiaoming Xu. A new image fusion performance metric based on visual information fidelity. Information Fusion, 14(2):127-135, 2013.
[12] Shuting He, Henghui Ding, and Wei Jiang. Primitive generation and semantic-related alignment for universal shot segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11 11238-11244 12247, 2023.
[13] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
[14] Ling Huang, Thierry Denocux, Pierre Vera, and Su Ruan. Evidence fusion with contextual discounting for medical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 401 401-411, 2022.
[15] Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. Livip: A visible-infrared paired dataset for low-light vision. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 3496-384 3504, 2021.
[16] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23606, 2022.
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[18] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between- class attribute transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 951-9551- 1988, 2009.

---

[19] Hui Li and Xiao-Jun Wu. Densefuse: A fusion approach to infrared and visible images. IEEE Transactions Image Processing, 2622226311,,,,,, 2018.
[20] Hui Li, Xiao-Jun Wu, and Josef Kittler. Rfn-nest: An end-to-end residual fusion network for infrared and visible images. Information Fusion, 73:72-86, 2021.
[21] Hui Li. Tianyang Xu, Xiao-Jun Wu, Jiwen Lu, and Josef Kittler. Lrmett A novel representation learning guided fusion network for infrared and visible images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9): 1040- 11052, 2023.
[22] Liunian Harold Li. Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang. et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965- 10975, 2022.
[23] Pengwei Liang, Junjun Jiang, Xianming Liu, and Jiayi Ma. Fusion from decomposition: A self-supervised decomposition approach for image fusion. In Proceedings of the European Conference on Computer Vision, pages 719-73977 735, 2022.
[24] Zhexin Liang, Chongyi Li, Shangehen Zhou, Ruicheng Feng, and Chen Change Loy. Itcrative prompt learning for unsupervised backlit image enhancement. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8094-8103, 2023.
[25] Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Ppattem Recognition, pages 5802-5811, 1, 2022.
[26] Jinyuan Liu, Runjia Lin, Guanyao Wu, Risheng Liu, Zhongxuan Luo, and Xin Fan. Coconet: Coupled con- trastive learning network with multi-level feature ensemble for multi- modality image fusion. International Journal of Computer Vision, 132: 19211740 1748 1775, 2023.
[27] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303. 054999 2023.
[28] Jiayi Ma, Yong Ma, and Chang Li. Infrared and visible image fusion methods and applications: A survey. Information Fusion, 45:153-178. 2019.
[29] Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. Fusiongan: A generative adversarial network for infrared and visible image fusion. Information Fusion, 48:11-24 2019.
[30] Jiayi Ma, Han Xu, Junjun Jiang, Xiaoguang Mei, and Xiao-Ping Zhang. Ddegan: A dusl-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Transactions on Image Processing, 09:4980-4995. 2020.
[31] Jiayi Ma, Hao Zhang, Zhenfeng Shao, Pengwei Liang, and Han Xu. Ganmee: A generative adversarial network with multiclassifcattion constraints for infrared and visible image fusion. IEEE Transactions on Instrumentation and Measurement, 70:1-14, 2020.
[32] Jiayi Ma, Linfeng Tang, Fan Fan, Jun Huang, Xiaoguang Mci, and Yong Ma. Swinfusion: Cross-domain long-range lcarning for general image fusion via swin transformer. IEEE/CAA Journal of Automatica Sinica, 1200:1200-11 2022.
[33] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann. Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Arnb,, Mostafa Debghani, Zhuoran Shen, et al. Simple open- vocabulary object detection. Proceedings of the European Conference on Computer Vision, pages 728-755, 2022.
[34] Nithin Gopalakrishnan Nair, Jeya Maria Jose Valanarasu, and Vishal M Patel. Maxfusion: Plug&play multi-modal generation in text-to-image diffusion models. arXiv preprint (XXX:22404009777 2024.
[35] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of th International Conference on Machine Learning, pages 8162-8171, 2021.
[36] Yizhong Pan, Xiao Liu, Xiangyu Liao, Yuanzhouhan Cao, and Chao Ren. Random sub-samples generation for self-supervised real image denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
12

---

[37] Dongyu Rao, Tianyang Xu, and Xiao-Jun Wu. Tgfuse: An infrared and visible image fusion approach based transformer and generative adversarial network. IEEE Transactions on Image Processing. 2023.
[38] Yun-Jiang Rao. In-fibre bragg grating sensors. Measurement Science and Technology. 8(4):355, 1997.
[39] Wesley Roberts, Jan A Van Aardt, and Fethi Babikker Ahmed, Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing, 2(1): 023522, 2008
[40] Zhihao Shuai, Yinan Chen, Shunqiang Mao, Yihan Zho, and Xiaohong Zhang. Diffieg: A segmentation model for skin lesions based on diffusion difference. arXiv preprint arXiv:2404. 2024.
[41] Jascha Sohl-Dioh Dickstiictt,in,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Machine Learning, pages 2256-2265, 2015.
[42] Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. Proceedings of the International Conference on Learning Representations, pages 1-20, 2021.
[43] Linfeng Tang, Jiteng Yuan, Hao Zhang, Xingyu Jiang, and Jiayi Ma. Piafusion: A progressive infrared and visible image fusion network based on illumination awarc. Information Fusion, 83:79 92, 2022.
[44] Wei Tang, Fazhi He, Yu Liu, and Yansong Duan. Matr: Multimodal medical image fusion via multiscale adaptive transformer. IEEE Transactions on Image Processing, 31:5134-5149 2022.
[45] Sanghyun Woo, Jongchan Park, Joon- Joon-Young I Lee, and In So Kweon. Cham: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision, pages 3-19, 2018.
[46] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13095 13105, 2023.
[47] Jun Xie, Jiazhen Dou, Liyun Zhong, Jianglei Di, and Yuwen Qin. A dual-mode intensity and polarized imaging system for assisting autonomous driving. IEEE Transactions on Instrumentation and Measurement, 73:1-13, 2024.
[48] Lijie Xie, Fubao Zhu, and Ni Yao. Mdr-net: Multiscale dense residual networks for liver image segmenta- tion. IET Image Processing, 17(8):3309225 2023.
[49] Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, and Haibin Ling. U2fusion: A unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):502-514 2022.
[50] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveyx, 56(4):1-39, 2023.
[51] Xunpeng Yi, Linfeng Tang, Hao Zhang, Han Xu, and Jiayi Ma. Diff-if: Multi-modality image fusion via diffusion model with fusion knowledge prior. Information Fusion, 110:102450 2024.
[52] Jun Yue, Leyuan Fang, Shaobo Xia, Yue Deng, and Jiayi Ma. Dif-fusion: Towards high color fidelity in infrared and visible image fusion with diffusion models. IEEE Transactions on Image Processing, 32: 5705-5720, 2023.
[53] Hao Zhang and Jiayi Ma. Sdnet: A versatile squeere-aamddeecooppsiito network for real-time image fusion. International Journal of Computer Vision, 129(10):2761-2785) 2021.
[54] Hao Zhang, Han Xu, Yang Xiao, Xiaojic Guo, and Jiayi Ma. Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12797 12804, 2020.
[55] Hao Zhang, Han Xu, Xin Tian, Junjun Jiang, and Jiayi Ma. Image fusion meets deep learning: A survey and perspective. Information Fusion, 76:323-336, 2021.
[56] Hao Zhang, Jiteng Yuan, Xin Tian, and Jiayi Ma. Gan-fm: Infrared and visible image fusion using gan with full-scale skip connection and dual markovian discriminators. IEEE Transactions on Computational Imaging, 7:1134-11 147, 2021.
[57] Hao Zhang, Tang Linfeng. Xinyu Xiang, Xuhui Zuo, and Jiayi Ma. Dispel darkness for better fusion: A controllable visual enhancer based on cross modal conditional adversarial learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
13

---

[58] Hao Zhang. Xuhui Zuo, Jie Jiang, Chunchao Guo, and Jiayi Ma. Mrfs: Mutually reinforcing image fusion and segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[59] Jiaming Zhang, Huayao Liu, Kailun Yang. Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. JEEE Transactions on Intelligent Transportation Systems, 24(12):1469--14664 2023.
[60] Xingehen Zhang and Yiannis Demiris. Visible and infrared image fusion using deep learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10535-10555 2023.
[61] Zhang, Xiaoyu Shi, Dasong Li, Xiaogang Wang, Jian Wang, and Hongsheng Li. unified conditional framework for diffusion- bbased image restoration. Advances in Neural Information Processing Systems, 36: 49703-49714, 49714, 2024.
[62] Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Shuang Xu, Zudi Lin. Radu Timofte, and Luc Van Gool. Cddfuse: Correlation-diit driven dual-branch feature decomposition for multi-modality image fusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Ppattern Recognition, pages 5906-5916, 2023.
[63] Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu. Yulun Zhang. Kai Zhang. Deyu Meng, Radu Timofte, Luc Van Gool. Ddfm:: denoising diffusion model for multi-modality image fusion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8082-8093, 2023.
[64] Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Kai Zhang, Shuang Xu, Dongdong Chen, Radu Timofte, and Lue Van Gool. Equivariant multi-modality image fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[65] Wujie Zhou, Jinfu Liu, Jingsheng Lei, Lu Yu, and Jenq-Neng Hwang. Gmnet: Graded-fcature multilabel- learning network for rgb-thermal urban scene semantic segmentation. IEEE Transactions Image Processing, 30:7790-7802, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]-
Justification:
The main claims of this work are solve the problem of multi-modal image fusion in degraded environments and to interactively achieve attention and enhancement of objects of interest. These claims correspond to our contributions and are verified in the methodological and experiment sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: We discuss the limitations of this work in the supplementary material. Specifi- cally, our proposed method has lower efficiency, which originates from the slow sampling process of the diffusion model. In the future, we will study the acceleration strategy of the diffusion model and further improve its integration in multi-modal modal image fusion to increase operating efficiency.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limmtations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
This paper does not include theoretical results. Actually, it is pioneering paradigm in which diffusion theory is explicitly embedded in multi-moda image fusion, achieving compound degradation removal while high-quality cross-modal information integration.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]].
This paper aims to propose new multi-modal image fusion algorithm that can adapt to degraded scenes. It is completely reproducible. First, we describe detail the experimental conditions of this work in the experimental configuration section, including datasets, training details, and computing hardware. Second, we provide a URL link to the code of our method in the abstract section, including all the necessary functions for training and inference. We also provide a README file within to outline the necessary environment configuration for rumning, well as detailed usage instructions. Together these ensure the experimental result reproducibility in this paper.
Justification:
Guidelines:
The answer means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
16

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: In the abstract section, we provide URL link to the code of our Text-DiFuse, in which all the necessary functional code for training and inference is included. Besides, we also provide a README file within to outline the configuration of the environment required for running, as well detailed usage instructions. In addition, all datasets used in this work are publicly available and we have provided accurate citations for them. We also describe the processing and partitioning of these datasets in the experimental configuration section.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ctttlpscccppblic//gee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: We describe in detail the experimental conditions this work in the experi- mental configuration section, including datasets, training details, and computing hardware.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
17

---

The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: The experimental results reported in this paper are the average of a large number of test results in the dataset. Therefore. they are statistically significant, being able to support and validate the contributions and claims of this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]-
Justification: the experimental configuration section, we provide the computing resources required reproduce the experiments in this paper, including NVIDIA RTX 3090 GPU and a 3.80 GHz Intel 17-10700K CPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes].
18

---

All data, codes, and methodologies involved in this paper comply with the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes].
Justification: We discuss the potential impacts of this work in the supplementary material. Specifically, this paper is devoted solving the problem of multi-modal image fusion under degraded scenes to provide high-quality fused results suitable for human and machine perception. Therefore, can expected that this work will demonstrate positive social impacts in many fields. For example, it can help drivers better perceive the road conditions ahead in environments with poor visibility through information fusion, such as at night, to improve driving safety. For another example, it can help poor areas that only have low-quality medical imaging equipment to enhance the perception of the body's condition through information recovery and fusion, thereby assisting in disease diagnosis and treatment. As far as we know, this work does not appear to have any negative social impacts and the risks are extremely low.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: This paper poses such risks.
Guidelines:
19

---

The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: All data covered in this paper are publicly available, and we provide accurate citations for them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
The code for our work is provided as a zip file, which already contains an MIT License.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
20

---

Answer:
[NA].
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "U0ZtZOJ5Pr", "reviewer": "Reviewer_DZyq", "summary": "This paper proposes an interactive framework that can exploit the intrinsic connection between image restoration and multi-modal image fusion.\nThe authors embed information fusion within the diffusion process and address the \"composite degradation challenge\" i.e., multi-modal information integration with\neffective information restoration from degradation like colour casts, noise, and improper lighting. Particularly, first, independent conditional diffusion models are applied\nto each modality with compound degradation -- the degradation removal priors are embedded into the encoder-decoder network. A fusion control module (FCM) sits in\nthe multi-step diffusion process to manage the integration of multi-modal features and remove degradation during T-step sampling. Next, to interactively enhance\nfocus on objects of interest during diffusion fusion, the authors designed a text-controlled fusion re-modulation strategy that incorporates a text and a zero-shot OWL-ViT to\nidentify the objects of interest. In other words, this step performs a secondary modulation with the built-in prior to enhance saliency.", "questions": "- In the proposed method, input visual image X is broken into brightness and chroma components. I wonder if this step is absolutely necessary -- or can we skip $\\eta^{c}_{\n    theta}$ and directly combine both $X$ and $Y$ as three-channel images $\\mathbb{R}^{H \\times W \\times 3}$.\n - What if I use InstructIR (for image restoration) followed by MaxFusion (for multi-modal fusion) -- how would it compare with the proposed method?\n    - (InstructIR) https://arxiv.org/pdf/2401.16468 | Github: https://github.com/mv-lab/InstructIR\n    - (MaxFusion) https://arxiv.org/pdf/2404.09977 | Github: https://github.com/Nithin-GK/MaxFusion\n - In the Limitation and Future work section, will a no-training approach be possible? For example, \"MaxFusion\"-like approach but with the proposed deep integration of image restoration and multi-modal fusion.", "limitations": "The authors discussed limitation section in supplementary A.4 -- Particularly, Table S1 shows number of parameters and runtime. This is highly appreciated.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- It is interesting to see the effect of combining image restoration and multi-modal image fusion in a single framework.\n - The proposed method is well-motivated and the authors provide a clear explanation of the method.\n - The Text-controlled fusion re-modulation strategy could be useful in many applications.\n - The authors provide the code in the supplementary material (although I have only dry run the code and not tested it).\n - Extensive experiments are conducted to validate the proposed method.\n - The authors provide ablation studies to show the effectiveness of each component of the proposed method.", "weaknesses": "For now, I have minor concerns and mostly questions (as listed in the next section).\n - The authors should add a brief discussion on the competitors in supplementary material. For example, differences between TarDAL, DeFusion, LRRNet, DDFM, and MRFS.\n - Typo in Eq. 2: $\\Theta_{t}^{X^{B}}$ should be $\\Theta_{t}^{X^{b}}$.\n - Improve the caption of Figure 2. I had to read the entire paper to understand the figure (it should be self-explanatory).\n - Not much of a weakness, but the authors could improve the clarity of the paper if they added the tensor dimension of each variable in Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Discussion on the necessity of the brightness-chrominance separation. \\\nReply: Unlike the image generation task emphasizing diversity, image fusion demands high color fidelity. For instance, in infrared and visible image fusion, the fused image should closely match the colors of the visible image. To this end, our method independently purifies and preserves the chrominance components of the visible image for color fidelity.\\\nWhile a direct processing approach without separating brightness and chrominance is appealing, it faces several challenges in our work. First, from image restoration alone, directly processing color images by the diffusion model is entirely feasible. However, our method integrates information fusion into the latent layers of the diffusion model, which can cause gray infrared features to interfere with the visible image’s color distribution. Such interference in the nonlinear latent space can cause the decoder to amplify small changes into significant color distortions. Second, the interference is related to the feature fusion way. Our method uses a nonlinear neural network, the Fusion Control Module, to aggregate features. This module preserves key thermal objects and background textures, guided by similarity loss functions based on pixel intensity and gradient (see Eqs. (9) and (10)). This optimization can disrupt color distribution due to direct pixel changes. To verify, we adapt our method to process three-channel images without separating brightness and chrominance, as shown in Fig. r1. This strategy causes noticeable color distortion. We also conduct a quantitative evaluation. Table r1 reveals that the direct processing strategy lowers the color score CIECAM16 and negatively impacts other metrics to varying extents.\n\nQ2: InstructIR plus MaxFusion.\\\nReply: MaxFusion is designed for conditional image generation, requiring results to meet conditions like depth, skeleton, segmentation, and edges. Its strength lies in extending single-condition to multi-condition generation through feature fusion. However, MaxFusion is unsuitable for the image fusion task due to lower fidelity. Specifically, image fusion needs the fused image to maintain pixel-level consistency with multiple source images, preserving significant objects and sharp textures. In contrast, MaxFusion focuses on semantic consistency, producing diverse results with different styles. In Fig. r2, using MaxFusion for infrared and visible image fusion with infrared and visible images as conditions shows that it does not meet the goal of enhancing scene representation in image fusion.\\\nThus, we conduct comparisons using the reviewer-recommended InstructIR followed by several advanced image fusion methods. First, we input different text prompts into InstructIR to address improper lighting, noise, and color distortion. The restored images are then fused with advanced fusion methods. Results are shown in Fig. r3 and Table r2. Our method which implicitly integrates image restoration and fusion, shows better performance than these methods following a sequential strategy. In particular, our method can balance thermally salient object retaining and degradation removal, while competitors cannot.\n\nQ3: No-training approach.\\\nReply: MaxFusion extends single-modal image generation models to multi-modal ones through feature-level fusion. Specifically, MaxFusion proposes a no-training fusion strategy, which uses the variance maps of the intermediate feature maps of the diffusion model to select or weighted sum multi-modal features, assuming that pixels with higher variance represent higher priority for condition control.\\\nDifferently, our method uses a neural network, the fusion control module (FCM), which is trained under the constraints of Eq. (11) to fuse multi-modal features aiming at preserving significant objects and textures. Of course, our method can also be extended to a non-training version by using a statistics-based fusion strategy. In the ablation study, we have evaluated three no-training fusion strategies: maximum, addition, and mean (see Table 5).  These methods performed worse than our learnable FCM. Furthermore, we incorporate MaxFusion’s variance-based fusion strategy in our method (see Fig. r4 and Table r3), and it still falls short compared to our FCM. In future research, we will explore more powerful no-training fusion strategies, achieving performance comparable to retraining.\n\nQ4: Discussion on the competitors.\\\nReply: Due to limited space, we discuss only a few newer competitors. TarDAL uses a GAN with dual discriminators to preserve objects and textures, and collaborates with object detection for semantic optimization. DeFusion employs a masked autoencoder to decouple unique and common features, achieving complementary feature aggregation. LRRNet introduces low-rank representation model-based networks and combines pixel-level and feature-level losses for multi-modal fusion. MRFS couples image fusion and segmentation at the feature level to improve fused image quality. However, these methods do not fully address composite degradation, resulting in lower robustness. Given the diffusion model’s strong generative capabilities, applying it to image fusion could address composite degradation issues, but the lack of clean references complicates this. DDFM uses source images to guide sampling direction for implicit fusion, but still cannot solve composite degradation due to the inability to retrain the diffusion model. Our method tackles this by first optimizing the diffusion model and then embedding a fusion module within it, achieving integration of image restoration and fusion. Additionally, our method includes a text control interface for further modulation and enhancement of objects. These designs make our method both robust and flexible.\n\nQ4: Figures, tables, and typos.\\\nReply: We will carefully correct all typos in the final version and further improve the presentation of the figures and tables.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9bi7G96pt1", "reviewer": "Reviewer_TtBn", "summary": "This paper addresses two primary challenges in multimodal image fusion: the mixed degradation of modalities and the insufficient salience of target objects. It proposes two methods to tackle these challenges: feature-level fusion diffusion and the re-modulation of fusion rules in target areas using a zero-shot segmentation model. They implement adequate experiments for evaluation, and the results demonstrate this method's advanced performance across various aspects, including the visual and semantic.", "questions": "1. How were the degradation condition data constructed, were paired supervised datasets used or synthetic datasets?\n2. Has there been an attempt to evaluate the fusion on the fusion of near-infrared and visible bands?\n3. Could you provide the quantitative results of polarization image fusion?\n4. The separation of chrominance and brightness requires more explanation.", "limitations": "Limitations and broader impacts have been included.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ The mixed degradation of modalities and the insufficient salience of target objects are two interesting problems in multimodal image fusion. This paper’s discussion and solution of these two problems may promote the usability of fusion methods in real scenarios. \n+ The information fusion at the feature level is integrated into the diffusion process, which effectively realizes the degradation removal.\n+ The customized object highlighting strategy based on the zero-shot segmentation model is flexible. In particular, its gain in semantic attributes will increase the usability of the fused image in downstream tasks.\n+ This paper conducts lots of comparative experiments and ablation studies on the overall method. \n+ The narrative of this paper is comprehensive and clear. For me, it's easy to follow.", "weaknesses": "- This paper mentioned that the diffusion model is pre-trained to enable the denoising network to have the degradation removal function. However, details about the construction of the data used to train the diffusion model are missing. They need to describe this process to make the overall approach clearer.\n- This paper focuses on multimodal image fusion, being reflected in the title. In the main text, the proposed method is evaluated in two scenarios: infrared and visible image fusion and medical image fusion. In the supplementary materials, they further provide experiments on polarization image fusion. I am curious whether the applicable scenarios of the proposed method can be further expanded, such as the typical fusion of near-infrared and visible bands.\n- The experiments on polarization image fusion only provide visual results, and it would be better to add a quantitative evaluation.\n- I noticed that the proposed method separates the chrominance component and the brightness component, and then performs de-degradation on them separately. An explanation of why this operation is needed should be given. Perhaps an ablation experiment could more intuitively show the effect of this operation.\n- There are some minor typos, such as potential misspellings of dataset names in Tables 1 and 2. In addition, there seems to be a lack of underline on AG's second place.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Dataset for training diffusion model.\\\nReply: In our work, acquiring image restoration capability depends on pre-training a conditional diffusion model, which needs paired clean and degraded data. The clean data are used to build the loss function for supervision, while the degraded data act as conditioning inputs for the denoising network. Therefore, we use existing supervised datasets and additionally simulate a portion of the data to meet the requirements of mixed degradation. \\\nOur method primarily addresses three common types of degradation in the fusion scenario: improper lighting, color distortion, and noise. For improper lighting, we use 2,220 image pairs from the MIT-Adobe FiveK Dataset [r1], covering images with varying exposures and their corresponding ground truth manually adjusted by photography experts. For color distortion, we use 1,031 image pairs from the Rendered WB dataset [r2], including color-biased images under various light sources such as fluorescent, incandescent, and daylight, as well as corresponding reference images manually calibrated under the Adobe standard. For noise, we add Gaussian noise, pulse noise, Poisson noise, Rayleigh noise, and uniform noise to 2,220 clean images from the MIT-Adobe FiveK Dataset and 2,220 clean images from the MSRS dataset to obtain noised images. All these image pairs constitute the complete dataset for training our diffusion model, driving our model’s learning for compound degradation removal.\\\n[r1] Learning photographic global tonal adjustment with a database of input/output image pairs. CVPR 2011.\\\n[r2] When color constancy goes wrong: Correcting improperly white-balanced images. CVPR, 2019.\\\n[r3] PIAFusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 2022.\\\n\nQ2: Expand application scenarios.\\\nReply: This is an insightful suggestion. Of course, our proposed method can be further generalized to other multi-modal image fusion scenarios, as its methodology is a general fusion paradigm. To prove this, we conduct comparative experiments in the near-infrared and visible image fusion scenario. The visual results are shown in Fig. r5, where our method effectively integrates texture details from the near-infrared band with those from the visible image, while preserving natural color attributes of the visible image. Notably, the inherent image restoration capability of our method allows it to produce vivid fused images in underexposed scenes without causing overexposure like MRFS, as seen in the results of the first row. Furthermore, the quantitative results in Table r4 show that our proposed method ranks first in three of all five metrics and second in the other two. Overall, our method can be generalized to near-infrared and visible image fusion scenario with promising performance.\n\nQ3: Quantitative evaluation of polarization image fusion.\\\nReply: We conduct a quantitative assessment of the polarization image fusion, as reported in Table r4. Our method achieves the best scores in four of all five metrics, including EN, AG, SD, and SCD. These results indicate that our fused image contains the most information, the richest texture, the best contrast, and the most feature transfer from the source images. Overall, the quantitative results validate the advantages of our method in the polarization image fusion scenario, demonstrating its strong multi-modal fusion generalization capability.\n\nQ4: Brightness-chrominance separation.\\\nReply: Image fusion requires a high level of color fidelity to the scene. Taking the infrared and visible image fusion as an example, the colors in the fused image are required to be as consistent as possible with those in the visible image. Therefore, by independently purifying and preserving the chrominance components in the visible image, our method can effectively and conveniently achieve color fidelity.\\\nNext, we discuss why our method does not directly process three-channel images. Firstly, from the perspective of image restoration alone, directly processing color images is entirely feasible. However, our method requires embedding information fusion into the latent layers of the diffusion model used for image restoration. This means that features from the gray infrared image could potentially interfere with the color distribution of features from the visible image. In particular, this interference occurs in the highly nonlinear latent space, where some small changes can be amplified by the decoder to produce large color distortions. In this case, ensuring the expected color fidelity is very difficult. Second, the interference is directly related to the way multi-modal features are fused. In our method, we use a nonlinear neural network called the Fusion Control Module to perform information aggregation, which is guided to retain significant thermal radiation objects while preserving rich background textures. These two goals correspond to the similarity loss functions (see Eqs. (9) and (10)) based on the indicators of pixel intensity and gradient. Under such optimization guidance, it is difficult to avoid disrupting the color distribution in the features from the visible image. For verifying, we adapt our proposed method to directly process three-channel images without separating brightness and chrominance components, and the results are presented in Fig. r1. Clearly, color distortion occurs. Furthermore, we implement quantitative evaluation in Table r1. The direct processing strategy decreases the color score CIECAM16 and also negatively affects other metrics to varying degrees\n\nQ5: Typos and underline on $AG$'s second place.\\\nReply: Thanks for pointing out these issues. We will carefully correct all typos in the final version and further improve the presentation of the figures and tables.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tXmEjcGCYh", "reviewer": "Reviewer_RoRE", "summary": "This work focuses on the topic of multi-modal image fusion. Two innovations enhance the performance of the fusion. One is the clever integration of information fusion into the diffusion process. This coupling way enables the fusion function to resist degradation. The other is the introduction of a text-based fusion remodulation strategy. This changes the limitation of previous fusion methods that could only use fixed mappings, allowing for the dynamic adjustment of the fused image based on specific requirements. This remodulation also enhances semantic attributes, improving the scores of the semantic segmentation task.", "questions": "Please answer the question raised in Weaknesses.", "limitations": "Yes, there are discussions about the limitations and potential negative societal impacts.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Integrating information fusion into the diffusion process is novel. Especially, each sampling step triggers an information fusion, which enhances the sufficiency of information fusion. This coupling can ensure the robustness of information fusion, addressing challenges such as low light, noise, and color cast. \n2. The introduction of multi-modal large models is interesting, particularly the ability to remodulate fused images using textual commands. This capability could potentially facilitate the flexible deployment of the proposed method across different application requirements. The demonstration of enhanced semantic attributes and improved semantic segmentation performance is good. \n3. Overall, the experiments are relatively sufficient. The comparative experiments include both baseline comparisons and pre-enhancement comparisons, which are important for ensuring fairness. \n4. The code is provided, which helps in reproducing the performance.", "weaknesses": "1. On page 5, line 174, the source data used for fusion contains degradation, [{Xb,Y}|N]. My question is, in Equations (9) and (10), where do the clean {Xb,Y} used to guide the fusion come from? Is there a multi-modal dataset that contains paired degraded and clean data? The paper seems to lack an explanation for this. \n2. The forward process of the diffusion model involves T steps of noise addition, while the reverse process consists of T steps of iterative sampling. Is the Z0 obtained in equation (8) a hypothetical Z0 derived from the diffusion relation at each sampling, or is it the Z0 after completing the full T steps of sampling? This determines the object of the constraints in the loss functions (9) and (10). It would be better to provide a detailed discussion on this. \n3. Only after the T steps of sampling can the data without degradation be obtained. So why can Z_{t-1}^b in equation (7) be considered free from degradation N? \n4. It's understandable that using textual modulation to control the desired targets of interest can enhance semantic attributes. My question is whether these enhanced semantic attributes can be generalized. In other words, can it also be effective in other high-level visual tasks besides semantic segmentation? \n5. Typo: The Zt on the left side of equation (8) seems to have a missing superscript b.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Clean data for the loss construction.\\\nReply: Constructing Eqs. (9) and (10) actually involves very stringent data requirements. Specifically, they require a pair of degraded multi-modal images describing the same scene, along with their corresponding clean versions. Unfortunately, such a dataset is currently not available. To alleviate this challenge, we adopt a two-step strategy. Specifically, we first pre-train the diffusion model to learn the image restoration capability. In this step, we only need degraded-clean image pairs, without the need for paired multi-modal images that describes the same scene. Once the diffusion model is trained, it can be used to process existing degraded multi-modal image fusion datasets, to generate the required clean multi-modal image pairs. At this point, all the data required for constructing Eqs. (9) and (10) has been obtained.\n\nQ2: Source of the constrained fused image in Eq. (8).\\\nReply: Our method couples image restoration and information integration by inserting a fusion control module within the diffusion model. In other words, each step of sampling will be accompanied by an information fusion. Theoretically, the final fused image requires T steps of continuous iterative sampling to obtain. However, during training, it is inefficient to wait for T steps of sampling to obtain the result and then construct the loss functions of Eqs. (9) and (10). Therefore, we customize Eq. (8) based on the diffusion relationship. According to it, we can derive the corresponding fake final fused image from the results of any step of sampling and apply the corresponding fusion constraints.\n\nQ3: Omission of degradation $N$ in Eq. (7).\\\nReply: Indeed, diffusion models often require a certain number of sampling steps to progressively remove the noise. However, it is needed to clarify that the degradation symbol $N$ in Eqs. (2)-(6) is not the same thing as the noise in the diffusion process. Specifically, the symbol $N$ refers to the degradation from the source images, specifically including improper lighting, color distortion, and random noise. They are used as the conditions of the diffusion model, included in the source image and fed into the denoising network. In contrast, the noise in the intermediate results obtained from continuous sampling of the diffusion model arises from the Gaussian noise assumption of the diffusion theory itself. Eqs. (2)-(6) represent the process of encoding, fusion, decoding, and the estimation of mean and variance for degraded multi-modal images. In this process, the objects being processed contain the degradation from source images, so they all include the condition $N$. Differently, Eq. (7) represents the intermediate result obtained after a single complete sampling step. Therefore, even though early sampling steps still contain Gaussian noise following the diffusion assumption (see Fig. r7), it does not need to include the symbol $N$. We will clarify their distinctions in the final version to avoid misunderstandings.\n\nQ4: Generalization of semantic attributes.\\\nReply: Our method supports the use of textual instructions to re-modulate the information fusion process. Its purpose is to increase the salience of the object of interest, and enhance its presentation quality on the fused image. In our method, no specific downstream task is used to guide this remodulation process, so the gain in semantic attributes is not fixed to any particular task. In other words, beyond the semantic segmentation validation presented in the main text, the semantic attribute gain achieved through textual remodulation can certainly be generalized to other downstream tasks.\\\nFor proving this point, we implement application experiments on the object detection task. Specifically, the MSRS dataset [r1] is used, which includes pairs of infrared and visible images with two types of detection labels: person and car. Therefore, the text instruction is formulated as: \"Please highlight the person and car,\" which guides our method to enhance the representation of these two types of objects in the fused image. Then, we adopt the YOLO-v5 detector to perform object detection on infrared images, visible images, and fused images generated by various image fusion methods. The visual results are presented in Fig. r6, in which more complete cars and people can be detected from our fused images while showing higher class confidence. Furthermore, we provide quantitative detection results in Table r5. It can be seen that the highest average accuracy is obtained from our fused images, demonstrating the benefits of text modulation. Overall, these results indicate that text control indeed provides significant semantic gains, benefiting downstream tasks.\\\n[r1] PIAFusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 2022.\n\nQ5: Typos.\\\nReply: Thanks for pointing out these issues. We will carefully correct all typos in the final version and further enhance the presentation of the figures and tables.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k52PWwQjHg", "reviewer": "Reviewer_NcHo", "summary": "A new paradigm of multi-modal image fusion named Text-DiFuse is introduced, based on the diffusion model. The paradigm embeds a mechanism for aggregating feature-level multi-modal image information into the diffusion process of degrading multi-modal images, addressing the optimization gap between \"degradation removal\" and \"multi-modal information fusion\". Additionally, a zero-shot model is introduced to modulate the fusion strategy based on user-input target text, enhancing the saliency of the target of interest. The conducted experiments suggest significant improvements in both human visual perception and advanced computer vision tasks.", "questions": "Please refer to the weaknesses part.", "limitations": "The authors have analyzed the limitations and potential negative impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1)\tEmbedding the mechanism of aggregating feature-level information into multiple diffusion processes to fuse multi-modal information is interesting. It is foreseeable that this diffusion paradigm produces fused images with better fidelity compared to methods based on likelihood-constrained diffusion models. \n2)\tThe coupled approach effectively resolves the issue of compound degradation in the process of multi-modal fusion, as evidenced by experimental results demonstrating significant advantages over the sequential approach.\n3)\tThe authors emphasize the importance of foreground targets in advanced visual tasks and propose enhancing target saliency through zero-shot assisted re-modulation. This approach diverges from traditional uniform fusion rules, demonstrating effectiveness.\n4)\tThis approach shows strong applicability. It demonstrates superior performance in multiple tasks including infrared and visible image fusion, medical image fusion, and polarization image fusion.", "weaknesses": "1)\tAfter the diffusion model is effectively trained, the sampling process can follow different step intervals. The information fusion in this method is integrated into the diffusion process, but the article does not seem to specify the sampling interval at which the results are obtained. Also, this article does not discuss the impact of the sampling interval on the fusion performance.\n2)\tThe presentation is slightly unclear. For example, from Equation 2 to Equation 6, both the features and the images carry the condition N that represents the degradation. Why does equation 7 no longer include N? Why can it be considered that the degradation has been removed at this point?\n3)\tIn Table 2 and Figure 4, some existing image restoration methods are cascaded in front of the fusion method to promote fairness in comparison, such as low-light enhancement (CLIP-LIT), denoising (SDAP), and white balance (AWB) algorithms. \nPlease explain the choice of the order in which they are connected in series, i.e. why low light enhancement first, then denoising, and finally white balance. \n4)\tModulating the salience of targets of interest in the fusion process through language is novel. Intuitively, I think the improvement in semantic properties brought about by this modulation is widespread. Currently, the effectiveness of language modulation has only been verified in the semantic segmentation scenario. It is recommended to provide an evaluation in the object detection scenario to further verify its role.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Sampling interval and its impact.\\\nReply: In our method, image restoration and information integration are mutually coupled. This is reflected in the physical connection, where a fusion control module is embedded within the internal structure of the diffusion model. Once all the networks are trained, we can follow the standard diffusion model testing procedure, which involves performing T steps of continuous sampling. It is worth noting that information fusion needs to be performed at each sampling step. In this case, the only factor affecting the final fusion result is the number of sampling steps. More sampling steps mean better performance, but they also result in significant time consumption. Therefore, setting an appropriate number of sampling steps is a matter worth discussing.\\\nIn tasks where the ground truth is available, the number of sampling steps can be well determined by checking whether the generated results are sufficiently close to the ground truth. However, for the image fusion task, where ground-truth data do not exist, we rely on visual perception and multiple no-reference metrics to make the assessment. Specifically, we set the number of sampling steps to 2, 3, 4, 5, 10, 25, 500, and 1000, with qualitative and quantitative results shown in Figs. r7 and r8. Notably, each metric is normalized along the step dimension for easier presentation. It can be observed that as the number of steps increases, noise is gradually removed and the scene texture becomes increasingly refined. Corresponding to the quantitative results, 25 steps achieve good performance saturation, with subsequent increases in the number of steps resulting in only slight fluctuations in scores. Note that the only exception is AG, as it is affected by noise during the diffusion process. Therefore, in our experimental section, the number of sampling steps is set to 25.\n\nQ2: Removal of degradation symbols $N$ in Eq. (7).\\\nReply: There are two conceptual differences that need to be clarified first. In all the equations, the symbol $N$ refers to the degradation from the source images, specifically including improper lighting, color distortion, and random noise. In contrast, the noise in the intermediate results obtained from continuous sampling of the diffusion model arises from the Gaussian noise assumption of the diffusion theory itself. Eqs. (2)-(6) represent the process of encoding, fusion, decoding, and the estimation of mean and variance for degraded multi-modal images. In this process, the objects being processed contain the degradation from source images, so they all include the condition $N$. Differently, Eq. (7) represents the intermediate result obtained after a single complete sampling step. Therefore, even though early sampling steps still contain Gaussian noise following the diffusion assumption (see Fig. r7), it does not need to include the symbol $N$. We will clarify their distinctions in the final version to avoid misunderstandings.\n\nQ3: Concatenate order of image restoration algorithm.\\\nReply: In Table 2, we introduce three image restoration algorithms as preprocessing steps of other comparative image fusion methods, including CLIP-LIT, SDAP, and AWB. Among them, CLIP-LIT is a low-light enhancement algorithm, SDAP is a denoising algorithm, and AWB is a white balance algorithm. In the experiment, we follow the processing sequence of low-light enhancement first, followed by denoising, and finally white balance. The choice of this sequence is related to the dependencies among the three types of degradations we are focusing on. Specifically, in low-light images, both scene content and degradation present low-intensity properties, and the signal-to-noise ratio is low. The deep entanglement of noise, color distortion, and useful signals makes degradation removal more challenging. Thus, we first use CLIP-LIT to improve exposure, thereby reducing the difficulty of denoising and color correction. Furthermore, color correction based on white balance requires locating the white light source, and noise interferes with the accuracy of finding the light source. Therefore, we then perform SDAP to remove noise. After addressing exposure and noise, AWB is applied last to achieve color correction.\n\nQ4: Verification of text control on the object detection.\\\nReply: Our method supports text control, enabling the enhancement of the salience of objects of interest based on instructions. Following the reviewer's suggestion, we further verify the semantic gain brought by text modulation on the object detection task. Specifically, the MSRS dataset [r1] is used, which includes pairs of infrared and visible images with two types of detection labels: person and car. Therefore, the text instruction is formulated as: \"Please highlight the person and car,\" which guides our method to enhance the representation of these two types of objects in the fused image. Then, we adopt the YOLO-v5 detector to perform object detection on infrared images, visible images, and fused images generated by various image fusion methods. The visual results are presented in Fig. r6, in which more complete cars and people can be detected from our fused images while showing higher class confidence. Furthermore, we provide quantitative detection results in Table r5. It can be seen that the highest average accuracy is obtained from our fused images, demonstrating the benefits of text modulation. Overall, these results indicate that text control indeed provides significant semantic gains, benefiting downstream tasks.\\\n[r1] PIAFusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank each of the reviewers, area chairs, and program chairs for investing their time and effort into our paper. These valuable comments have enriched our understanding of the research problem and will greatly improve the quality of our manuscript. \
According to the reviewers' comments, we have added some validations. Firstly, we compare a three-channel direct processing strategy without brightness-chrominance separation, demonstrating the necessity of handling brightness and chrominance separately for color fidelity in our method. Secondly, we utilize the state-of-the-art image restoration method InstructIR as a preprocessing step of several advanced image fusion methods for further comparison, showing the advantages of our method with implicit integration of restoration and fusion. Thirdly, drawing inspiration from MaxFusion, we showcase the performance of our method with the non-training fusion strategy. Fourthly, we further extend the application scenario of our method to the near-infrared and visible image fusion, and supplement the quantitative results of the polarization image fusion. Fifthly, we validate the generalizability of the semantic gain offered by our proposed textual control in the object detection task. Finally, we investigate the impact of sampling steps on the final fused result and provide the basis for selecting 25 sampling steps in the experiments. All the figures and tables from the above validations are included in the global PDF file. In addition, we have prepared detailed individual responses for each reviewer to address and clarify all the raised issues and concerns.

Meta-review: The authors have provided a good rebuttal. All four reviewers agreed that this paper proposed a novel and effective multi-modal image fusion method, and their concerns have also been properly resolved by the authors’ rebuttal. The final scores of this paper are all quite positive:  “Weak Accept”, “Weak Accept”, “Accept”, and “Accept”. The meta-reviewer agreed with the reviewers and recommended the acceptance of this paper.
Decision: Accept (spotlight)