A unified lightweight complex scenes-oriented network for infrared and visible image fusion

Authors: Xilai Li, Xiaosong Li, Tianshu TAN, Wuyang Liu, ye tao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: infrared and visible image fusion, complex scenes, unified network, frequency domain, real time

Abstract:
Existing infrared and visible image fusion (IVIF) techniques typically integrate the useful information from different modalities within the ideal conditions. Nevertheless, current state-of-the-art IVIF methods are ineffective when facing complex scene interferences such as bad weather, low light, and high noise, and they typically need to be used in conjunction with other de-interference baselines, which inevitably resulting in the high memory costs and error accumulation, thus yielding sub-optimal fusion results. To address these challenges, We propose a unified lightweight real-time IVIF network for multiple complex scenes. We conducted a theoretically thorough analysis of modal degradations in the frequency domain, leveraging the complementary strengths of both modalities to enhance network learning. Our method facilitates the extraction of critical features even amidst significant pixel interference. For reconstructing fusion results, we introduce a spatial domain branching strategy which significantly improves the local detail resolution, thereby mitigating potential omissions from frequency domain analysis. Extensive qualitative and quantitative experiments demonstrate that our framework excels in handling multiple complex scenes, while maintaining real-time computational efficiency for prompt image processing applications.

Decision: reject
Reviewer scores: 5, 5, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730367542355

thread_type:
official_review

<thread_object>
creator: reviewer_21oj

<content>
summary:
This paper proposes a new IVIF method designed to be robust in complex scenes, including rain, noise, and under-exposed conditions. The core concept is that the amplitude component of infrared images and the phase component of visible images provide complementary modal information for IVIF, and interactively fusing these components enhances fusion quality in challenging scenarios. Specifically, the method decomposes the infrared and visible images into amplitude and phase components by applying the Fourier Transform, thus transferring them into the frequency domain. Based on this, a “multi-modality interactive guidance mechanism” is introduced to fuse the frequency-domain information.

strengths:
+ The idea of using frequency domain information to address IVIF in complex scenes is interesting. 
+ The paper is well-organized, with a logical flow between sections. The language is clear, and the content is presented in an easily understandable manner. 
+ The proposed methods are grounded in theoretical explanations, and the experiments conducted are fair and comprehensive.

weaknesses:
- Despite the demonstrated effectiveness, similar methods have been widely applied in other fields, such as low-light image enhancement and image denoising. For example, “FourLLIE: Boosting Low-Light Image Enhancement by Fourier Frequency Information” transforms the image into the Fourier domain and utilizes an estimated amplitude map for enhancement. How does the proposed method improve upon or differ from these existing methods in the IVIF task?
- The authors claim that existing IVIF methods fail in complex scenes because they may misinterpret interference features as valuable. However, the proposed Spatial Domain Module also fails to avoid erroneous features. The convolution operations in this block extract all local features from the input image, including both degradation artifacts and valuable features. This could lead to the mistaken interpretation of degradation artifacts as valuable features, resulting in error accumulation or the introduction of incorrect features during the fusion stage. Max pooling cannot fully address the issue of misextracting degradation artifacts, especially when these artifacts have high intensity, as max pooling may still retain these distracting features as significant ones. More explanation is needed.
- The final fusion result is obtained by directly combining the source image information with the feature maps. Wouldn't this affect the fusion results, as interference features might also be added to the final image? 
- In Figure 5 (“FFT Domain”), “DWT: Discrete Wavelet Transform” is labeled, but I could not locate it in the figure. Furthermore, there seems to be no explanation of DWT in the paper. Please clarify it.
- Minor issues: There is a typo in Line 018: “, We.” Additionally, in the second column of the “Rain” table in Table 2, the value for LRRNet (“0.8050”) should also be highlighted in red.

questions:
- Same as the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730548154598

thread_type:
official_review

<thread_object>
creator: reviewer_5bes

<content>
summary:
This paper proposes a unified and lightweight framework designed for real-time infrared and visible image fusion in environments characterized by complex interferences from a frequency domain perspective. It tackles the issue of complex scenes fusion problems, such as adverse weather, low-light environments, and noisy fusion. Authors introduce a multi-modality information interaction guidance module for multi-modality feature interaction and extraction. Extensive fusion experiments in four complex conditions: noise, rain, overexposure, and low-light, verified the effectiveness of the proposed method in dealing with interfering information.

strengths:
(1) This paper introduces a unified framework for real-time infrared and visible image fusion in different complex scenes, this is the first work of addressing complex scenes image fusion problems in frequency domain. 

(2) The paper proposes a multi-modality interactive guidance mechanism within the Fourier domain, which efficiently extracts and restores useful features from degraded pixels by leveraging the complementary strengths of different modalities.

(3) The fusion performance of this work is very impressive. Extensive complex scenes fusion experiments cover rain, overexposure, low-light, and noisy demonstrate this method outperforms the state-of-the-art fusion methods in both subject and object evaluations.

weaknesses:
(1) In the caption of figure 5, it is recommended to add the words “complex scenes”.

(2) Section 4.4 and 4.5 could be combined as one part.

(3) The source code is suggested to be public.

questions:
n/a
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622959824

thread_type:
official_review

<thread_object>
creator: reviewer_pge1

<content>
summary:
This model adopts a new frequency-domain perspective to solve the IVIF problem in complex scenes, and uses a multimodal information interaction guidance module that not only utilizes frequency-domain information but also integrates spatial information to more comprehensively and effectively extract image details, compensating for the loss of details that may be caused by relying solely on frequency-domain information. The model has been extensively experimentally validated in four complex scenarios, including noise, rainy weather, overexposure, and low lighting, achieving excellent image fusion quality. Its framework integrates image restoration and fusion, avoiding the problems of error accumulation and irrelevant feature introduction that may occur in traditional two-stage processing methods.

strengths:
1. This paper proposes a multi-modal interactive guidance mechanism that combines frequency domain and spatial domain learning, which effectively enhances the effect of infrared and visible light image fusion in complex scenes. Through mutual guidance, the amplitude of the infrared image and the phase information of the visible light image are used to achieve richer feature extraction and fusion between different modalities.

2. Through a large number of qualitative and quantitative experiments, the article demonstrates the significant performance improvement of this method in removing interference information and restoring image details in complex scenes such as noise, rain, overexposure, and low light. Outperforms existing state-of-the-art methods in multiple metrics.

3. The network design emphasizes lightweight and efficient computing, and can achieve real-time image fusion under limited computing resources, which is a very important advantage in practical applications. Experimental data shows that this method only takes 0.033 seconds for each fusion when processing images of 640×480 size.

weaknesses:
Although the motivation of this article is relatively clear and attempts to solve the problem of image fusion in harsh environments, there are still some shortcomings as follows:
1. First of all, the method proposed by the author does not seem to have any theoretical innovation. Why does the method in the article achieve better performance than image restoration?
2. Although a combination of frequency domain and spatial domain is proposed to extract key features, this method does not have a clear optimization strategy for how to reduce unnecessary redundant information. In complex scenes, this may cause non-critical pixels to be mistaken for valuable information, thereby affecting the fusion effect.
3. The motivation of the article emphasizes the use of frequency domain learning to enhance information extraction under interference conditions, but does not fully consider the important balance of frequency domain and spatial domain information. In the IVIF task, relying solely on the frequency domain may lead to inaccurate processing of spatial details in complex scenes, thereby affecting the fusion quality.

questions:
My questions is already stated in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630559331

thread_type:
official_review

<thread_object>
creator: reviewer_xvyn

<content>
summary:
The paper presents a unified lightweight network designed for infrared and visible image fusion (IVIF) that aims to address the limitations of existing techniques in complex scene conditions such as bad weather, low light, and high noise. The authors conduct a frequency domain analysis of modal degradations, leveraging the complementary strengths of both infrared and visible modalities. A novel spatial domain branching strategy is introduced to enhance local detail resolution in the fusion results. The paper claims extensive qualitative and quantitative improvements in handling complex scenes while maintaining real-time computational efficiency.

strengths:
1. The writing of this paper is easy to understand.
2. The description of the methodology is detailed.
3. The motivation of the methodology is detailed.

weaknesses:
1. The proposed method based on exchanging amplitude and phase lacks novelty.
2. The advantages of the paper are not significant when considering the overall computational cost of the results.
3. The paper should also be compared with other SOTA methods, such as MURF (Xu et al., 2023), SegMiF (Liu et al., 2023), DDFM (Zhao et al., 2023), and EMMA (Zhao et al., 2024).

questions:
What specific types of degradation were most challenging for previous methods, and how does DSPFusion overcome these issues?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731349463751

thread_type:
official_review

<thread_object>
creator: reviewer_eiih

<content>
summary:
This paper addresses the limitations of current infrared and visible image fusion (IVIF) methods when applied to complex scenes with interferences such as rain, noise, and low-light conditions. The authors propose a unified, lightweight IVIF framework that integrates image restoration and fusion in real time, enabling high-quality fusion without the need for pre-processing. Using Fourier domain techniques, the framework captures amplitude information from infrared and visible images and combines these with spatial domain data to maintain clarity and reduce interference. The paper includes extensive qualitative and quantitative experiments to demonstrate the framework’s efficiency and robustness, with results showing improvements in both computational efficiency and image quality across diverse scenes.

strengths:
1.	The method combines frequency and spatial domain information, effectively separating critical scene features and suppressing interference by processing amplitude and phase components in the frequency domain. For example, in low-light or adverse weather conditions, the amplitude information of images is often disrupted (e.g., reduced brightness or increased reflections), while the phase information, reflecting structural aspects of the image, remains relatively stable. By using the amplitude of infrared images to guide the restoration of visible images and leveraging the phase information from visible images to enhance infrared images, the method achieves an optimal balance between the two modalities, ensuring clear detail retrieval in noisy scenarios.
2.	The lightweight design of the method allows for real-time processing with minimal computational resources, which is critical for practical applications. The paper highlights that the model processes a 640x480 image in just 0.033 seconds, demonstrating exceptional efficiency. This efficiency makes it suitable for systems requiring real-time processing, such as autonomous vehicles or industrial monitoring, further broadening its potential application scope.

weaknesses:
1.	The mutual guidance mechanism relies on the complementary characteristics of infrared and visible light, such as using the infrared amplitude to guide the visible amplitude, and vice versa. However, when the data quality of one channel is poor (for instance, when the thermal radiation information of the infrared image is weak), the effectiveness of this guidance mechanism may be significantly limited. In such situations, the original amplitude or phase information might not provide sufficient valuable features to complement the other channel, resulting in suboptimal fusion performance in certain scenes.
2.	The data used in the experiments section was selected from existing datasets (MSRS, AWMM-100k). However, the MSRS training set contains only 1,083 pairs of images, so randomly selecting 1,000 pairs from this limited set seems unnecessary. If the authors intend to augment MSRS, they should uniformly apply Gaussian noise and pixel intensity scaling across the entire MSRS training set.
3.	In Table 1, the authors list non-reference evaluation metrics; however, both SF and AG are gradient-based metrics, which are typically highly correlated. The lack of an information theory-based non-reference metric, such as entropy, is notable. In Table 2, the reference-based metrics include SSIM, but SSIM is already incorporated into the loss function during training for all four tasks, which I believe makes the comparison less fair. Additionally, there is insufficient explanation for why the authors used MSEC’s reconstruction structure in the overexposure scenario.
4.	Did the models of the other comparison methods undergo specific training on the dataset proposed by the authors? In fact, other methods are typically trained only on the original MSRS training set. If the evaluation metrics for these comparison methods are obtained by testing their open-source models on the data across these four scenarios, while the authors’ proposed model was specifically trained for each scenario, then I believe the experimental results lack sufficient credibility.
5.	The comparison methods listed in the experiments section require only a single model to handle all four scenarios, while the proposed method requires separate training for each scenario. In real-world applications, however, rain and overexposure could occur simultaneously. In such cases, the other methods can generalize directly using their models, but which model output would the proposed method select as the result? If the model trained for rain is chosen, would it affect the handling of the overexposed areas?

questions:
Please rebuttal according to the weaknesses item by item.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733304129898

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To PCs, SACs, ACs, and Reviewers

comment:
Dear PCs, SACs, ACs, and Reviewers,

We would like to thank you for your valuable feedback and insightful reviews, which have greatly contributed to improving the paper. Below, we summarize the key strengths of our work as highlighted in the reviews, which we believe support the merit and potential impact of our contribution:

**(1)	Innovative Methodology**

Our work introduces a novel method that effectively addressing image fusion in complex scenes such as noise, low light, and adverse weather conditions. ***Reviewer 5BEs*** emphasized that this is the first study to address complex scene fusion in the frequency domain.

**(2)	Efficiency and Practicality**

The lightweight design enables real-time processing, requiring just 0.033 seconds for a 640×480 image. This was noted by multiple reviewers (***Reviewers Eiih, Pge1, and 5BEs***) as a significant advantage for real-world applications like autonomous driving and industrial monitoring.

**(3)	Experimental Rigor and Performance**

Extensive experiments validate the robustness and superiority of our method over state-of-the-art approaches, as highlighted by ***reviewers Pge1 and 5BEs***.

**(4)	Clarity and Structure**

***Reviewers xvYN and 21oJ*** appreciated the clear writing, logical organization, and detailed explanation of the methodology.

---

**The main additions are as follows:**

-	Added a detailed explanation of the rationale for using the MSEC algorithm. ***(See our response to Reviewer Eiih W3 (3))***

-	Elaborated on the comparison with "image restoration + image fusion" approaches. ***(See our response to Reviewer Eiih C2 (1))***

-	Added experiments with new evaluation metrics, including the no-reference metric $EN$ and the reference-based metric $Q_{P}$. ***(See our response to Reviewer Eiih W3 (1) and (2))***

-	Conducted additional analyses, including computational efficiency comparisons of "restoration + fusion" methods, tests with DA-CLIP as a pre-restoration algorithm, and experiments on standard datasets (MSRS and RoadScene). ***(See our response to Reviewer xvYN W2 and Eiih C2 (2) and C4 (1))***

-	Included tests of the proposed algorithm trained on degraded scenes and evaluated on normal scenes. ***(See our response to Reviewer Eiih C4 (2))***

-	Added comparisons with four state-of-the-art methods: MURF, EMMA, DDFM, and SegMiF. ***(See our response to Reviewer xvYN W3)***

We understand the time constraints and workload that reviewers and Area Chairs face, and we deeply appreciate the effort in evaluating our work. Your feedback is invaluable in improving our research.

Best regards,

Authors of Submission 6459
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the limitations of current infrared and visible image fusion (IVIF) methods when applied to complex scenes with interferences such as rain, noise, and low-light conditions. The authors propose a unified IVIF framework that integrates image restoration and fusion without the need for pre-processing. Using Fourier domain techniques, the framework captures amplitude information from infrared and visible images and combines these with spatial domain data to maintain clarity and reduce interference.
This paper receives significant dispersed scores among the reviews. Reviewers with positive comments believe that the paper is well-described, with clear motivation and good experimental results. However, Reviewer Eiih with negative comments thinks that the methodology has flaws in experimental setup and model generalization, while Reviewer xvYN feels that the method of exchanging amplitude and phase lacks novelty. Thus, this paper is slightly below the acceptance threshold.