FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining

Authors: Dong Li, Yidi Liu, Xueyang Fu, Jie Huang, Senyan Xu, Qi Zhu, Zheng-Jun Zha
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Fourier transform, mamba, image deraining

Abstract:
Image deraining aims to remove rain streaks from rainy images and restore clear backgrounds. Currently, some research that employs the Fourier transform has proved to be effective for image deraining, due to it acting as an effective frequency prior for capturing rain streaks. However, despite there exists dependency of low frequency and high frequency in images, these Fourier-based methods rarely exploit the correlation of different frequencies for conjuncting their learning procedures, limiting the full utilization of frequency information for image deraining. Alternatively, the recently emerged Mamba technique depicts its effectiveness and efficiency for modeling correlation in various domains (e.g., spatial, temporal), and we argue that introducing Mamba into its unexplored Fourier spaces to correlate different frequencies would help improve image deraining. This motivates us to propose a new framework termed FourierMamba, which performs image deraining with Mamba in the Fourier space. Owing to the unique arrangement of frequency orders in Fourier space, the core of FourierMamba lies in the scanning encoding of different frequencies, where the low-high frequency order formats exhibit differently in the spatial dimension  (unarranged in axis) and channel dimension (arranged in axis). Therefore, we design FourierMamba that correlates Fourier space information in the spatial and channel dimensions with distinct designs. Specifically, in the spatial dimension Fourier space, we introduce the zigzag coding to scan the frequencies to rearrange the orders from low to high frequencies,  thereby orderly correlating the connections between frequencies; in the channel dimension  Fourier space with arranged orders of frequencies in axis, we can directly use Mamba to perform frequency correlation and improve the channel information representation. Extensive experiments reveal that our method outperforms state-of-the-art methods both qualitatively and quantitatively.

Decision: reject
Reviewer scores: 5, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729088054866

thread_type:
official_review

<thread_object>
creator: reviewer_nhz8

<content>
summary:
The paper presents a novel approach to image deraining by integrating Fourier transformation with a state-space model. Unlike previous methods, the proposed approach applies the state-space model in both the spatial and frequency domains, introducing a new scanning strategy referred to as zigzag scanning. This strategy gradually processes both low- and high-frequency components, aiming to enhance the model's ability to capture rain-related distortions. The key contribution of the paper lies in the introduction of this zigzag scanning mechanism, which differentiates it from existing methods.

strengths:
The proposed model involving zigzag scanning is novel and intuitive.

The paper provides a thorough and intuitive discussion of the advantages of zigzag scanning in the context of the deraining task.

The proposed model demonstrates state-of-the-art performance, balancing accuracy with reasonable computational cost and model size.

weaknesses:
Although the motivation for zigzag scanning is well-discussed, the paper lacks details on its implementation. Considering the complexity involved, the implementation is non-trivial and deserves attention. The way zigzag scanning is executed, along with pixel organization strategies, can significantly impact inference speed. Including a comparison of running times would enhance the evaluation.

The visual comparison baselines could also be updated. It would be beneficial to include comparisons with more recent methods (post-2022) and provide additional zoomed-in patches or error maps to better highlight performance differences.

There are several confusing typographical issues, such as "FCS" being used as the abbreviation for "Fourier Channel Evolution" and "C-FFT" lacking a clear explanation (possibly referring to "channel FFT"). These should be clarified.

Furthermore, the paper overlooks several closely related works, including:
[1] Wavelet Approximation-Aware Residual Network for Single Image Deraining, 2023.
[2] A Hybrid Transformer-Mamba Network for Single Image Deraining, 2024.
[3] Image Deraining with Frequency-Enhanced State Space Model, 2024.

Additionally, applying the L1 norm directly in the frequency domain may be less accurate than alternative metrics, such as coherence-based distances. An ablation study on the impact of different frequency-domain loss functions would strengthen the analysis.

Finally, the zigzag scanning strategy appears intuitive for other low-level vision tasks, such as deblurring. It would be valuable to investigate the model's performance on additional tasks to assess its generalizability.

questions:
The main points of confusion, as outlined in the Weakness section, are as follows:

**Implementation and Efficiency of Zigzag Scanning**:
The paper does not provide sufficient details on how the zigzag scanning is implemented, despite its potential complexity. Additionally, it is unclear how efficient the scanning strategy is in practice. A more thorough discussion of the implementation and its impact on inference speed would strengthen the work.

**Generalization to Other Tasks**:
Given that the motivation for zigzag scanning is not specific to rain removal, it would be valuable to evaluate the performance of the proposed method on other low-level vision tasks. Demonstrating its effectiveness beyond the deraining scenario would provide further evidence of its broader applicability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730475090281

thread_type:
official_review

<thread_object>
creator: reviewer_xrkz

<content>
summary:
This paper proposed a new network for image deraining by introducing the state space models, or more specifically, the recently proposed Mamba structure, in the Fourier spaces. By virtue of the Mamba, the frequency correlations can be better captured and utilized in the Fourier domain, such that the better deraining performance can be expected. Experiments on various deraining datasets have been conducted to demonstrate the effectiveness of the proposed method.

strengths:
- The application of Mamba to the Fourier domain is new and worthy to explore.

- The experimental results are promising

weaknesses:
- The motivation should be strengthened. Currently, it seems the work is mostly about applying Mamba in the Fourier domain, though this task itself could be non-trivial. However, it is unclear why Mamba is more effective than other architectures for processing the Fourier frequencies. Visualizing the intermediate features might be useful.

- As reviewed by the authors, another work (Zhen et al., 2024) considers Mamba in the frequency domain but with wavelet transformation. It is better to discuss more about wavelet transformation and Fourier transformation in dealing with frequencies. Besides, as shown in Table 1, the performance of (Zhen et al., 2024) is close to the proposed method on several datasets, but the comparison with this method on SPA-Data is missing.

- The quantitative comparison is conducted with respect to PSNR and SSIM. However, as more and more researchers realized, these two metrics can be often inconsistent with human perceptions and thus some newly developed metrics that can better reflect human perceptions should be considered.

questions:
- In Section 4.1, the authors mentioned the "progressive training strategy". What does it mean? And why is it called "progressive"?

- There are several writing issues that I can find:
1) Line 75-76: "...build correlation the correlation among...";
2) The rightmost of Line 73: ".arrangement";
3) According to the formatting of other parts of this manuscript, the "Fourier transformation." in Section 3.1 should not take the whole line.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558519383

thread_type:
official_review

<thread_object>
creator: reviewer_1tv5

<content>
summary:
This paper proposes Fourier Mamba, an image rain removal framework. It improves the rain removal effect by applying the Mamba technique in Fourier space and utilizing the correlation between different frequencies. This method uses zigzag encoding to rearrange the frequency order in the spatial dimension and directly uses Mamba for frequency correlation in the channel dimension. Experiments have shown that Fourier Mamba has achieved more competitive results in image rain removal tasks.

strengths:
1. The proposed framework combines Fourier priors and state space models to correlate different frequencies in Fourier space to enhance the rain removal effect of images. This combination utilizes the advantages of the Fourier transform in global modeling and introduces the efficient feature modeling capability of the Mamba model.

2. To rearrange the order from low frequency to high frequency in the Fourier space of the spatial dimension, Fourier Mamba proposed a scanning method based on zigzag encoding, which systematically associates different frequencies. This method introduces zigzag encoding in Fourier space to rearrange the frequency order, thereby orderly associating the connections between frequencies.

weaknesses:
1. The experiment is insufficient and lacks comparison with methods such as DRSformer [1] and FADformer [2].
2. I am a bit concerned about the quantitative results in this paper. The difference between the results shown in Table 1 and FreqMamba is not significant. Please further explain the differences between the proposed method and FreqMamba, as they both use Fourier correlation strategies. It is recommended to add visualization results to supplement this.
3. It is worth noting that FFT is a complex valued computation. Suggest the author to add a comparison of the actual inference time of the model. FLOP comparison is not enough, because many works now have lower FLOP but slower running speed, and other issues such as memory consumption may slow down computation speed.
[1] Chen X,  Li H, Li M, et al. Learning a sparse transformer network for effective image deraining.
[2] Gao N,  Jiang X, Zhang X, et al. Efficient Frequency-Domain Image Deraining with Contrastive Regularization.

questions:
Please see the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646816363

thread_type:
official_review

<thread_object>
creator: reviewer_yyrc

<content>
summary:
This paper presents a Mamba-based image deraining algorithm. The proposed FourierMamba mainly involves two components: spatial interaction SSM and channel evolution SSM. The spatial branch further consists of spatial mamba and frequency mamba. The main contributions of the paper primarily lie in the adaptation of mamba in the frequency domain for image deraining and the scanning strategy for the spatial-dimensional spectra.

strengths:
1.	The paper applies mamba in the frequency domain to enhance frequency interactions. To this end, it devises a zigzag scanning strategy.
2.	The proposed methods are evaluated on synthetic and real-world deraining datasets and achieve promising performance.

weaknesses:
1.	The paper lacks the theory analyses to support the importance of frequency interactions for image restoration.
2.	The channel fft has already been proposed in REVITALIZING CHANNEL-DIMENSION FOURIER TRANSFORM FOR IMAGE ENHANCEMENT. Why does the channel mamba outperform the 1x1 channel version? The reviewer thinks a 1x1 convolution can also model interactions between channel spectra. Moreover, the idea of interactions between frequencies is already implemented in Fourmer via convolutions.
3.	The authors replace mamba with 1x1 conv for the first ablation studies. However, the flops or parameter comparisons are not provided. Is it possible that the improvement of mamba versions is derived from the additional computation overhead compared to the 1x1 conv version?
4.	Regarding the experimental results, the paper only provides results on four datasets in Tab.1, lacking the test100 dataset. Moreover, the competitors in Fig.6 are a little outdated. From my side, the experiments are insufficient. Specifically, compared to other mamba-based methods, the evaluation data/tasks are fewer. Compared to other deraining methods like drsformer, the authors adopt only a rain13 dataset plus spad for evaluation, which is also insufficient.

questions:
Please refer to Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732842534314

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Eager to Hear From You

comment:
Dear reviewers:

Thanks a lot for your previous constructive comments. We would like to know if our revisions address your concerns. If you still have any concerns, we are eager to hear from you, please let us know and we are more than happy to discuss them with you.

Best regards,

Authors of #8897
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel framework named FourierMamba for image deraining, which integrates a Fourier representation with state space models (SSMs). The authors claim that their method can effectively capture and utilize frequency correlations in the Fourier space, leading to improved image deraining performance. The paper introduces a zigzag coding strategy to scan frequencies in the spatial dimension Fourier space and directly apply SSMs in the channel dimension Fourier space. The authors report state-of-the-art performance on several image deraining datasets.  

Strengths:
- The paper presents a novel idea of integrating Fourier space with SSMs for image deraining.  
- The proposed zigzag coding strategy is well-motivated and technically sound.  
- The experimental results demonstrate the effectiveness of the proposed method.  

Weaknesses:
- Limited novelty of the channel FFT concept.  
- The experimental validation could be more comprehensive, including additional datasets and comparisons with state-of-the-art methods.  
- The advantages of the proposed Mamba structure over other architectures for processing Fourier frequencies are not clearly demonstrated.  
- The paper could benefit from visualizing intermediate features to support the claims of improved frequency correlation modeling.  
- Limited Novelty with respect to previous work (in particular over FreqMamba).


This paper presents a novel and interesting idea, but requires further revisions before acceptance. While the rebuttal addressed some concerns, significant changes to the paper's body are needed to clearly establish its key contributions and differentiate it from existing work.

Specifically, the authors should:

- Broaden the scope: Explore the applicability of the proposed innovations to more general tasks beyond image deraining. This will increase the potential impact and relevance of the work.
- Clarify the paper's unique contributions: More explicitly highlight how this work advances the field beyond existing approaches to image deraining. Right now the paper seems like an incremental change over FreqMamba, which is interesting, but it's limited in terms of impact.

By addressing these points, the authors can strengthen the paper and better communicate its value to the research community.