FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining

## Abstract

Abstract Image deraining aims to remove rain streaks from rainy images and restore clear backgrounds.
Currently, some research that employs the Fourier transform has proved to be effective for image deraining, due to it acting as an effective frequency prior for capturing rain streaks.
However, despite there exists dependency of low frequency and high frequency in images, these Fourier-based methods rarely exploit the correlation of different frequencies for conjuncting their learning procedures, limiting the full utilization of frequency information for image deraining.
Alternatively, the recently emerged Mamba technique depicts its effectiveness and efficiency for modeling correlation in various domains (e.g., spatial, temporal), and we argue that introducing Mamba into its unexplored Fourier spaces to correlate different frequencies would help improve image deraining.
This motivates us to propose a new framework termed FourierMamba, which performs image deraining with Mamba in the Fourier space.
Owning to the unique arrangement of frequency orders in Fourier space, the core of FourierMamba lies in the scanning encoding of different frequencies, where the low-high frequency order formats exhibit differently in the spatial dimension (unarranged in axis) and channel dimension (arranged in axis).
Therefore, we design FourierMamba that correlates Fourier space information in the spatial and channel dimensions with distinct designs.
Specifically, in the spatial dimension Fourier space, we introduce the zigzag coding to scan the frequencies to rearrange the orders from low to high frequencies, thereby orderly correlating the connections between frequencies; in the channel dimension Fourier space with arranged orders of frequencies in axis, we can directly use Mamba to perform frequency correlation and improve the channel information representation.
Extensive experiments reveal that our method outperforms state-of-the-art methods both qualitatively and quantitatively.

## 1 Introduction

Images taken in rainy conditions exhibit significant degradation in detail and contrast due to rain in the air, leading to unpleasant visual results and the loss of frequency information. This issue can severely impact the performance of outdoor computer vision systems, such as autonomous driving and video surveillance []. To mitigate the effects of rain, many image deraining methods [, ] have emerged in recent years, aiming to remove rain streaks and restore clear backgrounds in images.

The advent of deep learning has spurred this field forward, with several learning-based deraining methods achieving remarkable success [, , ]. Among them, some studies utilize the Fourier transform for deraining in the frequency domain [, ], proving effective. The key insights inspiring the use of the Fourier transform for image deraining are twofold: 1) The Fourier transform can separate image degradation and content components to some extent, serving as a prior for image deraining, as shown in Figure 1; 2) The Fourier domain possesses global properties, where each pixel in Fourier space is involved with all spatial pixels. Thus, it makes sense to explore the task of rain removal using the Fourier transform. However, despite the existence of low frequency and high frequency dependencies in images, previous Fourier-based methods rarely utilize the correlation of different frequencies to combine their learning process. As shown in Figure 1, the commonly used $1\times 1$ convolutions cannot correlate different frequencies, limiting the full utilization of frequency information in the image. Therefore, we seek to exploit the beneficial properties of the Fourier transform while exploring correlating different frequencies.

Figure: Figure 1: Observation and comparison of different frequency modeling methods. (a) Observation of the amplitude spectrum exchange. The degradation is mainly in amplitude components, so the Fourier transform helps to disentangle the image content and rain. (b) The commonly used $1\times 1$ convolution cannot model the relationship between different frequencies. (c) Previous scanning in Fourier space will fail to establish the ordered dependence between frequencies. (d) Our proposed method achieves ordered frequency dependence from low to high (or vice versa), thus fully utilizing frequency information.
Refer to caption: x1.png

Recently, an improved structured state-space sequence model (S4) with a selective scanning mechanism, Mamba, gives us hope. The selective methodology of Mamba can explicitly build correlation the correlation among image patches or pixels. Recent studies have witnessed the effectiveness and efficiency of Mamba in various domains such as spatial and temporal. Therefore, we believe that introducing Mamba into its unexplored Fourier space to correlate different frequencies will be advantageous for improving image deraining.

In this paper, we propose a novel framework named FourierMamba, which performs image deraining using mamba in the Fourier domain. Following the "spatial interaction + channel evolution" rule that has also been validated on Mamba [, ], we design the Mamba framework in the Fourier domain from both spatial and channel dimensions. Considering the unique arrangement of frequency orders in the Fourier domain, the core of FourierMamba lies in the scanning encoding of different frequencies, where the low-high frequency order formats unarranged in the spatial axis and arranged in the channel axis. Therefore, our proposed FourierMamba correlates Fourier space information in spatial and channel dimensions with distinct designs.

Specifically, in the spatial dimension of the Fourier space, low-high frequencies follow a concentric circular arrangement with lower frequencies near the center and higher frequencies around the periphery. If previous scanning method [] is used directly, the orderliness between frequencies will be destroyed, as shown in Figure 1. We note that the zigzag coding in the JPEG compression field can place lower-frequency coefficients at the forefront of the array, while higher-frequency coefficients are positioned at the end. Hence, we introduce the zigzag coding to scan the frequency in the spatial dimension, rearranging the order from low to high frequency. arrangement
Due to the symmetry of the frequency orders in the Fourier space, we do not directly employ the zigzag coding in its originally used space; instead, we implement it in a circling-like manner that matches the symmetric frequency orders in Fourier space.
In this way, this method orderly correlates the connections between frequencies, as shown in Figure 1.
In the channel dimension of the Fourier space, the frequency order is arranged along the axis, following the order of low in the middle to high on both sides. Therefore, we can directly use Mamba for frequency correlation, thus improving channel information representation and enhancing global properties on the channels.

In summary, our contributions are as follows: (1) We propose a novel framework FourierMamba that combines Fourier priors and State Space Model for correlating different frequencies in the Fourier space to enhance image deraining. (2) To rearrange the order from low to high frequency in the spatial dimension Fourier space, we propose a scanning method based on zigzag coding to orderly correlate different frequencies. (3) Based on the channel-dimension Fourier transform, we utilize Mamba to scan on the channels and correlate different frequencies to improve channel information representation. Extensive experiments demonstrate that the proposed FourierMamba surpasses state-of-the-art methods both qualitatively and quantitatively.

## 2 Related Works

Image deraining. Traditional image deraining methods focus on separating rain components by utilizing meticulously designed priors, such as Gaussian Mixture Models [], Sparse Representation Learning [, ], and Directional Gradient Priors []. Although these methods are insightful, they often struggle to cope with complex precipitation patterns and the diverse real-world scenarios. The advent of deep learning has heralded a new era for image deraining. [] introduces pioneering deep residual networks for image deraining. The initiation of CNNs marked a significant advancement, facilitating more nuanced and adaptive processing of rain streaks across a vast array of images [, ]. With the evolution of transformers, the development of architectures that incorporate attention mechanisms [, , ] has further refined the capacity to recognize and eliminate rain components, addressing previous shortcomings in model generalization and detail preservation. In this work, we propose a novel baseline with a block based on Fourier and Mamba to enhance deraining performance.

Fourier transform.
Recently, the Fourier Transform has demonstrated its effectiveness in global modeling [, ]. This transformation converts signals into a domain characterized by global statistical properties, facilitating advancements across various fields [, , , , , ]. Due to its efficacy in global modeling, the Fourier Transform has been introduced into low-level vision tasks [, ]. As an early attempt, [] proposes a Fourier Transform-based loss to optimize global high-frequency information for efficient image super-resolution. DeepRFT [] is proposed for image deblurring, employing a global receptive field to capture both low and high-frequency characteristics of various blurs, a concept similarly applied in image inpainting []. FECNet [] demonstrates that the amplitude of Fourier features decouples global luminance components, thereby proving effective for image enhancement. [] observes a similar phenomenon in image dehazing, where the amplitude reflects global haze-related information. In contrast, we introduce a progressive scanning strategy in the Fourier domain, enhancing the global modeling capability while addressing the directional sensitivity issues of visual Mamba.

State Space Models.
State Space Models (SSMs) have received a lot of attention recently due to their global modeling capabilities as well as linear complexity, with [] initially introducing the base design of SSM models, and [] further enhancing their performance through gating units.More recently, the performance of Mamba [], proposed based on selective scan mechanism and efficient hardware design, has seen significant enhancement. It stands as an efficient alternative to Transformers, finding applications in various domains including image classification [][], object detection[], and remote sensing[].In the field of image restoration, [] [] initially introduced a general restoration framework based on the Mamba module but did not fully exploit the frequency domain information of images. [] introduced a wavelet transform branch, yet the scanning in the wavelet domain fails to fully extract global frequency domain information. This paper proposes a novel Mamba restoration network based on Fourier transform, aiming to comprehensively exploit the frequency domain information of images.

Figure: Figure 2: Our proposed Fourier space scanning method in the spatial dimension (top) and channel dimension (bottom). For simplicity, only one direction is shown for each scanning method, and in fact each method also performs a scan opposite to that shown.
Refer to caption: x2.png

### 2.1 Preliminary

Fourier transform. Fourier transform is a widely used technique for analyzing the frequency content of an image. For images with multiple color channels, the Fourier transform is applied to each channel separately. Given an image $X\in\mathbb{R}^{H\times W\times C}$, the Fourier transform $\mathcal{F}$ converts it to Fourier space as the complex component F(x), which is expressed as:

$$ $\mathcal{F}\left(x\right)\left(u,v\right)=\frac{1}{\sqrt{HW}}\sum_{h=0}^{H-1} \sum_{w=0}^{W-1}x\left(h,w\right)e^{-j2\pi\left(\frac{h}{H}u+\frac{w}{W}v \right)},$ (1) $$

where $u$ and $v$ indicate the coordinates of the Fourier space. $\mathcal{F}^{-1}\left(x\right)$ defines the inverse Fourier transform accordingly. Both the Fourier transform and its inverse procedure can be efficiently implemented using FFT/IFFT algorithms []. The amplitude component $\mathcal{A}\left(x\right)\left(u,v\right)$ and phase component $\mathcal{P}\left(x\right)\left(u,v\right)$ are expressed as:

$$ $\begin{split}&\mathcal{A}\left(x\right)\left(u,v\right)=\sqrt{R^{2}\left(x \right)\left(u,v\right)+I^{2}\left(x\right)\left(u,v\right)},\\ &\mathcal{P}\left(x\right)\left(u,v\right)=\arctan\left[\frac{I\left(x\right) \left(u,v\right)}{R\left(x\right)\left(u,v\right)}\right],\end{split}$ (2) $$

where $R(x)\left(u,v\right)$ and $I(x)\left(u,v\right)$ represent the real and imaginary parts respectively. The Fourier transform and its inverse procedure are applied independently to each channel of the feature maps.

Channel-dimension Fourier transform.
We introduce the channel-dimension Fourier transform by individually applying Fourier transform along the channel dimension for each spatial position. For each position ($h\in\mathbb{R}^{H-1}$, $w\in\mathbb{R}^{W-1}$) within $X\in\mathbb{R}^{H\times W\times C}$, denoted as $x(h,w,0:C-1)$ and abbreviated as $y(0:C-1)$, Fourier transform $\mathcal{F}\left(\cdot\right)$ converts it to Fourier space as the complex component $\mathcal{F}\left(y\right)$, which is expressed as:

$$ $\mathcal{F}(y(0:C-1))(z)=\frac{1}{C}\sum_{c=0}^{C-1}y(c)e^{-j2\pi\frac{c}{C}z},$ (3) $$

.

Similarly, the amplitude component $\mathcal{A}(y(0:C-1))(z)$ and phase component $\mathcal{P}(y(0:C-1))(z)$ of $\mathcal{F}(y(0:C-1))(z)$ are expressed as:

$$ $\begin{split}&\mathcal{A}(y(0:C-1))(z)=\sqrt{R^{2}\left(y(0:C-1)\right)\left(z \right)+I^{2}\left(y(0:C-1)\right)\left(z\right)},\\ &\mathcal{P}(y(0:C-1))(z)=\arctan\left[\frac{I\left(y(0:C-1)\right)\left(z \right)}{R\left(y(0:C-1)\right)\left(z\right)}\right].\end{split}$ (4) $$

These operations can also be applied for the global vector derived by the pooling operation. In this way, $\mathcal{A}(z)$ and $\mathcal{P}(z)$ signify the magnitude and directional changes in the magnitude of various channel frequencies, respectively. Both of these metrics encapsulate global statistics related to channel information.

State Space Models.
State Space Models (SSMs) serve as the cornerstone for transforming one-dimensional inputs into outputs through latent states, utilizing a framework of linear ordinary differential equations. Mathematically, SSMs can be formulated as follows, representing linear ordinary differential equations (ODEs):

$$ $\displaystyle h^{\prime}(t)$ $\displaystyle=\boldsymbol{A}h(t)+\boldsymbol{B}x(t),$ (5) $\displaystyle y(t)$ $\displaystyle=\boldsymbol{C}h(t)+\boldsymbol{D}x(t),$ $$

where, $h(t)\in\mathbb{R}^{N}$ denotes the hidden state vector, where N represents the size of the state. The parameters$\boldsymbol{A}\in\mathbb{R}^{N\times N}$, $\boldsymbol{B}\in\mathbb{R}^{N}$, and $\boldsymbol{C}\in\mathbb{R}^{N}$ are associated with the state size N, while
$\boldsymbol{D}\in\mathbb{R}^{1}$ represents the skip connection.

Discrete versions of these models, such as Mamba[], include a discretization step via the zero-order hold (ZOH) method. This enables the models to adaptively scan and adjust to the input data using a selective scanning mechanism. This mechanism provides a global receptive field with linear complexity, which is advantageous for image restoration tasks.

Figure: Figure 3: The overall architecture of the FourierMamba. Our FourierMamba consists of multiscale hierarchical design Fourier Residual State-Space Blocks(FRSSB). The core modules of FRSSB are Fourier Spatial Interaction SSM(FSI-SSM) and Fourier Channel Evolution SSM(FCS-SSM).
Refer to caption: x3.png

### 2.2 Scanning in Fourier Space

Despite the unique characteristics of the selective scan mechanism (S6), it processes input data causally. Given the non-causal nature of visual data, directly applying this strategy to patches and flat images fails to estimate relations with unscanned patches, leading to a "directional sensitivity" issue constrained by the acceptance domain. Numerous methods have attempted to tackle this problem in the spatial domain [, ]. However, for image restoration, the Fourier space and its associated priors are crucial. Hence, we explore addressing the "directional sensitivity" issue within this domain. Specifically, we customize Fourier scanning strategies from both spatial and channel dimensions.

For the spatial dimension, each pixel point in the Fourier space contains global information, with its frequencies distributed in concentric circles. Scanning methods based on spatial arrangements [] disrupt the high-low frequency relationships in the frequency domain, thus hindering the modeling of image degradation information.

Therefore, we aim to devise a scanning method in the Fourier space to progressively model the frequency characteristics of images. An intuitive approach is to calculate the Euclidean distance from each point in the spectrum to the center point. On the shifted Fourier spectrum, the smaller the distance to the center point, the lower the frequency. The flaw of this intuitive approach is that for images of different sizes, it requires recalculating the Euclidean distance from each point to the center point. The additional computational overhead introduced by this flaw makes this approach impractical in the field of image restoration.

In JPEG compression, zigzag coding is commonly used among the Discrete Cosine Transform (DCT) coefficients of JPEG, where it prioritizes the energy-concentrated low-frequency coefficients at the beginning of the array, and places the less significant high-frequency coefficients towards the end, thereby facilitating more effective compression. Inspired by compression algorithms, we introduce a method that adopts the zigzag coding approach to scan the magnitude and phase spectra.

Additionally, due to the symmetry of the two-dimensional Fourier transform, scanning the entire spectrum would disrupt the symmetry in the Fourier space, potentially leading to the collapse of network optimization. Therefore, we scan half of the spectrum and then deduce the other half based on the central symmetry of the amplitude and the anti-central symmetry of the phase.

Specifically, we design two scanning strategies, as illustrated in
the Figure 2. The first scanning method employs a dual zigzag pattern named bilateral zigzag, starting from the vertex of the highest frequency on one side of the spectrum, progressing in a zigzag pattern toward the center’s low frequencies; similarly, it then zigzags to the opposite side’s highest frequency. This scanning approach not only models the association between high and low frequencies but also takes into account the periodicity of the Fourier spectrum. Due to the periodic nature of the Fourier transform, the high-frequency ends on either side should, in fact, be contiguous. The second method builds upon the low-to-high frequency sequence established by zigzag scanning and conducts a scan from low to high frequencies, which is named progressive zigzag. This method is motivated by the tendency of neural networks to initially learn low-frequency information when extracting image characteristics. Following the previous method [, ], we reverse the above two scanning methods as additional scanning directions.

For the channel dimension Fourier space, since it is a one-dimensional sequence arranged in order of low to high frequencies, we directly scan it one-dimensionally. Similarly, due to the symmetry of the Fourier transform, we scan only half and derive the other half. Through Fourier space scanning in both spatial and channel dimensions, we can correlate the connections between frequencies in an orderly manner, thereby making full use of frequency information to improve rain removal.

Figure: Figure 4: The error map between the GT and the restored images using various scanning methods in Fourier space. The two scanning methods we propose can achieve smaller errors than using classical scanning method []. And the combination of the two scanning methods is better than either one.
Refer to caption: x4.png

### 2.3 FourierMamba

#### 2.3.1 Overall Framework

In Figure 3, we illustrate our proposed FourierMamba. Given a rainy image $I\in\mathbb{R}^{H\times W\times 3}$, FourierMamba first uses $3\times 3$ convolution layers to generate shallow features with dimensions of $H\times W\times C$, where $H$ and $W$ represent height and width, and $C$ denotes the number of channels. Subsequently, we employ a multi-scale U-Net architecture to obtain deep features. This stage consists of a stack of Fourier Residual State-Space Groups, each containing several Fourier Residual State-Space Blocks (FRSSB). The FRSSB incorporates our two core designs: the Fourier Spatial Interaction SSM block and the Fourier Channel Evolution SSM block. They correlate Fourier domain information from spatial and channel dimensions, respectively, to fully leverage frequency information.

#### 2.3.2 Fourier Spatial Interaction SSM

The structure of the Fourier Spatial Interaction State Space Model (FSI-SSM) is shown in Figure 3. We first apply LayerNorm to transform the input features $F_{in}$ into $F_{l}$. To facilitate the interaction between spatial and frequency information, FSI-SSM employs both a Fourier branch and a spatial branch to collaboratively process $F_{in}$.

Fourier Branch: $F_{l}$ is transformed into the Fourier spectrum through the Fast Fourier Transform, subsequently decomposed into the amplitude spectrum $\mathcal{A}(F_{l})$ and phase spectrum $\mathcal{P}(F_{l})$.
The amplitude spectrum and phase spectrum are then processed separately using the progressive frequency scanning method illustrated in Figure 2 to obtain $\mathcal{A}^{\prime}(F_{l})$ and $\mathcal{P}^{\prime}(F_{l})$.

$$ $\begin{split}&\mathcal{A}^{\prime}(F_{l})={\rm FourScan}(\mathcal{A}(F_{l})), \\ &\mathcal{P}^{\prime}(F_{l})={\rm FourScan}(\mathcal{P}(F_{l})),\end{split}$ (6) $$

where ${\rm FourScan}$ is the sequence transformation using the Fourier space scan described in Sec. 2.2. Following a series of works [, , ], the sequence transformation employs the following operation sequence: $DWConv\to SiLU\to SSM\to LayerNorm$.
We then perform an inverse Fourier transform on the processed spectrum and multiply it with the output of $\rm SiLU$.

$$ $F_{f}=(\mathcal{F}^{-1}(\mathcal{A}^{\prime}(F_{l}),\mathcal{P}^{\prime}(F_{l} )))\odot{\rm SiLU}(F_{l}),$ (7) $$

where $F_{f}$ is the output of the fourier branch, and $\odot$ is the Hadamard product.

Spatial Branch
In the spatial domain, we feed the input features $F_{l}$ into two parallel sub-branches. One sub-branch activates the features using the $\rm SiLU$ function. The other sub-branch performs spatial Mamba on features after $1\times 1$ convolution. Specifically, spatial Mamba adopts the same operation sequence as the above frequency branch but the scanning in SSM uses the two-dimensional selective scanning module shown in Figure 3, which follows previous work [, ]. Finally, the outputs of the two sub-branches are multiplied element-wise to obtain the output $F_{s}$.

$$ $F_{s}={\rm SpaScan}({\rm Conv}(F_{l}))\odot{\rm SiLU}(F_{l}),$ (8) $$

where ${\rm Conv}$ is $1\times 1$ convolution and SpaScan is the spatial Mamba mentioned above.
Subsequently, we employ a residual connection to add the spatial output to $F_{in}$. The spatial branch captures global features in the spatial domain which complement the frequency correlations captured by the Fourier branch in the frequency domain, thereby benefiting the performance of image deraining. Hence, we concatenate the outputs of the spatial and frequency branches and use a $1\times 1$ convolution for the fusion of spatial and frequency information.

#### 2.3.3 Fourier Channel Evolution SSM

Previous work [] claims that selecting key channels can avoid channel redundancy in SSM. Since each channel contains the information of all channels after the channel-dimension Fourier transform, we perform channel interaction in the Fourier domain to efficiently correlate different frequencies of channels. As depicted in Figure 3, our proposed Fourier Channel Evolution SSM (FCS-SSM) consists of three sequential parts: applying the Fourier transform along the channel dimension to obtain channel-wise Fourier domain features, scanning its amplitude and phase, then restoring to the spatial domain. Specifically, assuming the input features are denoted as $F_{r}\in\mathbb{R}^{H_{r}\times W_{r}\times C_{r}}$, we first perform global average pooling on it.

$$ $F_{g}=\frac{1}{H_{r}W_{r}}\sum_{h=0}^{H_{r}-1}\sum_{w=0}^{W_{r}-1}F_{g}(h,w),$ (9) $$

where $F_{g}\in\mathbb{R}^{1\times 1\times C_{r}}$ corresponds to the center point of the amplitude spectrum of $F_{r}$ (see supplementary material), which effectively encapsulates the global information of the feature. Then, we use the channel-dimensional Fourier transform shown in Equ. 3 on $F_{g}$ to obtain $\mathcal{F}(F_{g})(z)$. Based on this, we use Equ. 4 for $\mathcal{F}(F_{g})(z)$ to obtain its amplitude component $\mathcal{A}(F_{g})(z)$ and phase component $\mathcal{P}(F_{g})(z)$. Since the amplitude spectrum and phase spectrum have obvious information meaning, we choose to perform Mamba scanning on these two components.

$$ $\begin{split}&\mathcal{A}(F_{g})(z)^{\prime}={\rm ChaScan}(\mathcal{A}(F_{g})( z)),\\ &\mathcal{P}(F_{g})(z)^{\prime}={\rm ChaScan}(\mathcal{P}(F_{g})(z)),\end{split}$ (10) $$

where ${\rm ChaScan}$ is a one-dimensional sequence transformation that uses the following sequence of operations: $DWConv\to SiLU\to SSM\to LayerNorm$. Its scanning method is shown in Figure 2. After the Mamba correlates different frequencies in the channel dimension, we perform an inverse Fourier transform on it and multiply the result with the channel features after $\rm SiLU$ activation.

$$ $F_{a}=(\mathcal{F}^{-1}(\mathcal{A}(F_{g})(z)^{\prime},\mathcal{P}(F_{g})(z)^{ \prime}))\odot{\rm SiLU}(F_{g}),$ (11) $$

where $F_{a}\in\mathbb{R}^{1\times 1\times C_{r}}$ is the channel feature after correlating different frequencies. Finally, we multiply it with the spatial feature in a form of attention to get the output $F_{c}\in\mathbb{R}^{H_{r}\times W_{r}\times C_{r}}$.

$$ $F_{c}=F_{a}\odot F_{r}.$ (12) $$

#### 2.3.4 Optimization

We impose constraints in both the spatial and frequency domains. In the spatial domain, we utilize the L1 loss between the final output $Y_{out}$ and the ground truth $Y_{gt}$. In the frequency domain, we apply the L1 loss based on the Fourier transform. The overall loss function is formulated as follows:

$$ ${\mathcal{L}_{total}}=\left\|Y_{out}-Y_{gt}\right\|_{1}+\lambda\left\|\mathcal {F}(Y_{out})-\mathcal{F}(Y_{gt})\right\|_{1},$ (13) $$

where $\lambda$ is the balancing weight. In particular,$\lambda$ is set to 0.02 empirically.

## 3 Experiment

### 3.1 Experimental Settings

Datasets.
For training, we employ the widely used Rain13k dataset []. It contains 13,712 image pairs in the training set, and we evaluate the results on Rain100H [], Rain100L [], Test2800 [], and Test1200 [].

Figure: Figure 5: Visual quality comparison on Rain100H []. Zoom in for better visualization.
Refer to caption: x5.png

Implementation details.
Our model is implemented within the PyTorch framework and executed on an NVIDIA A100 GPU. The number of blocks per layer has an impact on both the model’s parameter count and its deraining performance. After balancing the weights, we configure the blocks per layer as [2, 3, 3, 4, 3, 3, 2], which allows us to achieve commendable performance with a reasonable number of parameters. We adopt the progressive training strategy. Specifically, we set the total number of iterations to 80,000 and image sizes to [160, 256, 320, 384], with the corresponding batch sizes of [8, 4, 2, 1]. We utilize the Adam optimizer with default parameters. The initial learning rate is established at $3\times e^{-4}$, followed by a gradual decay to $1\times e^{-6}$ using a cosine annealing schedule.

### 3.2 Comparison with State-of-the-art Methods

We compare our method with state-of-the-art deraining methods: DerainNet [], UMRL [], RESCAN [], PreNet [], MSPFN [] , SPAIR [] , MPRNet [] , Restormer[], Fourmer [], IR-SDE [], MambaIR [] VMambaIR[] and FreqMamba[].

**Table 1: Quantitative comparison (PSNR/SSIM) for Image Deraining on five benchmark datasets. The highest and second-highest performances are marked in bold and underlined. ’-’ indicates the result is not available.**
| Method | Venue | Rain100H [] | Rain100L [] | Test2800 [] | Test1200 [] | Param(M) | GFlops |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$ |  |  |  |  |
| DerainNet [] | TIP’17 | 14.92 | 0.592 | 27.03 | 0.884 | 24.31 | 0.861 | 23.38 | 0.835 | 0.058 | 1.453 |
| UMRL [] | CVPR’19 | 26.01 | 0.832 | 29.18 | 0.923 | 29.97 | 0.905 | 30.55 | 0.910 | 0.98 | - |
| RESCAN [] | ECCV’18 | 26.36 | 0.786 | 29.80 | 0.881 | 31.29 | 0.904 | 30.51 | 0.882 | 1.04 | 20.361 |
| PreNet [] | CVPR’19 | 26.77 | 0.858 | 32.44 | 0.950 | 31.75 | 0.916 | 31.36 | 0.911 | 0.17 | 73.021 |
| MSPFN [] | CVPR’20 | 28.66 | 0.860 | 32.40 | 0.933 | 32.82 | 0.930 | 32.39 | 0.916 | 13.22 | 604.70 |
| SPAIR [] | ICCV’21 | 30.95 | 0.892 | 36.93 | 0.969 | 33.34 | 0.936 | 33.04 | 0.922 | - | - |
| MPRNet [] | CVPR’21 | 30.41 | 0.890 | 36.40 | 0.965 | 33.64 | 0.938 | 32.91 | 0.916 | 3.64 | 141.28 |
| Restormer [] | CVPR’22 | 31.46 | 0.904 | 38.99 | 0.978 | 34.18 | 0.944 | 33.19 | 0.926 | 24.53 | 174.7 |
| Fourmer [] | ICML’23 | 30.76 | 0.896 | 37.47 | 0.970 | - | - | 33.05 | 0.921 | 0.4 | 16.753 |
| IR-SDE [] | ICML’23 | 31.65 | 0.904 | 38.30 | 0.980 | 30.42 | 0.891 | - | - | 135.3 | 119.1 |
| MambaIR [] | arxiv’24 | 30.62 | 0.893 | 38.78 | 0.977 | 33.58 | 0.927 | 32.56 | 0.923 | 31.51 | 80.64 |
| VMambaIR [] | arxiv’24 | 31.66 | 0.909 | 39.09 | 0.979 | 34.01 | 0.944 | 33.33 | 0.926 | - | - |
| FreqMamba [] | arxiv’24 | 31.74 | 0.912 | 39.18 | 0.981 | 34.25 | 0.951 | 33.36 | 0.931 | 14.52 | 36.49 |
| FourierMamba(Ours) | - | 31.79 | 0.913 | 39.73 | 0.986 | 34.23 | 0.949 | 34.76 | 0.938 | 17.62 | 22.56 |

Quantitative Comparison. Following previous work, we compute the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) scores using the Y channel in the YCbCr color space. Table 1 reports the performance evaluation on four datasets. It can be seen that our method achieves the highest PSNR and SSIM across all datasets, which underscores the efficacy of FourierMamba in enhancing deraining performance.

Qualitative Comparison.
To demonstrate the enhanced fidelity and detail levels exhibited by the images generated by our proposed FourierMamba, we compare the visual quality of challenging degraded images from the Rain100H dataset in Figure 5. Our method achieves excellent results when faced with complex or extremely severe rain streaks. Compared to previous methods, our FourierMamba achieves impeccable performance in both global and local restoration. For instance, by zooming into the red boxed area in Figure 5, our method removes more rain streak residues while better restoring texture details. We provide additional visual results in the Appendix.

### 3.3 Ablation Studies

We perform ablations on the key designs and scanning methods of the framework on the Rain100L.

Fourier Spatial Interaction SSM (FSI-SSM) and Fourier Channel Evolution SSM (FCI-SSM). We replace the mamba scan in FSI-SSM and FCE-SSM with $1\times 1$ convolution, called w/o FSI-SSM and w/o FCE-SSM respectively. It can be seen from Table 2 that since $1\times 1$ convolution cannot model the dependence of different frequencies, its performance is worse than the mamba scan in the Fourier domain in both the spatial dimension and the channel dimension.

Fourier prior. We do not use Fourier transform in the spatial dimension and channel dimension respectively, but directly perform mamba scanning, which are called without spatial dimension Fourier (w/o SDF) and without channel dimension Fourier (w/o CDF) respectively. It can be seen from Table 2 that after losing the Fourier prior in the spatial dimension and channel dimension, the performance drops significantly. This proves the effectiveness of Fourier prior for removing rain from images.

Scanning method in Fourier space. We compare several scanning methods of the spatial dimension Fourier space, with the same amount of calculation. Table 3 illustrates that the performance of the two scanning methods we proposed is better than the classic two-dimensional scanning method []. And thanks to complementarity, the combination of the two methods can also further improve performance. The visual comparison in Figure 4 supports this.

**Table 2: Ablation studies of key designs in the proposed method.**
|  | w/o FSI-SSM | w/o FCE-SSM | w/o SDF | w/o CDF | Ours |
| --- | --- | --- | --- | --- | --- |
| PSNR | 39.05 | 39.08 | 38.25 | 38.72 | 39.73 |
| SSIM | 0.9835 | 0.9836 | 0.9810 | 0.9827 | 0.9856 |

**Table 3: Ablation study of different scanning methods in Fourier space.**
|  | Classic[] | Bilateral | Progressive | Ours |
| --- | --- | --- | --- | --- |
| PSNR | 38.82 | 39.31 | 39.28 | 39.73 |
| SSIM | 0.9817 | 0.9844 | 0.9843 | 0.9856 |

## 4 Conclusion

In this paper, we propose a novel image deraining framework, FourierMamba, which utilizes mamba to correlate frequencies in the Fourier space, thus fully exploiting frequency information. Specifically, we design the mamba framework by integrating the unique arrangement of frequency orderings within the Fourier domain across spatial and channel dimensions. In the spatial dimension, we devise two zigzag-based methods to scan frequencies, systematically correlating them. In the channel dimension, due to the ordered arrangement of frequencies along the axis, we directly apply mamba for frequency correlation. This work introduces a new research strategy to address the underutilization of frequency information in image deraining that affects performance. Extensive experimental results on multiple benchmarks validate the effectiveness of the proposed method.

Figure: Figure 6: Correction of the second picture in Figure 5.
Refer to caption: x6.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to Weaknesses
soundness: 2
strengths: 1.	The paper applies mamba in the frequency domain to enhance frequency interactions. To this end, it devises a zigzag scanning strategy.
2.	The proposed methods are evaluated on synthetic and real-world deraining datasets and achieve promising performance.
summary: This paper presents a Mamba-based image deraining algorithm. The proposed FourierMamba mainly involves two components: spatial interaction SSM and channel evolution SSM. The spatial branch further consists of spatial mamba and frequency mamba. The main contributions of the paper primarily lie in the adaptation of mamba in the frequency domain for image deraining and the scanning strategy for the spatial-dimensional spectra.
weaknesses: 1.	The paper lacks the theory analyses to support the importance of frequency interactions for image restoration.
2.	The channel fft has already been proposed in REVITALIZING CHANNEL-DIMENSION FOURIER TRANSFORM FOR IMAGE ENHANCEMENT. Why does the channel mamba outperform the 1x1 channel version? The reviewer thinks a 1x1 convolution can also model interactions between channel spectra. Moreover, the idea of interactions between frequencies is already implemented in Fourmer via convolutions.
3.	The authors replace mamba with 1x1 conv for the first ablation studies. However, the flops or parameter comparisons are not provided. Is it possible that the improvement of mamba versions is derived from the additional computation overhead compared to the 1x1 conv version?
4.	Regarding the experimental results, the paper only provides results on four datasets in Tab.1, lacking the test100 dataset. Moreover, the competitors in Fig.6 are a little outdated. From my side, the experiments are insufficient. Specifically, compared to other mamba-based methods, the evaluation data/tasks are fewer. Compared to other deraining methods like drsformer, the authors adopt only a rain13 dataset plus spad for evaluation, which is also insufficient.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the Weaknesses.
soundness: 3
strengths: 1. The proposed framework combines Fourier priors and state space models to correlate different frequencies in Fourier space to enhance the rain removal effect of images. This combination utilizes the advantages of the Fourier transform in global modeling and introduces the efficient feature modeling capability of the Mamba model.

2. To rearrange the order from low frequency to high frequency in the Fourier space of the spatial dimension, Fourier Mamba proposed a scanning method based on zigzag encoding, which systematically associates different frequencies. This method introduces zigzag encoding in Fourier space to rearrange the frequency order, thereby orderly associating the connections between frequencies.
summary: This paper proposes Fourier Mamba, an image rain removal framework. It improves the rain removal effect by applying the Mamba technique in Fourier space and utilizing the correlation between different frequencies. This method uses zigzag encoding to rearrange the frequency order in the spatial dimension and directly uses Mamba for frequency correlation in the channel dimension. Experiments have shown that Fourier Mamba has achieved more competitive results in image rain removal tasks.
weaknesses: 1. The experiment is insufficient and lacks comparison with methods such as DRSformer [1] and FADformer [2].
2. I am a bit concerned about the quantitative results in this paper. The difference between the results shown in Table 1 and FreqMamba is not significant. Please further explain the differences between the proposed method and FreqMamba, as they both use Fourier correlation strategies. It is recommended to add visualization results to supplement this.
3. It is worth noting that FFT is a complex valued computation. Suggest the author to add a comparison of the actual inference time of the model. FLOP comparison is not enough, because many works now have lower FLOP but slower running speed, and other issues such as memory consumption may slow down computation speed.
[1] Chen X,  Li H, Li M, et al. Learning a sparse transformer network for effective image deraining.
[2] Gao N,  Jiang X, Zhang X, et al. Efficient Frequency-Domain Image Deraining with Contrastive Regularization.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - In Section 4.1, the authors mentioned the "progressive training strategy". What does it mean? And why is it called "progressive"?

- There are several writing issues that I can find:
1) Line 75-76: "...build correlation the correlation among...";
2) The rightmost of Line 73: ".arrangement";
3) According to the formatting of other parts of this manuscript, the "Fourier transformation." in Section 3.1 should not take the whole line.
soundness: 3
strengths: - The application of Mamba to the Fourier domain is new and worthy to explore.

- The experimental results are promising
summary: This paper proposed a new network for image deraining by introducing the state space models, or more specifically, the recently proposed Mamba structure, in the Fourier spaces. By virtue of the Mamba, the frequency correlations can be better captured and utilized in the Fourier domain, such that the better deraining performance can be expected. Experiments on various deraining datasets have been conducted to demonstrate the effectiveness of the proposed method.
weaknesses: - The motivation should be strengthened. Currently, it seems the work is mostly about applying Mamba in the Fourier domain, though this task itself could be non-trivial. However, it is unclear why Mamba is more effective than other architectures for processing the Fourier frequencies. Visualizing the intermediate features might be useful.

- As reviewed by the authors, another work (Zhen et al., 2024) considers Mamba in the frequency domain but with wavelet transformation. It is better to discuss more about wavelet transformation and Fourier transformation in dealing with frequencies. Besides, as shown in Table 1, the performance of (Zhen et al., 2024) is close to the proposed method on several datasets, but the comparison with this method on SPA-Data is missing.

- The quantitative comparison is conducted with respect to PSNR and SSIM. However, as more and more researchers realized, these two metrics can be often inconsistent with human perceptions and thus some newly developed metrics that can better reflect human perceptions should be considered.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: The main points of confusion, as outlined in the Weakness section, are as follows:

**Implementation and Efficiency of Zigzag Scanning**:
The paper does not provide sufficient details on how the zigzag scanning is implemented, despite its potential complexity. Additionally, it is unclear how efficient the scanning strategy is in practice. A more thorough discussion of the implementation and its impact on inference speed would strengthen the work.

**Generalization to Other Tasks**:
Given that the motivation for zigzag scanning is not specific to rain removal, it would be valuable to evaluate the performance of the proposed method on other low-level vision tasks. Demonstrating its effectiveness beyond the deraining scenario would provide further evidence of its broader applicability.
soundness: 3
strengths: The proposed model involving zigzag scanning is novel and intuitive.

The paper provides a thorough and intuitive discussion of the advantages of zigzag scanning in the context of the deraining task.

The proposed model demonstrates state-of-the-art performance, balancing accuracy with reasonable computational cost and model size.
summary: The paper presents a novel approach to image deraining by integrating Fourier transformation with a state-space model. Unlike previous methods, the proposed approach applies the state-space model in both the spatial and frequency domains, introducing a new scanning strategy referred to as zigzag scanning. This strategy gradually processes both low- and high-frequency components, aiming to enhance the model's ability to capture rain-related distortions. The key contribution of the paper lies in the introduction of this zigzag scanning mechanism, which differentiates it from existing methods.
weaknesses: Although the motivation for zigzag scanning is well-discussed, the paper lacks details on its implementation. Considering the complexity involved, the implementation is non-trivial and deserves attention. The way zigzag scanning is executed, along with pixel organization strategies, can significantly impact inference speed. Including a comparison of running times would enhance the evaluation.

The visual comparison baselines could also be updated. It would be beneficial to include comparisons with more recent methods (post-2022) and provide additional zoomed-in patches or error maps to better highlight performance differences.

There are several confusing typographical issues, such as "FCS" being used as the abbreviation for "Fourier Channel Evolution" and "C-FFT" lacking a clear explanation (possibly referring to "channel FFT"). These should be clarified.

Furthermore, the paper overlooks several closely related works, including:
[1] Wavelet Approximation-Aware Residual Network for Single Image Deraining, 2023.
[2] A Hybrid Transformer-Mamba Network for Single Image Deraining, 2024.
[3] Image Deraining with Frequency-Enhanced State Space Model, 2024.

Additionally, applying the L1 norm directly in the frequency domain may be less accurate than alternative metrics, such as coherence-based distances. An ablation study on the impact of different frequency-domain loss functions would strengthen the analysis.

Finally, the zigzag scanning strategy appears intuitive for other low-level vision tasks, such as deblurring. It would be valuable to investigate the model's performance on additional tasks to assess its generalizability.

Meta-review: This paper proposes a novel framework named FourierMamba for image deraining, which integrates a Fourier representation with state space models (SSMs). The authors claim that their method can effectively capture and utilize frequency correlations in the Fourier space, leading to improved image deraining performance. The paper introduces a zigzag coding strategy to scan frequencies in the spatial dimension Fourier space and directly apply SSMs in the channel dimension Fourier space. The authors report state-of-the-art performance on several image deraining datasets.  

Strengths:
- The paper presents a novel idea of integrating Fourier space with SSMs for image deraining.  
- The proposed zigzag coding strategy is well-motivated and technically sound.  
- The experimental results demonstrate the effectiveness of the proposed method.  

Weaknesses:
- Limited novelty of the channel FFT concept.  
- The experimental validation could be more comprehensive, including additional datasets and comparisons with state-of-the-art methods.  
- The advantages of the proposed Mamba structure over other architectures for processing Fourier frequencies are not clearly demonstrated.  
- The paper could benefit from visualizing intermediate features to support the claims of improved frequency correlation modeling.  
- Limited Novelty with respect to previous work (in particular over FreqMamba).


This paper presents a novel and interesting idea, but requires further revisions before acceptance. While the rebuttal addressed some concerns, significant changes to the paper's body are needed to clearly establish its key contributions and differentiate it from existing work.

Specifically, the authors should:

- Broaden the scope: Explore the applicability of the proposed innovations to more general tasks beyond image deraining. This will increase the potential impact and relevance of the work.
- Clarify the paper's unique contributions: More explicitly highlight how this work advances the field beyond existing approaches to image deraining. Right now the paper seems like an incremental change over FreqMamba, which is interesting, but it's limited in terms of impact.

By addressing these points, the authors can strengthen the paper and better communicate its value to the research community.

Decision: Reject