FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 17:20:51

Abstract: Images corrupted by rain streaks often lose vital frequency information for perception, and image deraining aims to solve this issue which relies on global and local degradation modeling.
Recent studies have witnessed the effectiveness and efficiency of Mamba for perceiving global and local information based on its exploiting local correlation among patches, however, rarely attempts have been explored to extend it with frequency analysis for image deraining, limiting its ability to perceive global degradation that is relevant to frequency modeling (e.g. Fourier transform).
In this paper, we propose FreqMamba, an effective and efficient paradigm that leverages the complementary between Mamba and frequency analysis for image deraining. The core of our method lies in extending Mamba with frequency analysis from two perspectives: extending it with frequency-band for exploiting frequency correlation, and connecting it with Fourier transform for global degradation modeling.
Specifically, FreqMamba introduces complementary triple interaction structures including spatial Mamba, frequency band Mamba, and Fourier global modeling. Frequency band Mamba decomposes the image into sub-bands of different frequencies to allow 2D scanning from the frequency dimension. Furthermore, leveraging Mamba's unique data-dependent properties, we use rainy images at different scales to provide degradation priors to the network, thereby facilitating efficient training. Extensive experiments show that our method outperforms state-of-the-art methods both visually and quantitatively.

Main paper: 
 related work 2.1 image deraining: 
 Image deraining has witnessed substantial evolution, transitioning from early model-based strategies to advanced data-driven techniques. Initially, model-driven methods [20, 27] , separate rain streaks from the background using hand-crafted features and physical priors. These approaches, while insightful, often struggle with complex rain patterns and diverse real-world scenarios, leading to limitations in their practical applicability and performance.The advent of deep learning ushered in a new era for image deraining, with data-driven approaches demonstrating remarkable adeptness in extracting and learning features directly from data. The introduction of CNNs marked a significant advancement, enabling more nuanced and adaptive handling of rain streaks across a wide array of images [7, 47, 53] . Besides, the development of architectures incorporating attention mechanisms [38, 40, 44] has further refined the capacity to discern and eliminate rain components, addressing previous shortcomings in model generalization and detail preservation. In this work, we propose a novel baseline with a novel triple interaction block to improve the deraining performance jointly.The Fourier Transform is a fundamental technique in frequency domain analysis, enabling the conversion of signals to a domain where their global statistical properties can be more easily analyzed. This capability is extensively leveraged across various computer vision tasks. By adept modeling of global domain information, the FT facilitates advancements in various areas [17, 22, 23, 32, 45, 48, 57] . In terms of image restoration, FECNet [16] highlights the utility of Fourier feature amplitudes in isolating global lightness components, thus improving the aesthetic appeal and clarity of images. Similarly, FSDGN [50] reveals how the amplitude of Fourier features serves as a key indicator of global haze information in image dehazing tasks. Despite these advances, the inherent constraints of FT in signal processing hint at untapped potential, suggesting that the efficacy of these methodologies could be augmented further.Beyond Fourier transform, the Wavelet Transform (WT) is also a mathematical tool used in the analysis of signals and images, offering a complementary perspective to FT. Unlike FT, which excels in capturing frequency information, Wavelet transform provides a multi-resolution analysis that is particularly effective in detecting and representing localized variations in signals.State Space Models (SSMs) [11, 37] have garnered significant attention in recent years for their ability to effectively model longrange dependencies while exhibiting linear scalability with sequence length. The foundational work of S4 introduced by [11] laid the groundwork for deep state-space modeling, demonstrating promising linear scaling properties. A recent innovation, [31] enhances SSMs' capabilities by integrating gating units. Furthermore, Mamba [10] , a data-dependent SSM featuring a selective mechanism and efficient hardware design, has emerged as a standout performer, surpassing Transformers in natural language tasks while maintaining linear scalability with input length. The applicability of SSMs extends beyond NLP, with pioneering works leveraging Mamba for various vision tasks, including image classification [26] , biomedical image segmentation [9, 29] , and others [3, 34, 39] . 
method: 
 In this section, we first introduce the motivation of our proposed method, then introduce the preliminaries of frequency analysis and SSMs. Finally, we outline the overall framework of FreqMamba.Degraded images, particularly those afflicted by rain, suffer from both global degradation and local detail destruction. Frequencybased methods are utilized for mitigating global degradation, leveraging the significant correlation between raindrop effects and the Fourier amplitude spectrum (refer to Fig. 2 ). Nonetheless, due to the inherent gap between the frequency and spatial domain, the global nature of frequency domain operations makes it impossible to model local dependencies of the spatial domain. Mamba is a state space model distinguished by its selective scanning mechanism, which skillfully facilitates the explicit modeling of interactions among sequences with linear complexity. Applied to vision tasks, it can effectively establish correlations between areas in 2D images.Recognizing the complementary strengths of frequency-based methods and the Mamba model in addressing different aspects of image degradation, we introduce the FreqSSM Block. This novel structure features a three-branch design engineered to transition smoothly from the correction of global degradation to the refinement of local content. Further leveraging Mamba's distinctive data-dependent properties, we employ degraded images across various scales to derive attention maps and add them in the encoder stage, thereby enhancing the efficiency of training. Expanding upon these foundations, we propose FreqMamba, a specialized solution designed specifically to solve the single-image deraining task.Imaging. The Fourier Transform is a mathematical technique for transforming a signal from its original domain (often time or space) into a representation in the frequency domain and vice versa with the inverse Fourier Transform (iFT). Specifically for a single-channel image 𝑥 with size of 𝐻 × 𝑊 , the Discrete Fourier Transform is defined as: This process converts spatial features into a complex component, illustrating the image's frequency components. The frequency domain representation F (𝑥) can be decomposed into real R (𝑥) and imaginary I (𝑥) parts, leading to amplitude spectrum and phase spectrum, describing the image's frequency content:EQUATIONEQUATIONAnother tool, the Discrete Wavelet Transform (DWT), decomposes an image 𝐼 ∈ R 𝐻 ×𝑊 ×𝐶 into four sub-bands representing low-frequency approximations and high-frequency details:EQUATIONEach sub-band, corresponding to different directional details, reduces the original image's dimensions by half. In contrast, the Wavelet Packet Transform offers a more detailed frequency content analysis by further decomposing all sub-bands at each level.State Space Models (SSMs) are fundamental in translating one-dimensional inputs into outputs through latent states, utilizing a framework of linear ordinary differential equations. For a system with input 𝑥 (𝑡) and output 𝑦 (𝑡), the model dynamics are described by:EQUATIONwhere A, B, C, and D are the model parameters. The discrete versions of these models, such as Mamba, incorporate a discretization step through the zero-order hold (ZOH) method while allowing the models to scan and adjust to the input data adaptively through a selective scanning mechanism. This adaptability is particularly useful in complex applications like image restoration, where understanding contextual relationships between different image regions is crucial.We show our proposed method in Fig. 4 , which uses a multi-scale U-Net architecture with three-branch Frequency-SSM Blocks as the core components. At the same time, the input-dependent nature of Mamba is used to generate attention maps using degradation images of different scales, allowing the model to capture degradation distributions at different scales. Now we introduce the details of the different components of the network respectively.Frequency-SSM Block. The frequency-SSM block is designed to address complex challenges by leveraging the synergy of three distinct branches: the Fourier Modeling Branch, the Spatial Branch, and the Frequency Band Branch. Fourier Modeling Branch. For the Fourier Modeling Branch, the input feature 𝐹 𝑖𝑛 first undergoes a convolution layer to produce 𝐹 0 . 𝐹 0 is transformed into its Fourier spectrum via the Fast Fourier Transform (FFT), which is then decomposed into amplitude spectrum A (𝐹 0 ) and phase spectrum P (𝐹 0 ). The amplitude and phase spectrum are refined through convolution blocks in the frequency domain and finally returned to the spatial domain through iFFT:EQUATIONwhere 𝑊 1 (•) donates a 1 × 1 convolution operation and 𝐶𝑜𝑛𝑣𝐵𝑙𝑜𝑐𝑘 (•) signifies a series of convolution operations and activation functions. This branch captures and processes the frequency domain representation of features, mastering the global recovery of images. Spatial Branch. We first use Layernorm to process the input features 𝐹 𝑖𝑛 to get 𝐹 𝐿𝑁 . The features then pass through two parallel subbranches. The first sub-branch simply SiLU activates them. The other sub-branch performs spatial Mamba on features after 1 × 1 convolution. Spatial Mamba consists of sequence : DWConv → SiLU → Mamba-scan → LN, where Mamba-scan means vanilla 2D Mamba-scan shown in Fig. 3 (a) . The outputs of two sub-branches are then element-wise multiplied to yield the spatial output 𝐹 𝑠 : where ⊙ donates element-wise multiplication.EQUATIONFrequency Band Branch. 𝐹 𝐿𝑁 is subjected to a 2-layer wavelet packet transform to capture the information of different frequency bands with the size of 𝐻 4 , 𝑊 4 . They are arranged back to the original resolution from top left to bottom right, then processed by frequency Mamba, which is unique in its frequency scan as shown in Fig. 3 (b) . Since the strategy in Fig. 3 (a) cannot completely match WPT along the low to high frequency, we use a strategy similar to LocalMamba [18] . We divide the wavelet features into four blocks (red line in Fig. 3 (b) ) and scan separately block by block.The output is returned to the spatial domain after the corresponding k-layer wavelet packet inverse transformation and lastly multiplied element by element with 𝑆𝑖𝐿𝑈 (𝐹 𝐿𝑁 ) to obtain the frequency band scanning output 𝐹 𝑏 . This procedure is encapsulated as:EQUATIONwhere 𝑊 𝑃𝑇 and 𝐼𝑊 𝑃𝑇 donate discrete wavelet packet transform and inverse discrete wavelet packet transformation, respectively. We regard the frequency dimension of Mamba modeling as a transition between Mamba and Fourier modeling to achieve global degradation processing and local detail restoration seamlessly. 𝐹 𝑖𝑛 is added with the output of the spatial branch using a residual connection. By concatenating the output features of three branches and applying a final 1 × 1 convolution operation, the block achieves a harmonized synthesis of the features. In Fig. 5 , we present visualizations of the features from different branches to illustrate the variations in rain capture across each branch. the image background's complexity. As illustrated in Fig. 6 , the restoration performed by Restormer [51] exhibits a huge error with GT in areas such as mountains, characterized by dense rain patterns and intricate backgrounds. This discrepancy stems from the absence of explicit modeling of the degradation in the rainy images.To address this issue, we leverage the unique data-dependent property of the Mamba module, which allows it to dynamically focus on or disregard specific input features based on their significance. We introduce an innovative approach wherein we utilize the Mamba module to generate degradation priors, thereby enhancing our model's ability to discern and address the varying degrees of degradation across different regions of the image. This process involves the generation of degradation priors through spatial 2D scanning of input images at multiple scales. These degradation priors are then element-wise multiplied with features corresponding to the same scale and subsequently summed with the features:EQUATIONwhere 𝐼 𝑖𝑛 donates low-resolution rainy image and 𝐹 donates feature at the same scale. By incorporating input features of different scales, our model can effectively delineate the possible locations of degradation at various levels of granularity. As shown in Fig. 5 , we show some rainy images and the corresponding degradation attention maps. The rain streaks are highlighted in the attention maps. In essence, this approach harnesses the inherent adaptability of the Mamba module to dynamically allocate attention to different regions of the input image, thereby enabling our model to better address the intricate of degradation distribution across the image.In addition to new modules and degradation priors, we also introduce new loss functions to optimize the training process of the network to achieve good results in both spatial and frequency domains. The loss function consists of three parts: a spatial domain loss, a phase spectrum loss, and an amplitude spectrum loss.In the spatial domain, we use L1 loss between the final output and GT to achieve supervision. At the same time, to achieve better global information reconstruction, we calculate the L1 loss on the amplitude spectrum and phase spectrum respectively to obtain the amplitude and phase spectrum loss, expressed as:EQUATIONFinally, the overall composition of the loss function can be briefly expressed as follows:EQUATIONwhere 𝛼 and 𝛽 are both empirically set to 0.05 in our implementation. 
experiments: 
 In this section, we evaluate our method through extensive experiments. First, We describe the experimental setup. Then we present the comparison results between our method and state-of-the-art approaches qualitatively and quantitatively. Lastly, a detailed ablation analysis of our proposed method is conducted. Datasets. For validation, we train and validate our model on the widely-used datasets Rain100H [47] , Rain100L [47] , Test1200 [53] , and Test2800 [7] datasets. Rain100L is selected from BSD200 [30] with only one type of rain streaks, which consists of 200 image pairs for training and 100 image pairs for testing. Rain100H contains 1800 image pairs for training and 100 image pairs for testing with five types of streak directions. test1200 [53] has three groups of 12,000 light, medium, and heavy rainy images for training, with each group 4,000 images. The same three groups of 1,200 images are used for testing. Test2800 [7] contains 14000 image pairs, of which 12,600 pairs are used for training and 1,400 for testing. Implementation details. Our model was implemented using the Pytorch framework and executed on NVIDIA RTX 3090 GPUs. The number of blocks in each layer affects the amount of model parameters and rain removal performance. After weighing the balance, we set blocks of each layer to [2, 3, 3, 4, 3, 3, 2] , which can achieve good performance with a reasonable amount of parameters. We use the progressive training strategy. Taking the RAIN100L dataset as an example, we set the total number of iterations to 75000 and set the image size to [160, 256, 320, 384] while the corresponding batch sizes were [8, 4, 2, 1] . We use the Adam [21] optimizer with default parameters to train the network by minimizing the loss function 𝐿 𝑡𝑜𝑡𝑎𝑙 . The initial learning rate is set to 3 × 𝑒-4 and then gradually decays to 1 × 𝑒-6 using the cosine annealing strategy.In this section, we compare our method with state-of-the-art deraining approaches: DerainNet [6] , UMRL [49] , RESCAN [24] , PreNet [35] , MSPFN [19] , SPAIR [33] , MPRNet [52] , Restormer [51] , Fourmer [56] , IR-SDE [28] , MambaIR [13] , and VMambaIR [36] . Quantitative Comparison. Following [51] , we compute Peak Signalto-Noise Ratio(PSNR) and Structural Similarity(SSIM) scores using 8 , our method removes more rain streak residue while better restoring texture details. We provide more visualization results in the supplementary material.We performed ablations on different components of the model on the Rain100L dataset. More results are provided in the supplementary. Investigation of space mamba branch. To verify the efficacy of the space mamba branch, we replace Mamba blocks with standard convolution layers and keep other parts unchanged. The result presented in Tab.2 shows that the performance has dropped significantly, as Investigation of attention map. Our degradation prior attention strategy adaptively learns degradation distributions based on rainy images at different scales. To verify its effectiveness, we conduct ablation experiments in Tab. 2. It can be clearly seen that when equipped with the attention map, the algorithm gives better results than that without using degradation prior. Further, we show an error map in Fig. 9 with or without the attention map, which demonstrates the effectiveness of our degradation prior attention strategy. More comparisons can be found in the supplementary.Investigation of loss function. The frequency loss aims to directly emphasize global frequency information optimization. We remove it and its two components in Tab. 3 respectively to check the validity. The results show that removing it decreases performance metrics, indicating its importance.To demonstrate the potential of our FreqMamba, we extend it to low-light image enhancement and Real-world image dehazing.Extension on low-light image enhancement. Low-light image enhancement mainly focuses on illuminating the darkness of the scene and removing amplified noise. Our three-branch structure copes with this scenario well. We adopt LOL-V1 and LOL-V2-Synthetic datasets to evaluate the performance of our method. Several low-light image enhancement methods are selected for comparison: RetinexNet [41] , KinD [55] , ZeroDCE [12] , KinD ++ [54] , URetinex-Net [43] , FECNet [17] , and SNR-Aware [46] . Tab. 4shows the quantitative comparison.Extension on real-world image dehazing. Real-world dehazing aims to restore a clean scene from a real-world hazy image. For this task, we apply two datasets: Dense-Haze [1] and NH-HAZE [2] . We compare our method with others including DCP [15] , DehazeNet [4] , GridNet [25] , MSBDN [5] , and AECR-Net [42] . We present the quantitative comparison in Tab. 5.  
conclusion: 
 In this work, we introduce FreqMamba, an innovative deraining network that seamlessly integrates spatial domain sequence modeling and frequency domain global modeling to address the challenge of image deraining. The core of the FreqMamba network utilizes the unique frequency SSM block. Meanwhile we leverage Mamba's input dependency properties to generate attention maps across multiple scales. This integration enables a nuanced understanding and processing of rain streaks, distinguishing our approach from existing methods. Our comprehensive experiments on various datasets highlight the effectiveness and efficiency of FreqMamba. Notably, the model is not only good at removing rain from images but also preserves the integrity and detail of the underlying scene. In applications where final image quality is critical, the balance of performance and fidelity is critical. Furthermore, FreqMamba has applications beyond rain removal, demonstrating its versatility in a variety of image restoration tasks. This adaptability illustrates the robustness of the underlying architecture and its potential as a foundational model for future image restoration research.  
: 
 difficult job due to the complex coupling of raindrops to the background and the loss of important perception frequency information.Traditional methods have relied on prior knowledge to dissect the physical properties of rain and background layers, introducing various methods to distinguish rain streaks from clean images. However, these priors, being based on specific observations, might not be reliable for modeling intrinsic features of images or estimating transmission maps in physical models. The advent of deep learning has heralded new directions in rain removal techniques. Many methods are proposed to improve the performance of rain removal methods from different perspectives [8, 14, 35, 44] . Especially, effective global degradation modeling has proved crucial for addressing intricate challenges of image deraining. For instance, the attention mechanism in Transformer achieves great success with the capacity to model image-internal correlations. While, attention mechanisms face scalability challenges due to their quadratic complexity, posing a significant challenge when addressing large images.Recently, an improved structured state-space sequence model (S4) with a selective scanning mechanism, Mamba, stands out due to its ability to model long-range sequence relationships with linear complexity. Specifically, Mamba's selective methodology can explicitly build the correlation among image patches, further enabling the guidance of clean to degraded areas. However, it is noteworthy that Mamba's approach to processing visual tasks is fundamentally based on pixel sequences. While this allows it to model long-range dependencies, the selective modeling of one-dimensional sequences limits its ability in "global degradation perception" like the Fourier transform. Recognizing this, we attempt to extend it from the perspective of frequency analysis for image deraining.We integrate two typical frequency analysis techniques into our method: Fourier transform (FT) and wavelet packet transform (WPT). It is well known that the Fourier transform has effective global modeling capabilities. In addition, it has good degradation separation properties (Fig. 2 ). This unique perspective is invaluable for tasks such as image deraining, where understanding the entire image structure can significantly enhance the quality of the restoration result. However, there are inherent gaps between different domains, so simply combining the Fourier transform with Mamba modeling can be quite blunt. We need an intermediate state for a seamless transition. The wavelet packet transform decomposes the image into a series of sub-bands with varying frequencies in the spatial domain, encapsulating both frequency analysis and spatial information simultaneously. By utilizing the wavelet domain as the intermediate state between the spatial and Fourier domains, we establish a smoother transition, enhancing the overall effectiveness of the analysis.Recognizing the potential for perceiving global degradation via Frequency analysis and Mamba's ability to capture regional correlations within the spatial domain, we introduce FreqMamba. This effective and efficient paradigm utilizes the complementarity between Mamba and frequency analysis for image deraining. The cornerstone of our approach lies in a three-branch structure.• Spatial Mamba: This branch operates on the original image features. It extracts details and correlations within the image, providing crucial insights into degradation patterns. • Frequency Band Mamba: This branch employs the WPT to dissect the input features into a spectrum of features spread across various frequency bands. Arranging these low-resolution features back to the original scale in frequency order, we perform a frequency dimension scanning from low to high frequency as well as in reverse as shown in Fig. 3 . This strategy not only enriches the model's analytical breadth but also acts as a pivotal bridge, melding the spatial and Fourier domains into a unified analysis framework, offering a novel perspective on modeling. • Fourier Modeling: This branch equips the model with the ability to conduct global analysis by leveraging the Fourier Transform to process the input. It captures the overarching degradation patterns that affect the image, offering a panoramic view of the frequency spectrum. This global modeling capability is instrumental in comprehensively understanding and mitigating the effects of degradation, ensuring an in-depth removal of rain streaks from the image for a cleaner visual outcome. Together, these branches form a robust architecture for tackling the challenge of image deraining. The Spatial Scanning Mamba analyzes intricate spatial details, while the Fourier Modeling branch offers a holistic view, enabling the model to understand global degradation phenomena. The Frequency Band Scanning Mamba explores the frequency dimension and provides a new perspective for 2D modeling. Furthermore, leveraging the unique data-dependent characteristics of Mamba, we apply this methodology to rainy images across different scales to derive attention maps based on degradation prior. These attention maps are then integrated into the backbone network to assist in effective training. 
 
 Appendix:  


Decision: The overall quality of the paper is high. It presents a well-structured approach combining Mamba and frequency domain analysis, leading to impressive results on standard benchmarks. The experimental validation is robust, and the theoretical foundation is sound.

The paper is generally well-written, but some sections could benefit from improved clarity and additional explanations.

The paper presents a novel integration of Mamba with frequency analysis for image deraining, which is original and contributes to the field.

The paper presents a novel and effective approach to image deraining, with strong experimental results. However, addressing the mentioned concerns will strengthen the paper and provide a more comprehensive evaluation of its contributions.

Reviews: review: paper_summary: paper_summary: This paper proposes FreqMamba, a method that leverages the complementary between Mamba and frequency analysis for image deraining.  Specifically, FreqMamba comprises three branches: Spatial Mamba, Frequency Band Mamba, and Fourier Modeling. Spatial Mamba focuses on extracting details and correlations from original image features. Frequency Band Mamba dissects input features into various frequency bands using wavelet packet transform. Fourier Modeling utilizes the Fourier Transform for global analysis capabilities and refining overarching degradation patterns.
strength_weakness: strength_weakness: **Experimental validation**

- This paper presents state-of-the-art results on the Rain100L, Rain100H, Test2800 and Test1200 benchmark datasets for single image deraining. In addition, the results in low light enhancement and removal of non-uniform haze were also additionally demonstrated.
- This paper provides a rich analysis of visual results.

**Technical correctness**
- The paper formulation is technically sound.

**method novelty**
- This paper combines Mamba from the perspective of frequency domain to explore the characteristics and connections of degraded images in the frequency domain state space. It has certain inspiration and reference value in low level visual tasks.
limitations: limitations: - Although this paper compares many methods on the derivation task, there is still a lack of comparison with some representative methods, such as DRSformer [1], IDT [2]. In addition, there is a lack of comparison with Retinexformer [3] in low light image enhancement, and a lack of comparison with the last two years in the NH-HAZE dataset.

- This paper for image deraining whether experiment within the following benchmark data: DID Data, DDN Data and SPA Data?

- Is the comparison method in this article retrained while maintaining the same fair participation?

- This paper lacks generalization experiments on real-world datasets.

[1] Chen X, Li H, Li M, et al. Learning a sparse transformer network for effective image deraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 5896-5905.

[2] Xiao J, Fu X, Liu A, et al. Image de-raining transformer[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

[3] Cai Y, Bian H, Lin J, et al. Retinexformer: One-stage retinex-based transformer for low-light image enhancement[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 12504-12513.
paper_summary: paper_summary: FreqMamba utilizes the complementarity between Mamba and frequency analysis to de-rain images, extends Mamba with frequency bands to take advantage of frequency correlation, and connects it with the Fourier transform to model global degradation. It is visually and quantitatively superior to the most advanced methods (e.g. VMambaIR, MambaIR)
strength_weakness: strength_weakness: The paper proposes FreqMamba which lies in a three-branch (Spatial Mamba + Frequency Band Mamba + Fourier Modeling) structure. These branches form a robust architecture for tackling the challenge of image deraining..
limitations: limitations: 1. The improvement of Frequency brand is marginal (shown in Table 2), which makes me concerned about the the novelty of the paper.
2. Why not use FFT in Frequency brand? DWT is still a local transform.
3. The paper lacks the reference for Frequency Domain Learning, e.g. FFC [1], LaMa [2], DeepRFT [3] and so on.

[1] Fast fourier convolution. NeurIPS, 2020.

[2] Resolution-robust large mask inpainting with fourier convolutions. WACV, 2022.

[3] Intriguing findings of frequency selection for image deblurring. AAAI, 2023.
paper_summary: paper_summary: The paper proposes a novel method for image deraining, called FreqMamba, which leverages frequency analysis and the Mamba model to achieve state-of-the-art results. The method consists of three branches: spatial Mamba, frequency band Mamba, and Fourier global modeling, which work together to capture both local and global degradation patterns. The authors conduct extensive experiments to validate the effectiveness of their approach, comparing it to several state-of-the-art methods on various datasets. The results show that FreqMamba outperforms the other methods both visually and quantitatively, achieving better spectral fidelity and fewer residues.
strength_weakness: strength_weakness: Integrating frequency analysis with spatial modeling via Mamba is a novel approach for image deraining. Exploiting the complementary strengths of frequency and spatial domains could lead to improved global and local degradation modeling.
The paper provides a sound theoretical foundation, describing frequency analysis concepts like Fourier transform, wavelet transform, and state space models clearly. The proposed three-branch architecture (Fourier Modeling, Spatial Mamba, Frequency Band Mamba) is well-motivated.
limitations: limitations: While the integration of different components is novel, the individual components (Fourier transform, wavelet transform, Mamba) are well-established techniques.
The proposed architecture appears to be computationally expensive due to the multiple branches and transformations involved, which could limit its practical applications.
paper_summary: paper_summary: This paper proposes to utilize the latest Mamba architecture together with Fourier transform to address image deraining task. It introduces a Frequency-based Mamba framework, which consist of three branches, i.e., spatial mamba, frequency band mamba and fourier modeling. Extensive experiments are conducted on diverse low-level vision tasks, e.g., image deraining, image dehazing, low-light image enhancement.
strength_weakness: strength_weakness: It is an interesting and pioneer work to introduce SSMs into low-level vision tasks.
Extensive experiments on image deraining, dehazing, low-light image enhancement are conducted, demonstrating its effectiveness.
Many visualization results of frequency and attention maps are displayed to better demonstrate the effectiveness of the proposed modules.
limitations: limitations: 1. The motivation of using Mamba is unclear. Mamba is designed to causally process text data in NLP tasks. However, for non-causal data 2D images, Mamba takes no advantage and may destroy the spatial relations by flattening them into sequences. Moreover, Mamba have superior performance when coming to long sequences. For 2D images without high resolution, self-attention is sufficient to address such short sequences in image-deraining task. Please further clarify.
2. Related work on State Space Models is insufficient. Many relevant works have sprung out, such as MambaIR, VMambaIR, T-Mamba, SegMamba, Vivim. The author should clarify the core differences compared to other works.
3. Sec. 3.1 Motivation is cumbersome to understand. The definition of global degradation is unclear. The author should explain more specifically. In addition, the author stated that frequency domain operations only focus on the global dependency while Mamba can complement it. However, as far as I know, Mamba is introduced to replace self-attention for efficient global modeling in many works. There is no analysis or experimental results to validate the effectiveness of Mamba in modeling local dependencies.
4. In Fig. 7&8, the author should also display the results of the second-best method VMambaIR.
5. In Tab. 2, removing the spatial branch brings a significant drop in PSNR, SSIM while Fourier and FrequencyBand contribute little gain. It seems that vanilla Mamba is taking the heavy lifting.