# Harnessing Spatial-Frequency Information for Enhanced Image Restoration
## 1 Introduction  
Image restoration, a classic task in computer vision, aims to recover a high-quality, clean image by removing specific types of degradation, such as haze, blur, and low light. Since images affected by degradation can adversely impact the performance of computer vision models, the importance of image restoration is emphasized in various applications, including autonomous driving, surveillance, and remote sensing. However, obtaining a single effective solution is challenging because image restoration is an inherently ill-posed problem. To address this challenge, conventional approaches [1, 2] often rely on strong image priors, which are limited in producing consistent results across diverse, unseen environments and scenarios. With the advent of deep learning, the image restoration task has experienced a rapid paradigm shift from conventional approaches to deep learning-based methods. Various deep learning models have been proposed, demonstrating substantial improvements over traditional approaches, such as multi-stage frameworks [3, 4], normalization techniques [5], and attention mechanisms [6, 7]. More recently, many researchers have shifted their focus tomulti-head self-attention (MHA) [8], which captures long-range dependencies and demonstrates remarkable performance [7, 9]. However, when an image with a resolution of \(h\times w\) is given, the computational complexity of MHA is \(\mathcal{O}(h^{2}w^{2})\), which requires significant memory resources, making it unsuitable for high-resolution images. Since various types of degradation distort texture, edges, and other features in an image, the discrepancy between the degraded and clean images in the frequency domain is essential. Nonetheless, only a few approaches utilize the latent space of the frequency domain [10, 11, 12, 13, 14]. Among them, most methods process features by directly transforming them using various transformation tools, such as the wavelet transform [12, 13] and the Fourier transform [14, 15]. Some recently published studies designed learnable low-pass filters through pooling techniques [10, 11] to implicitly process frequency information. These approaches have substantially improved restoration capabilities in simple yet powerful ways, demonstrating promising potential in image restoration. However, previous methods [14, 15] related to modeling the frequency space, particularly those using the discrete Fourier transform (DFT), apply two independent convolutional neural networks (CNNs) to process complex features consisting of real and imaginary components, e.g., \(Y=X_{real}W_{real}+j(X_{imag}W_{imag})\). This approach has an upper limit on performance due to a mathematical error in the operation. In this study, we designed a Frequency-Aware Net work (FreANet) to efficiently the bridge frequency gaps between degraded and sharp images and capture spatial interactions with low computational complexity. Specifically, we propose a multi-domain feature extractor (MDFE) that adaptively aggregates spatial and frequency contexts for various types of degradation. The MDFE is composed of spatial and frequency branches, which utilize convolutions with large strip kernels (\(1\times k\) or \(k\times 1\)) to produce rich representations [16]. In the frequency branch, we introduce complex convolutions (CCNNs) [17] to handle the complex features transformed via the DFT. To reduce the computational complexity of the attention mechanism, we developed a multi-scale pooling attention (MPA) mechanism, which leverages unidirectional global average pooling (GAP) and strip convolutions with multiple scales. This design bypasses the quadratic computational complexity, \(\mathcal{O}(h^{2}w^{2})\), of MHA, allowing the network to achieve linear computational complexity, \(\mathcal{O}(hw)\). We conducted extensive experiments on four image restoration tasks, including defocus deblurring, motion deblurring, dehazing, and low-light enhancement.