# Harnessing Spatial-Frequency Information for Enhanced Image Restoration
## 3 Methodology
### Multi-Domain Feature Extractor (MDFE)  
As illustrated in Figure 1c, the MDFE is composed of inter-multi-branches, each operating in different spaces, including the spatial and frequency domains. The spatial  
Figure 1: The overall architecture of our FreANet. (**a**) FreABlock is composed of MPA, the RDB, the MDFE, and a standard feed-forward network. (**b,c**) The detailed structures of the MDFE and MPA, respectively. branch is designed with an intra-multi-branch structure consisting of parallel branches with small to large receptive fields, effectively capturing the multi-scale spatial context. We adopt two serial depthwise convolutions with strip kernels [16] to maintain a lightweight network in each branch. The process in the spatial branch is expressed as  
\[f_{\text{s}}=\sum_{i=0}^{2}\text{DWConv}_{k_{i}\times 1}\big{(}\text{DWConv}_{1 \times k_{i}}(x)\big{)}, \tag{2}\]  
where \(f_{\text{s}}\) represents the output features of the spatial branch. DWConv and \(k_{i}\), with \(i\in\{0,1,2\}\), refer to the depthwise convolution and the kernel size of the \(i\)-th branch, respectively. Before being fed into the frequency branch, the input features are converted into their corresponding complex features by applying the FFT algorithm. Given that the real and imaginary components of complex features originate from corresponding spatial features, applying individual convolutions to these components is inappropriate. Therefore, we apply complex convolutions to the complex features, which involve interactions between the real and imaginary components. Complex convolutions are implemented similarly to standard convolutions, but each convolution kernel is responsible for the real \(W_{\text{re}}\) and imaginary \(W_{\text{im}}\) weights. The complex kernel \(W\) is defined as \(W=W_{\text{re}}+jW_{\text{im}}\), and the complex convolution is expressed as \(W\)\(X=(W_{\text{re}}X_{\text{re}}-W_{\text{im}}X_{\text{im}})+j(W_{\text{re}}X_{ \text{im}}+W_{\text{im}}X_{\text{re}})\). Like the spatial branch, we utilize two serial depthwise convolutions with strip kernels, employing them in the manner of complex convolutions. The process in the frequency branch is expressed as  
\[F_{\text{f,re}}+jF_{\text{f,im}}=\text{DWCConv}_{21\times 1}\big{(}\text{DW CConv}_{1\times 21}(X)\big{)}, \tag{3}\]  
\[X=\mathcal{F}(x),\quad f_{\text{f}}=\mathcal{F}^{-1}(F_{\text{f,re}}+jF_{ \text{f,im}}), \tag{4}\]  
where \(F_{\text{f,re}}\) and \(F_{\text{f,im}}\) are the real and imaginary output features of the frequency branch. DWCConv refers to the depthwise complex convolution, and \(F_{\text{f}}\) the output features of the frequency branch, which are converted to the spatial domain through the inverse Fourier transform. Finally, inspired by SKNet [36], the output features extracted from different domains are integrated by computing weights for each channel. The fusion process is expressed as  
\[Z=\text{MLP}\big{(}\text{GAP}(f_{\text{s}}+f_{\text{f}})\big{)}, \tag{5}\]  
where \(Z\in\mathbb{R}^{1\times 1\times D}\) represents compressed features, with the number of channels denoted by \(D\). MLP and GAP refer to a fully connected layer and global average pooling. Next, two separate fully connected layers are applied, followed by a softmax operation, to compute the weights for each channel:  
\[[\alpha,\ \beta]=\big{[}\text{MLP}_{\alpha}(Z),\ \text{MLP}_{\beta}(Z)\big{]}, \tag{6}\]  
\[\alpha^{\text{c}}=\frac{e^{\alpha^{\text{c}}}}{e^{\alpha^{\text{c}}}+e^{\beta ^{\text{c}}}},\ \beta^{\text{c}}=\frac{e^{\beta^{\text{c}}}}{e^{\alpha^{\text{c}}}+e^{\beta^{ \text{c}}}}, \tag{7}\]  
\[f^{\text{c}}=\alpha^{\text{c}}\otimes f_{\text{s}}^{\text{c}}+\beta^{\text{c} }\otimes f_{\text{f}}^{\text{c}},\ \text{where}\ \alpha^{\text{c}}+\beta^{\text{c}}=1, \tag{8}\]  
where \(f=[f^{1},\,f^{2},\,\cdots,\,f^{\text{c}}]\). \(\alpha\) and \(\beta\) represent the weights per channel for \(f_{\text{s}}\) and \(f_{\text{f}}\), respectively.