Selective Frequency Network for Image Restoration
Authors: Yuning Cui
Yi Tao
Zhenshan Bing
Wenqi Ren
Xinwei Gao
Xiaochun Cao
Kai Huang
Alois Knoll
Venue: ICLR 2023 poster (2023)
Keywords: Image restoration
Frequency domain
Frequency selection

Abstract: Image restoration aims to reconstruct the latent sharp image from its corrupted counterpart. Besides dealing with this long-standing task in the spatial domain, a few approaches seek solutions in the frequency domain in consideration of the large discrepancy between spectra of sharp/degraded image pairs. However, these works commonly utilize transformation tools, e.g., wavelet transform, to split features into several frequency parts, which is not flexible enough to select the most informative frequency component to recover. In this paper, we exploit a multi-branch and content-aware module to decompose features into separate frequency subbands dynamically and locally, and then accentuate the useful ones via channel-wise attention weights. In addition, to handle large-scale degradation blurs, we propose an extremely simple decoupling and modulation module to enlarge the receptive field via global and window-based average pooling. Integrating two developed modules into a U-Net backbone, the proposed Selective Frequency Network (SFNet) performs favorably against state-of-the-art algorithms on five image restoration tasks, including single-image defocus deblurring, image dehazing, image motion deblurring, image desnowing, and image deraining.

Published as a conference paper at ICLR 2023
SELECTIVE FREQUENCY NETWORK FOR IMAGE RESTORATION
Yuning Cui', Yi Tao?, Zhenshan Bing', Wenqi Ren 3.5, Xinwei Gao Xiaochun Cao Kai Huang". Alois Knoll'
TTechnical University of Munich MIT Universal Village Program 3Sun Yat-sen University *Tencent "Chinese Academy of Sciences
(yuning.cum cui, bing, knollj@in tum. de, de.yttaoeumiiversl--llleee (renwq3, caoxia huangk3 6) @mail 1 sysu. edu. on, wit.gaatteteeeenttcccmmm
ABSTRACT
Image restoration aims reconstruct the latent sharp image from its corrupted counterpart. Besides dealing with this long-standing task in the spatial domain, a few approaches seek solutions in the frequency domain in consideration of the large discrepancy between spectra of sharp/degraded image pairs. However, these split works commonly utilize transformation tools, e.g., wavelet transform, features into several frequency parts, which is not flexible enough to select the most informative frequency component recover. In this paper, we exploit a multi-branch and content-aware module to decompose features into separate fre- quency subbands dynamically and locally, and then accentuate the useful ones via channel- -wise attention weights. In addition, handle large-scale degradation blurs, we propose an extremely simple decoupling and modulation module to en- large the receptive field via global and window-based average pooling. Integrating two developed modules into U-Net backbone, the proposed Selective Frequency Network (SFNet) performs favorably against state-of-the-art algorithms five image restoration tasks, including single-image defocus deblurring, image dehaz- ing, image motion deblurring, image desnowing, and image deraining
INTRODUCTION
Image restoration aims to recover high-quality image by removing degradations, e.g. noise, blur, and snowflake. In view of its important role in surveillance, self-driving techniques, and remote sensing, image restoration has gathered considerable attention from industrial and academic com- munities. However, due to its ill-posed property, many conventional approaches address this prob- lem based on various assumptions (Zhang et al. 2022 Yang al. 2020b) or hand-crafted features
Karaali & Jung 2017). which are incapable of generating faithful results in weal-world scenarios.
Recently, deep neural networks have witnessed the rapid development of image restoration and ob- tained favorable performance compared to conventional methods. A flurry of convolutional neural networks (CNN) based methods have been developed for diverse image restoration tasks by invent- ing or borrowing advanced modules, including dilated convolution (Luo et al., [022] Zou et al.] 2021.. U-Net (Ronneberger et al. (2015), residual learning (Zhang et al., | 2017). multi-stage pipeline (Zhang et al. 2019b), and attention mechanisms (Liu et al. 2019). However, with convolution units, these methods have limited receptive fields, and thus they are not capable of capturing long- oangerange dependencies. This requirement is essential for restoration tasks, since a single pixel needs infor- mation from its surrounding region to be recovered. More recently, many researchers have tailored Transformer (Vaswani et al. 2017) for image restoration tasks, such as motion deblurring (T'si et al.) 2022], dehazing (Guo al.
2022 Song et al., 2022) and desnowing (Chen et al., 2022b c).
Nonetheless, the above-mentioned methods mainly conduct restoration in the spatial domain, which do not sufficiently leverage frequency discrepancies between sharp/degraded image pairs. To this end, few works utilize the transformation tools, e,g. wavelet transform or Fourier transform, to
"Corresponding author
'Ou code and models are available http:////gggihib.cooo///////////

---

Published as a conference paper at ICLR 2023
decompose features into different frequency components and then treat separate parts individually to reconstruct the corresponding feature (Selesnick et al. 2005 Yang & Fu 2019 Zou et al. 2021 Mao et al. 202I). Nevertheless, wavelet transtorm decouples the feature map into different subbands in a fixed manner, and thus it is not capable of distinguishing the most informative or useless fre- quency components enhance or suppress. In addition, these methods need corresponding inverse Fourier/wavelet transform, leading to additional computation overhead.
To overcome the above drawbacks and select the most informative frequency component re- construct, we propose a novel decoupling and recalibration module for image restoration tasks, named Multi-branche Dynamic Selective Frequency module (MDSF). Specificallyy, we utilize the multi-branch learnable filters to generate high- and low-frequency maps dynamically and locally, and then leverage the channel-wise attention mechanism, modified from (Li al.) 2019), to empha- size or suppress the resulting frequency components. Our module has two key advantages. Firstly, according to the input and task, the decoupling step dynamically generates filters decompose feature maps. Secondly, our module does not introduce extra inverse transform.
Receptive field is another critical factor for image restoration tasks due to the various sizes of degra- dation blurs (Suin et al. 2020 Son et al. 2021). To complement the above dynamic module, MDSF, that processes features locally, we further propose a simple yet effective module, dubbed Multi- branch Compact Seleetive Frequency module (MCSF). to enhance the helpful frequency signals based on multiple and relatively global receptive fields. Specifically, we utilize global and window- based average pooling techniques to attain disparate frequency maps, and then use learnable pa- rameters to modulate the resulting maps without resorting to any convolution layers. Compared MDSF, besides the enlarged receptive fields, MCSF is lightweight enough to be embedded in multiple positions of the backbone. The main contributions of this study are summarized as follows:
We propose multi-branch dynamic selective frequency module (MDSF) that is capable of decoupling feature maps into different frequency components dynamically via the theoret- ically proved filters, and selecting the most informative components to recover.
We develop multi-branch compact selective frequency module (MCSF) that performs frequency decoupling and recalibration using multi-scale average pooling operations to pursue a large receptive field for large-scale degradation blurs.
Incorporating MDSF and MCSF into U-shaped backbone, the proposed selective fre- quency network (SFNet) achieves state-of-the-an results on five image restoration tasks, including image defocus/motion deblurring, dehazing, deraining, and desnowing.
RELATED WORK
Image Restoration. Prior to the deep learning cra, a great number of methods have been proposed for image restoration problems based on various assumptions and hand-crafted features (Sezan &
Tekalp 1990, Kundur & Hatzinakos] 1996 Calvetti et al. 1999). In recent years, with the rapid development of deep learning, a flurry of approaches have been investigated utilizing convolutional
neural networks for image motion deblurring (Zamir et al. 2021 Yuan et al., 2020 Cui et al.) 2023
Purohit et al. 2021), defocus deblurring (Ruan et al.) 2022] Abuolaim & Brown] 2020
Son al.
2021
desnowing (Chen et al. 2021c| 2020a), dehazing (Dong et al. [2200] Liu et al.
2019 Ren
al.
2016 Zhang al. 2018), and deraining (Yang al. 2020b Wang al. 2019).
More recently, to capture long-range dependencies, many works have borrowed Transformer (Vawwani al. 2017) from the natural language processing field into image restoration (Chen al.
2021a Liang et al. 2021 Zamir et al. 2022 Wang et al. (022) and specific tasks such as image motion deblurring (Tsai et al.) 2022). dehazing (Song et al.J 2022). and desnowing (Chen et al.) 2022b). In this study, instead of exploiting more advanced backbone for image restoration, we pay
more attention to the frequency selection mechanism based on efficient CNN.
Frequency Based Image Restoration. Many algorithms have been developed to address various low-level vision problems from a frequency perspective. Specifically, Chen et al. (Chen et al.) 202 propose hierarchical desnowing network based on dual-tree complex wavelet representa- tion (Selesnick et al. 2005). Yang et al. (Yang & Ful 2019) develop the wavelet based U-Net to replace up-sampling and down-sampling, Zou et al. (Zou et al. 2021) utilize wavelet transform based module to help recover texture details. Yang et al. (Yang et al. 2020a) devise a wavelet
structure similarity loss function for training. Mao el al. (Mao al. 2021
Fourier transform to
2

---

Published as a conference paper at ICLR 2023
Degraded image
Image
Conv 1
Com 3x3
MCS
Come 3x3
Moriulator MDSE
MCSF
Eoni
[b] Shallow Layer
Cony 3x3
Cony 3x
Sofumax
X
Flement-won
Murtplication
Resälock
ResBlock
Flement-wwie Adcition
Shallow Layer
Conv
Concateration
Invert
Upsample
ResBlock
20 x
ResBlock
Conv 3x3
X/e
Downsomple
[d) Decoupler
+ Convolution
Layer
Conv 3x3
Conv 1x1
Softmax
Spii
ResBlock
ResBlock
Canv 3anv 3x
Overall architecture
Modulator
Figure 1: (a) Overall architecture of the proposed SFNet. (b) Shallow layer extracts the shallow feature for low-resolution images. (c) ResBlock contains the proposed modules: MDSF (Decoupler (d) and Modulator (e)) and MCSF, MDSF is shown in the one-branch case for clarity. Invert depicts the operation of subtracting the low-pass filter from the identity filter.
integrate both high- and low-frequency residual information. Yoo el al. Yoo et al. 2018) complete image restoration based the estimation of the DCT coefficient distribution. In this work, pur- sue dynamic and efficient manner to select the useful frequency part at multiple receptive fields.
METHODOLOGY
We first describe the overall architecture of SFNet (Fig. | modules, MDSF (Fig. [ (d.e)) and MCSF. The loss functions follow in the final part.
(a)). Then we present the proposed
3.1 OVERALL ARCHITECTURE
Our network adopts the encoder-decoder architecture learn hierarchical representations. Specifi- cally, SFNet consists three-scale decoder and a three-scale encoder. Each scale is comprised of
ResBlock (Fig. (c)). MDSF only deployed in the last residual block of each ResBlock while
MCSF exists in all blocks. Following previous methods (Cho et al. 2021 Mao et al. 2021 Tu
al. 2022). multi-input and multi-output mechanisms are used case training difficulty. Specili- cally. input images of reduced sizes are merged into the main path via the shallow layer (Fig. (b)), and the predicted images are produced by In addition, adopt feature-level and image-level skip connections to assist training. In Fig. [ we only show the level image skip connection for clarity. The up-sampling and down-sampling layers are implemented by transposed and strided convolutions, respectively.
convolutional layers after each scale of decoder.
3.2 MUL.TIBBRANCH DYNAMIC SELECTIVE FREQUENCY MODULE (MDSF)
To select the informative frequency component reconstruct, MDSF mainly contains two elements: frequency decoupler (Fig. (d)) and modulator (Fig. [ (e)). Decoupler decomposes features into separate frequency parts dynamically based on learned filters, and then modulator utilizes channel- wise attention to accentuate the useful frequency. Additionally, to provide various local receptive fields, MDSF splits features among the channel dimension, and then applies different filter sizes to separate parts. We only show the one-branch case in Fig. (d) for simplicity.
To dynamically decompose feature maps, we utilize the learnable and theoretically proven low-pass filter (refer Appendix B for the proof) and the corresponding high-pass filter to generate low- and high-frequency maps. The learned filters are shared across the group dimension to strike a balance between complexity and feature diversity. Specifically, given any feature map X CR x where C' is the number of channels and H x W' denotes the spatial dimension. we firstly leverage the generaigne layer to produce the low-pass filter for each group of the input, formulated Fl Softmax (GAP(XX))
3

---

Published as a conference paper at ICLR 2023
k k is the kernel size of low-pass filter; g denotes the number of groups; B, W. and GAP are Batch Normalization (loffe & Szegedy 2015), the parameters of convolution and global average pooling, respectively. The group- group-based based operation has fewer parameters and lower complexity than generating filters for each pixel. The number of groups is discussed in Sec 4.3
where Fl E RXXxxxx,, poxxxxi,
To attain the high-pass filter, subtract the resulting low-pass filter from the identity kernel with central value as one and everywhere else as zero. Next, for each group feature X; E xx
where is the group index and C C its low- and high-frequency components can be obtained by 9 using the corresponding reshaped filter FL and FH (E Roxkxa), which is expressed as:
X Kin
-F Pt.por Kipph X L.c.h,,
Σ F Ximaaiiichppno
P-q
P-
where e is the index of a channel; h and w denote spatial coordinates; and p. E f-1,0,
After decoupling the feature map into different frequency components, we leverage the frequency modulator to emphasize the genuinely useful part for reconstruction, as illustrated in Fig. 1 (e).
The modulator works among the channel dimension based on the modified SKNet (Li et al. 2020 Formally, given two frequency maps, X and X²,, we first generate the fused feature by.
Z (GAP((( + +X))
where Wre is the parameters of fully connected layer. To attain channel- weights, we use two other fully connected layers followed by concatenation and Softmax function, formulated as:
(.W.(2),W.
W Wh) 2C (I1(2)).%
(4)
where W and hh are thannel-wise attention weights for two frequency parts; W2 and W,, are parameters of fully connected layers: [;,-] denotes concatenation; and is the channel index of con- catenated features. Compared to SKNet (Li et al., 2020). which performs Softmax on each channel [,,(Z) W _ we consider all channels into consideration facilitate interactions between different channels of two maps. Then, the final weights can be obtained by split operation.
Based on the above one-branch case, the multiple branches with varied filter sizes can be express as: X = (M))) .... Mm (Dm((x))
(5)
where D and M denote decoupler and modulator, respectively, and Xm the equally split feature.
3.3 MULTI-BRANCH COMPACT SELECTIVE FREQUENCY MODULE (MCSF)
Since receptive field plays a critical role in image restoration, where degradation blurs always differ in size (Son et al., 2021 Mao et al. 2021 J, develop MCSF to efficiently enlarge the receptive field of SFNet. MCSF has two branches with different receptive fields, i.e., the global branch and window-based branch. Considering these branches share a similar paradigm, we only detail the window-based one, which is inspired by the idea of window-based attention (Liu et al. 2021).
Specifically, given the split feature X E x is partitioned into four windows, each with the size of 2C H x W. To get the low-frequency part, global average pooling applied the result- ing windows (refer to Appendix C for analyses of this option). The corresponding high-frequency part can be obtained by subtracting the low-frequency map from the partitioned feature. To select the useful frequency subbands, we rescale these two maps by learnable weights, which are directly optimized by backpropagation. Finally, the updated frequency maps are reversed to the original resolution. The global branch has similar pipeline, yet with global receptive field.
Compared to MDSF, besides the enlarged receptive field, MCSF does not accomplish frequency decoupling and modulating with the aid of convolution layers, resulting in fewer parameters and lower complexity (see Tab. 9 for details). Hence, MCSF can be embedded in multiple positions.
3.4 LOss FUNCTION
To facilitate the frequency selection process, we adopt L, loss in both spatial and frequency domains:
3
3
Lepotial
Σ
1
X
Yelli;
Lirequency
- - -FX)))
(6)
r=1
4

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative comparisons with previous leading single-image defocus deblurring methods
on DPDD testset (Abuolaim & Brown 2020), which contains 39 outdoor and 37 indoor scenes.
Indoor Scenes
Outdoor Scenes
Combined
Method
PSNR1
SSIM MAE LPIPS
PSNRT
SSIM1 MAEJ
LPIPS
PSNR
SSIM MAE
LPIPS.
EBDB
25.77
0.772
0.040
0.297
21.25
0.599
0.058
0.373
23.45
0.683
0.049
0.336
DMENet
25.50
0.788
0.038
0.298
21.43
0.644
0.063
0.397
23.41
0.714
0.051
0.349
NB
26.73
0.828
0.031
0.273
21.10
0.608
0.064
0.355
23.84
0.715
0.048
0.315
DPDNet
26.54
0.816
0.031
0.239
22.25
0.682
0.056
0.313
24.34
0.747
0.044
0.277
KPAC
27.97
0.852
0.026
0.182
22.62
0.701
0.053
0.269
25.22
0.774
0.040
0.227
IFAN
28.11
0.861
0.026
0.179
22.76
0.720
0.032
0.254
25.37
0.789
0.039
0.217
DRBNet
25.72
0.791
0.183
Restormer
28.87
0.882
0.025
0.145
23.24
0.743
0.050
0.209
25.98
0.811
0.038
0.178
SFNet
29.16
0.878
0.023
0.168
23.45
0.747
0.049
0.244
26.23
0.811
0.037
0.207
Blany Imsuge
PSNR Refermce
23.15 JB Inpur
23.04 dB KPAC
25,84 DesplFFi+
26.50 DRINNet
25.68 Uiferrrr
25.60 Restoneer
27.77 dB SPNet
Figure 2: Single-image defocus deblurring results on the DPDD dataset (Abuolaim & Brown 2020
where r denotes the index of input/output images of different resolutions; represents fast Fourier transform; S, is the number of elements for normalization; and X,, Y, are output and target images, respectively. The final loss function is given by I. Isspatial Alefrequency- where À is 0.1.
4 EXPERIMENTAL RESULTS
4.1 SETTINGS
We evaluate the proposed SFNet on five restoration tasks: image motion/defocus deblurring, image deraining. image dehazing, and image desnowing. More details of the used datasets and training settings for each task are provided in Appendix A. FLOPs are computed on patch size of 256 x 256.
We train separate models for different tasks. Unless mentioned otherwise, the following parameters are adopted. The batch size is set as 4 with patch size of 256 x 256. Each patch is randomly flipped horizontally for data augmentation. The initial learning rate 1 and gradually reduced 1 with the cosine annealing (Loshchilov & Hutter 2016). Adam (8) 0.9, 99 = 0.999) is used for training. N set to 15 in Fig. [ MDSF has two branches with filter kernel sizes of x 3 and 5 x 5, respectively, and the number of groups is 8. We use PyTorch to implement our models an NVIDIA Tesla V100 GPU.
4.2 MAIN RESULTS
Single-image defocus deblurring results. Tab. shows comparisons of defocus deblurring meth- ods on DPDD (Abuolaim Brown 2020). Our SFNet surpasses other statc-of-the-ar art methods on most metrics. Particularly on the combined scene, SFNet obtains 0.25 dB PSNR improvement com- pared to the strong Transformer method Restormer (Zamir et al., 2022). provides significant gain of 0.51 dB PSNR over the pre- wre-trained network DRBNet (Ruan et al.) 2022]. The visual results in Fig. show that our method recovers more details than other algorithms. Due space limitations, experimental results of the dual-pixel setting provided in Appendix E.
addition, our method
PSNR
Iapui
Ghdllebereeee
FFA-Na
MANI
Delffamer
SEN
Figure Image dehazing results on the SOTS-Indood dataset (Li et al. 2018).

---

Published as a conference paper at ICLR 2023
Table 2: Image dehazing comparisons the synthetic dehazing datasets: SOTS-Outdoor and SOTS-Indoor (Li et al. 2018).
Table 3: Image motion deblurring results GoPro Nahh et al. 2017) and HIDE (Shen ct 2020) datasets.
SOTS-Indoon SOTS-Outdoor
GoPro
HIDE
Method
PSNRT SSIMT
PSNRT SSIMT
Method
PSNR1
SSIMT
PSNRT
SSIM
DehazeNet
19.82
0.821
24.75
0.927
DeblurGAN-v2
29.55
0.934
26.61
0.875
AOD-Net
20.51
0.816
24.14
0.920
DBGAN
31.10
0.942
28.94
0.915
GridDehazeNet
32.16
0.984
30.86
0,982
DMPHN
31.20
0.940
29.09
0.924
MSBDN
33.67
0.985
33.48
0,982
SPAIR
32.06
0.953
30.29
0,931
FFA-Net
36.39
0.989
33.57
0.984
MIMO-UNet+
32,45
0.957
29.99
0.930
ACER-Net
37.17
0.990
IPT
32.52
DeHamer
36.63
0.988
35.18
0.986
MPRNet
32.66
0.959
30.96
0.939
MAXIM-25
38.11
0.991
34.19
0,985
HlNet
32.71
0.959
30.32
0.932
PMNet
38.41
0.990
34.74
0.985
Restormer
32.92
0.961
31.22
0.942
DehazcFormer-I.
40.05
0.996
Stripformer
33.08
0.962
31.03
0.940
SENet
41.24
0.996
40.05
0.996
SFNet
33.27
0.963
31.10
0.941
Table 4: Image dehazing re- sults on Dense-Haze dataset (Ancuti et al., 2019).
Table 5: Image motion deblur- ring results on RSBlur dataset (Rim et al.] (202)).
Table 6: Image desnowing re- sults on CSD (2000) dataset
(Chen et al. 2021b).
Method
PSNRT SSIM
Method
PSNR
SSIM
Method
PSNR1 SSIM
DehazeNet
9.48
0.438
32.53
0.840
DesnowNet
20.13
0.81
GridDehazeNet
14.96
0.533
MIMO-UNet+
33.37
0.856
JSTASR
27.96
0.88
FFA-Net
12.22
0.444
MPRNet
33.61
0.861
HDCW-Net
29.06
0.91
MSBDN
15.13
0.555
Restormer
33.69
0.863
TransWeather
31.76
0.93
DeHamer
16.62
0.560
Uformer-B
33.98
0.866
MSP-Former
33.75
0.96
SFNet
17.46
0.578
SFNet
34.35
0.872
SFNet
38.41
0.99
Image dehazing results. We perform dehazing experiments on the synthetic benchmark RESIDE (Li et al. 2018) and real hazy dataset Nense-Haze (Ancuti et al., 2019). For RESIDE, train our models for the indoor and outdoor scenarios separately, and then evaluate on the corresponding SOTS-Outdoor SOTS- and SOTS- Outdoor testsets. The quantitative results are shown in Tab. Our method
obtains the highest scores on all metrics. Particularly on the outdoor scene, our network generates substantial gain of 4.87 dB PSNR over DeHamer (Guo et al.] 2022). Compared to the recent works, DehazeFFommer-L (Song et al. 2022) and PMNet (Ye et al. 2022). our method receives 0.19 dB and 2.83 dB higher PSNR SOTS-Indoor testset, respectively. Additionally, performance of our approach on the real hazy dataset Dense-Haze (Ancuti et al. 2019). The results are shown in Tab. [ As we can see, SFNet exhibits the superior ability of dealing with the real- world dehazing problem, receiving gain of 0.84 dB over DeHamer (Guo et al.) 2022). The dehazed results in Fig. iillustrate that SFNet more effective removing haze than other methods.
validate the
Image motion deblurring results. We evaluate our method on both the synthetic and real-world datasets. The numerical comparisons on the synthetic GoPro (Nah et al.) (017) and HIDE (Shen) et al. 2020) datasets are reported in Tab. 3 On GoPro, SFNet shows 0.35 dB PSNR performance improvement over Restormer (Zamir et al., 2022 with ~3x faster inference speed (Tab. 8). More- over, our method receives performance gain of 0.19 dB compared the recent algorithm Strip- former (Tsai et al., 2022) with 34% fewer parameters and 26% fewer FLOPs. Notably, our SFNet
PSNR Referenee
20 Impul
23.53 dB DBGAN
24.01 dB DMPHN
21.41 dB Sain at.
Blurry Image
23.15 MPRNet
23.55 cBB MAXIM
21.92 JB Stripformer
25.75 SENet
Restomeer
Figure 4: Image motion deblurring results on the GoPro dataset NNah et al. 2017

---

Published as a conference paper at ICLR 2023
Table 7: Deraining comparisons with previous methods on five deraining datasets: Rain 100H (Yang
al. 2017). Rain100L Yang al. 2017). Test100 (Zhang al. 2019a). Test1200 Zhang & Patel
2018 8 and Test2800 (Fu et al. (017))
Test100
Rain100H
Rain100L
Test2800
Test
Average
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
DerainNet
22.77
0.810
14.92
0.592
27.03
0.884
24.31
0.861
23.38
0.835
22.48
0.796
SEMI
22.35
0.788
16.56
0.486
25.03
0.842
24.43
0.782
26.05
0.822
22.88
0.744
DIDMDN
22.56
0.818
17.35
0.524
25.23
0.741
28.13
0.867
29.65
0.901
24.58
0.770
UMRL
24.41
0.829
26.01
0.832
29.18
0.923
29.97
0.905
30.55
0.910
28.02
0,880
RESCAN
25.00
0.835
26.36
0.786
29.80
0.881
31.29
0.904
30.51
0.882
28.59
0.857
PreNet
24.81
0.851
26.77
0.858
32.44
0,950
31.75
0.916
31.36
0.911
29.42
0.897
MSPEN
27.50
0.876
28.66
0.860
32.40
0.933
32.82
0,930
32.39
0.916
30.75
0,903
MAXIM-2S
31.17
0.922
30.81
0.903
38.06
0.977
33.80
0.943
32.37
0.922
33.24
0.933
SFNet
31.47
0.919
31.90
0.908
38.21
0.974
33.69
0.937
32.55
0.911
33,56
0.929
PSNK Reforesee
13.30 dB
21,01 JSTASR
14.30 DesoowNet
27.59 dB HDCW-Net
32.21 dB
35.24dil
SENet
Figure 5: Image desnowing results the CSD dataset (Chen al. 2021c).
demonstrates stronger generalization capability HIDE dataset than Stripformer on all metrics. In addition to the synthetic datasets, we further evaluate the effectiveness of our network on the real- world dataset. Tab. S shows quantitative comparisons on the newly proposed RSBlur (Rim et al.) 2022) dataset. SFNet sets new state-of-the-art on this dataset, providing a substantial gain of 0.37 dB PSNR over the previous best method Uformer-B (Wang et al. 2022). Fig. illustrates that SFNet produces more visually pleasant result than competing algorithms.
Image desnowing results. We compare our method on the CSD (Chen et al. 2021b) dataset with existing state-of-the-arr methods (Chen et al., 2022a Valanarasu et al. 2022 Chen al.) 2022c). As shown in Tab. [ our framework yields 4.66 dB PSNR improvement over the Transformer
model MSP-Former (Chen et al., 2022c). The visual results in Fig. show that method is more effective in removing spatially varying snowflakes than competitors. More results on SRRS (Chen) al. 2020a) and Snow 100K et. 2018) are provided in Appendix E.
Image deraining results. Following recent works (Jiang et al.) 2020 Purohit et al. 2021 Tu et al.
2022), we compare PSNR/SSIM scores on the channel YCbCr color space. Tab, 7 shows that our method achieves the best performance on the average PSNR category compared to competing approaches. Moreover, on the Test100 100 dataset (Zhang al. 2019a), the proposed SFNet obtains performance boost of 0.30 dB PSNR over MLP model MAXIM-2S (Tu et al. (2022). Visual results shown Fig. Gillsstrate that our model recovers more fine details without artifacts.
Computational overhead compar- isons. In Tab. X we evaluate the com- putational costs of five motion deblur- ring methods on the GoPro testset (Nah al. 2017). Evaluated on the full- resolution image, our method achieves fastest speed than other state-of- of-the-art algorithms while achieving comparable performance with fewer parameters.
Table 8: Overall comparisons between motion deblurring methods the GoPro (Nah al. 2017) set.
Method
NAFNet MPRNet DeepRFT+ Restormer
SFNet
PSNR
33.63
32.66
33.23
32.92
33.27
Time/s
0.833
1.148
0.806
1.218
0.408
Params/M
67.8
20.1
23.0
26.13
13.27
FLOPs/B
63.33
777.01
187.04
140.99
125.43
PSNR
10 Input
19:41 RESCAN
21 51 PreNee
20 70 dR UMRL
24 29 dR MAXIM
26.48 SENNet
Reference
Figure 6: Image deraining results on the Rain 100H dataset
Yang et al.) 2017

---

Published as a conference paper at ICLR 2023
Table 9: Ablation studies for individual
Table 10:
Ablation study for the
Table 1I: Ablation
proposed modules.
number of MCSE
studies for MDSE
Method
PSNR
Params (M)
FLOPs (G)
Number
PSNR
Params
FLOPs
Group
PSNR
Baseline 31.20
6.90
66.32
2
31.22
6.91
66.34
2
31.40
MDSF 31.42
7.04
66.55
4
31.33
6.91
66.35
4
31.40
MCSF 31.45
6.92
66.38
31,42
6.92
66.37
8
31.42
Full
31.68
7.05
66.61
31.45
6.92
66.38
16
31.30
4.3
ABLATION STUDIES
In this section, we first demonstrate the effectiveness of the proposed modules, and then investigate the effects of different designs for each module. Finally, we delve into the mechanism of MDCF to demonstrate its validity. Following the recent method (Tu et al. 2022), all models are trained on the GoPro (Nah et al. 2017) dataset for 1000 epochs, and N is set to 7 Fig.
Influence of each module. Tab. shows that MDSF and MCSF yield performance gains of 0.22 and 0.25 dB the baseline model with few introduced computing burdens. De- ployed only in single position in each scale, MDSF produces the similar performance with MCSF, demonstrating the effectiveness of dy- namic frequency selection mechanism. Fur- thermore, Fig- 7 we plot the statistic dif- ferences between the ground truth and results of three methods on dehazing. With the fre- quency selection mechanism, the statistics of our results are closer to that of ground truth.
Density
0.004
MA00 28
Density
0.4
MA0M
Vp-1
0.3
0.002
0.1
0.000
0.0
3000 6000
-100
-50
Variance difference
Mean difference
Figure 7: Variance (Left) and Mean (Right) differ- ence between the ground truth and input/results of three methods on SOTS-Outdoor testset. Models are trained on OTS dataset (Li et al. 2018).
Design choices for MCDE We study the influence of the number of MCDF in Tab. 10 where MCSF means that we employ the proposed MCSF in last two residual blocks of each ResBlock. As be seen, using more MCSF leads the consistently increasing performance from 31.22 to 31.45 dB PSNR while only introducing 0.01 M parameters and 0.04 FLOPs. Due to its few introduced parameters and low complexity, we insert MCSF in each residual block for frequency learning.
Design choices for MDSF. To understand the impact of the number of groups in MDSF, we test various configurations in Tab. [1] Generally, the increasing number of groups leads higher PSNR, demonstrating the effectiveness of the filter diversity. However, the accuracy saturates at group 16, which is probably caused by overfitting. We finally pick 8 groups for better performance.
Alternatives for MDSF. To examine the advan- tage of our design, we compare our decoupler with several alternatives in Tab. [2] We first substitute the learning-basee based and fixed frequency separation methods for our decoupler. We form Conv method (Tab. 12p) by using strided convo- lution to generate different frequency parts with reduced resolution (Pang et al. 2020). The Oct- (Tab. 12p) version (Chen et al.) (020bb shares the similar idea with Comy, which utilizes down-sampling to reduce network redundancy.
Table 12: Alternatives for MDSF.
Method
PSNR
Params (M)
FLOPs (B)
(a) Conv
31.11
8.65
79.35
(b) OctConv
31.07
6.99
66.86
(c) Gaussian
30.98
7.02
66.81
(d) Wavelet
30.97
7.06
66.33
(c) Window
31.22
6.98
66.73
(T) Local
31.23
7.00
66.34
(g) MDSF
31.42
7.04
66.55
These variants only introduce extra tow-frequency signal to the network. We further utilize fixed separation methods replace the proposed decoupler. Gaussian (Tab. (2pp) and Waveler (Tab. 12
produce the similar results, much lower than our MDSF. Additionally, Waveler needs more parame- ters to deal with its multiple branches.
Since our filter kernel is generated by learning, we further compare our MDSF with two attention approaches to verify the validity of the proposed selection mechanism. Specifically. we utilize the widely used window-based self-atted attentttnioo (Wang et al.) 2022) (Tab. (22b and dynamic convolution (Han et al., 2022: Wu et al. 2019) (Tab. 12 to conduct comparisons. we can see from the table, our method has huge advantages over these methods, demonstrating the effectiveness of MDSF
8

---

Published as a conference paper at ICLR 2023
40-
Variance
Variance
0.04
Iaralomri
baraton-1
0.03
10
107
10 Iterations
Iterations
Teraton 100
10a0
teratoon2
Herallomo
Figure 8: The variance and discrete Fourier transform of the resulting images iteratively impose the produced filters on the image. Left: The low-pass filter. Right: The high-pass pass filter.
Group
Group2
Group4
Groups
Groupó
Group?
Hegh
Low
Figure 9: Discrete Fourier transform results of group-wise features generated by our MDSF decou- pler. The results are sampled from the last decoder. Top: High-frequency. Bottom: Low-frequency.
4.4 QUALITATIVE ANALYSES MDSF
We provide qualitative analyses of MDSF based on discrete Fourier transform. Results obtained from the branch of 3 3 filter. The input image is sampled from GoPro (Nah et al. 2017). shown in Fig.
TO
The features are obtained from the last residual block in the last ResBlock of decoder.
We first verify the properties of the alleged low-/high- hhtrhhigg-pssssss filters in MDSF. To this end, we iteratively apply the produced filters to the image. The variance and corresponding spectral features of inter- mediate images are provided Fig. 8 Taking the low-pass filter as an example, with the increasing of iteration times, the variance of the image decreases constantly, and the high-frequency signals in spectral features are reduced drastically. The high-pass filter exhibits the opposite properties. These results demonstrate the effectiveness our filters. remarkable that the high-pass filter produces large variance with fewer iterations, hence it is more effective than the low-pass filter. As result, it easy for MDSF introduce more high- high-frequency signals into the network for reconstruction.
In MDSF, we generate different filters for each group enhance the diversity of frequency features. To delve into this mechanism, we visualize the group-wise spectral features in Fig. [9] As expected, different groups focus on the learning of disparate low-/high- freuuerreunncy signals, enriching the di- versity frequency representations
selection. We further compare the feature maps before and after our MDSF in Fig. TO Using the attained filters, the decoupler of MDSF produces different frequency components. The high-frequency feature contains much edge information. The resulting feature after modulator recovers more details of the number plate that is blurry in the initial feature.
CONCLUSION
We present an image restoration framework, SFNet, which built on frequency selection mecha- nism. We develop two key modules, MDSF and MCSF, to conduct frequency decomposition and recalibration with different receptive fields. Specifically, our multi-branch dynamic selective fre- quency module (MDSF) builds dynamic filter to decompose feature maps into various frequency parts and utilizes channel attention to perform accentuation, thus effectively selecting the most infor- mative frequency to recover. Furthermore, the proposed multi-branch compact selective frequency module (MCSF) introduces simple yet effective manner to enlarge the receptive field and conduct frequency selection. With both designed modules, SFNet achieves state-of-the-ar results on five image restoration tasks, demonstrating the validity of our frequency selection mechanism.
Bluny Image
Intial Pestume
Lom-trqquncy Feature
High-Iroqunny Frestune
MOSF
Stapp Image
Figure 10: The internal features of MDSF. With our frequency selection mechanism, MDSF pro- duces more fine details than the initial feature, e.g., the number plate. Zoom in for the best view.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation China under Grant (62172409). Shenzhen Science and Technology Program (JCYJ202208 (111122220011102000110 8102012025, JCYI2002205331145000022)2, and Beijing Nova Program (Z201 100006820074).
REFERENCES
Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixel data. In European Conference on Computer Vision, pp. 111-126. Springer, 2020.
C. O. Ancuti, C. Ancuti, M. Sbert, and R. Timofte. Dense haze: A benchmark for image dehazing with dense-haze and haze-free images, In arXiv, 2019.
Daniela Calvetti, Lothar Reichel, and Qin Zhang, Iterative solution methods for large linear discrete ill-posed problems. In Applied and computational control, signals, and circuits, pp. 313-367. Springer, 1999.
H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng. Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In Computer Vision and Puttern Recognition, 2021a.
Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European Conference Computer Vision, 2022a.
Sixiang Chen, Tian Ye, Yun Liu, Erkang Chen, Jun Shi, and Jingchun Zhou. Snowformer: Scale-aware transformer via context interaction for single image desnowing. arXiv preprint arXiv:2208. 097003 2022b.
Sixiang Chen, Tian Ye, Yun Liu, Taodong Liao, Yi Ye, and Erkang Chen. Msp-former: Multi-scale projection transformer for single image desnowing. arXiv preprint arXiv:2207.0561 2022c.
W. T. Chen, H. Y. Fang, C. Hsieh, C. C. Tsai, I. H. Chen, J. J. Ding, and S. Y. Kuo. All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet repre- sentation and contradict channel loss. In International Conference on Computer Vision. 2021b.
Wei-Ting Chen, Hao- Yu Fang, Jian-Jiun Ding, Cheng-Che Tsai, and Yen Kuo. Jstasr: Joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal. In European Conference on Computer Vision, pp. 754-770. Springer, 2020a.
Wei-Ting Chen, Hao-Y Yu Fang, Cheng-Lin Hsieh, Cheng-Che Tsai, Chen, Jian-Jiun Ding, Sy-Yen Kuo, et al. All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4196 4205, 2021c
Y. Chen, H. Fan, Xu, Z. Yan, Y. Kalantidis, M. Rohrbach, S. Yan, and J. Feng. Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2020b.
Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Geung-Won Jung, and Sung-Jea Ko. Rethinking coarse- to-fine approach in single image deblurring. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 4641 4650, October 2021.
Yuning Cui, Yi Tao, Wenqi Ren, and Alois Knoll. Dual-domain attention for image deblurring. In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuar Yang- Multi-scale boosted dehazing network with dense feature fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2157-2167, 2020.
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via a deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
10

---

Published as a conference paper at ICLR 2023
Chun-Le Guo, Qixin Yan, Saeed Anwar, Rummin Cong, Wenqi Ren, and Chongyi Li. Image dehaz- ing transformer with transmission-awee aware 3d position embedding. In Proceedings the Conference Computer Vision and Pattern Recognition, pp. 5812-5820, 2022.
the IEEE/CVF
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. On the connection between local attention and dynamic depth-wise convolution. In International Conference on Learning Representations, 2021.
loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In orLR.org, 2015.
Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8346-8355, 2020.
Ali Karaali and Claudio Rosito Jung. Edge-based defocus blur estimation with adaptive scale selec-
tion. IEEE Transactions Image Processing, 27(3):1226-113 122-11377 2017.
Deepa Kundur and Dimitrios Hatzinakos. Blind image deconvolution. IEEE signal processing magazine, 13(3):33-64 1996.
B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang. Benchmarking single image dehazing and beyond. IEEE Transactions on Image Processing, 1P:1-1, 2018.
X. Li, W. Wang, X. Hu, and Yang. Selective kernel networks. In 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 510-519, 2019.
J. Liang, J. Cao, G. Sun, K. Zhang, and R. Timofte. former. IEEE, 2021.
Swinir: Image restoration using swin trans-
Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-based multi- scale network for image dehazing. In Proceedings of the IEEE/CVF international conference computer vision, pp. 7314-7323, 2019.
Yun-Fu Liu, Da- Jaw, Shih-Chia Huang, and Jenq-Neng Hwang. Desnownet: Context-aware deep network for snow removal. IEEE Transactions on Image Processing, 2766)3304-33733, 2018. doi: 10.1109/T1P200182286622
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608, 03983, 2016.
Pinjun Luo, Guoqiang Xiao, Xinbo Gao, and Song Wu. Lkd-net: Large kernel convolution network for single image dehazing. arXiv preprint arXiv:2209.0/78 2022.
Xintian Mao, Yiming Liu, Wei Shen, Qingli Li, and Yan Wang. Deep residual fourier transformation for single image deblurring, arXiv preprint arXiv:2111.117455 2021.
S. Nah, T. H. Kim, and K. M. Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Computer Vision and Pattern Recognition, 2017.
Yingxue Pang, Xin Li, Xin Jin, Yaojun Wu, Jianzhao Liu, Sen Liu, and Zhibo Chen. Fan: Frequency aggregation network for real image super-resolution. In European Conference on Computer Vi- sion, pp. 468-483. Springer, 2020.
Kuldeep Purohit, Maitreya Suin, AN Rajagopalan, and Vishnu Naresh Boddeti. Spatially-adaptive image restoration using distortion guided networks. In Proceedings of the IEEE/CVE Interna- fional Conference on Computer Vision, pp. 2309-2319, 2021.

---

Published as a conference paper at ICLR 2023
Wenqi Ren, Si Liu, Hua Zhang, Jinshan Pan, Xiaochun Cao, and Ming- Miug-Hsuan Yang. Single image dehazing via multi-scale convolutional neural networks. In European Conference on Computer Vision, 2016.
Jaesung Rim, Geonung Kim, Jungeon Kim, Junyong Lee, Seungyong Lee, and Sunghyun Cho. Realistic blur synthesis for learning image deblurring. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-24-4 241. Springer. 2015.
Lingyan Ruan, Bin Chen, Jizhou Li, and Miuling Lam. Learning deblur using light field generated and real defocus images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 16304-16313, 2022.
Ivan W Selesnick, Richard G Baraniuk, and Nick C Kingsbury, The dual-tree complex wavelet transform. IEEE signal processing magazine, 22(6):123-11))) 2005.
M Ibrahim Sezan and Murat Tekalp. Survey of recent developments in digital image restoration. Optical Engineering, 29(5):393 404, 1990.
Z. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao. Human-aware motion deblurring.
In International Conference on Computer Vision, 2020.
Hyeongseok Son, Junyong Lee, Sunghyun Cho, and Seungyong Lee. Single image defocus deblur- ring using kernel-sharing parallel atrous convolutions. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp.
2021.
Yuda Song, Zhuqing He, Hui Qian, and Xin Du. Vision transformers for single image dehazing. arXiv preprint arXiv:2204.03883, 2022.
Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. Spatially-attentive patch- patch-hierarchical net- work for adaptive motion deblurring, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3606-3615, 2020.
Fu-Jen Tsai, Yan-Tsung Peng, Yen- Yu Lin, Chung-Chi Tsai, and Chia-Wen Lin. Stripformer: Strip transformer for fast image deblurring. In ECCV, 2022.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mip for image processing. CVPR, 2022.
Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M Patel. Transweather: Transformer- restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2353-2363, 2022.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need. In arXiv, 2017.
Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson W.H. Lau. Spatial attentive single-image deraining with a high quality real rain dataset. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2019.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: general u-shaped transformer for image restoration. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 17683-17693, 2022.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.104:4 2019.
Hao-Hsiang Yang and Yanwei Fu. Wavelet u-net and the chromatic adaptation transform for single image dehazing. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 2736- 2740. IEFE, 2019.
12

---

Published as a conference paper at ICLR 2023
Hao-Hsiang Yang, Chao-Han Huck Yang, and Yi-Chang James Tsai. Y-net: Multi-scale feature aggregation network with wavelet structure similarity loss function for single image dehazing. In ICASSP 2020-2020 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 1628-26322 IEEE, 2020a.
Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). July 2017.
Wenhan Yang, Robby T Tan, Shiqi Wang, Yuming Fang. and Jiaying Liu. Single image derain- ing: From model-based to data-driven and beyond. IEEE Transactions on pattern analysis and machine intelligence, 43(1 1:4459-44077 2020b.
Tian Ye, Mingchao Jiang, Yunchen Zhang, Liang Chen, Erkang Chen, Pen Chen, and Zhiyong Lu. Perceiving and modeling density is all you need for image dehazing. In European Conference on Computer Vision, 2022.
Jaeyoung Yoo, Sang-ho ho Lee, and Nojun Kwak. Image restoration by estimating frequency distribu- tion of local patches. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Yuan Yuan, Wei Su, and Dandan Ma. Efficient dynamic scene deblurring using spatially variant deconvolution network with optical flow guided training. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 3555-3564, 2020.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14821-14831, 2021.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-
Hsuan Yang. Restormer: Efficient transformer for nigh-ressoluionn image restoration. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728-5739, 2022.
He Zhang and Vishal M. Patel. Density-aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
He Zhang, Vishwanath Sindagi, and Vishal M Patel. Multi-scale single image dehazing using per- ceptual pyramid deep network, In The IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR) Workshops, 2018.
He Zhang, Vishwanath Sindagi. and Vishal M Patel. Image de-raining using conditional generative adversarial network. IEEE transactions on circuits and systems for video technology, 30(1 1): 3943-3956, 2019a.
Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi- patch network for image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5978-5986, 2019b.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions image processing, 26 (7):3142-115) 2017.
Kaihao Zhang, Wenqi Ren, Wenhan Luo, Wei-Sheng Lai, Björ Stenger, Ming-Hsuan Yang, and Hongdong Li. Deep image deblurring: A survey. International Journal of Computer Vision, 130 (9):2103-2130, 2022.
Wenbin Zou, Mingchao Jiang, Yunchen Zhang, Liang Chen, Zhiyong Lu, and Yi Wu. Sdwnet: A straight dilated network with wavelet transformation for image deblurring. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 1895-1904, 2021.
13


Reviews:
[{"review_id": "kExitN9vdm5", "reviewer": "Reviewer_BBSS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author proposes a Selective Frequency Network (SFNet) for image restoration. In SFNet, the author designs a multi-branch and content-aware module to decompose the feature into separate frequency sub-bands, and then uses the channel-wise attention mechanism to emphasize the useful information. In addition, to cope with the large-scale degradation kernel, the author proposes an decoupling and modulation module to enlarge the receptive field based on global and window-based average pooling.", "summary_of_the_review": "The paper is well presented and claims better performance on different tasks, while there are some problems in theoretical proof and experimental part.", "strength_and_weaknesses": "The paper is well presented and claims better performance on different tasks, while there are some problems in theoretical proof and experimental part.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cHX0gHUERA-", "reviewer": "Reviewer_4vXg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on image restoration and introduces a multi-branch dynamic selective frequency (MDSF) module with learned kernels to generate high and low-frequency features and a multi-branch compact selective frequency (MCSF) module to enhance the features with an attention mechanism. The proposed method achieves several state-of-the-art performances for image restoration tasks.", "summary_of_the_review": "The paper achieves good results for different image restoration tasks. However, the paper lacks some validation of the proposed method and comparison to more recent works. Also, the paper quality needs to be improved.\n\n", "strength_and_weaknesses": "Strength\n-The paper works on frequency decomposition and manipulation design, which is important for image restoration.\n-The proposed method outperforms many state-of-the-art methods on different image restoration tasks.\n\nWeakness\n-Comparison with other attention mechanisms. The proposed MDSF and MCSF are similar to the multi-brach pathways with attention mechanisms. Since the kernel weights are learned, it is hard to tell whether the learned dynamic frequency decomposition benefits more than attention. The authors can replace the modules in Fig 2. (b) with other attention forms to verify the argument.\n-Analysis of the high- and low-frequency features. The authors should provide more analysis of the obtained features of MSDF to verify the high- and low-frequency properties and frequency diversity.\n-Ablation studies. The authors conduct ablation studies with different settings. It is hard to compare across different components. The authors should give a more unified ablation.\n-Comparison to state-of-the-art. The authors should include more recent works, such as NAFNet for image debarring, in the comparison. It is also not obvious whether the proposed modules or training strategies contribute more to the final results.  Liangyu Chen, et al. \"Simple baselines for image restoration.\" arXiv preprint arXiv:2204.04676 (2022).\n-The authors can include other related works, e.g., Jaeyoung Yoo, Sang-ho Lee, and Nojun Kwak. \"Image restoration by estimating frequency distribution of local patches.\" CVPR 2018\n-The paper quality needs to be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NcrOFTCvxN", "reviewer": "Reviewer_Qxbc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new deep model framework named Selective Frequency Network (SFN) for multiple image restoration tasks, including single image defocus deblurring, image dehazing, image motion deblurring, image desnowing and image deraining. SFN builds on two types of basic blocks, i.e., Multi-branch Dynamic Selective Frequency module (MDSF) and Multi-branch Compact Selective Frequency module (MCSF). MDSF is proposed to decompose the feature into frequency subbands dynamically and locally for accentuating the effective ones further, while MCSF is designed by using global and window-based average pooling to deal with the large-scale degradation kernel. The proposed SFN exploits a U-type structure to assemble the modules of MDSF and MCSF. Extensive experiments are carried out to validate the superiority of SFN and the effectiveness of block components MDSF and MCSF. ", "summary_of_the_review": "The paper is novel enough for its proposed modules MDSF and MCSF as well as SFN established with them. The content is presented clearly and organized reasonably, and the technique described could be reproduced easily. Lastly, the writing is good and fluent.", "strength_and_weaknesses": "The paper pays attention to the large discrepancy of sharp and degraded image pairs in the frequency domain, and proposes the modules MDSF and MCSF which are used to build SFN net. It is novel in terms of method. The structure of the proposed SFN is clearly presented in the text, and more details are further discussed in Appendex A-H. The content of the paper is very sufficient. The proposed method is validated by extensive experiments to be effective on many image restoration tasks, and perform favorably compared with lots of SOTS methods on various datasets. In addition, ablation studies are made to explain the design of the modules MDSF and MCSF.\n\n1.What are the advantages of the “Shallow Layer and output layer” over existing multi-scale methods? \n2.Why MDSF and MCSF are concatenated in Figure 1? These two modules are used independently, right?\n3.How to control the kernel size of different decouplers in MDSF? From figure 2(b), it’s seems that all the kernel size are k^2?\n4. There are still some typos in the paper:\n(1) In the last paragraph of Section 2, both “Xu et al.” and “Mao et al.” lack reference numbers.\n(2) In Figure 2(b), the output of MDSF is the concatenation of the low-frequency feature and the high-frequency feature, while in Figure 2(c), MCSF receives two inputs, i.e., the low-frequency feature and the high-frequency feature from MDSF. I wonder if the concatenation operation in Figure 2(c) is necessary.\n(3) The parameter c in equations (2) and (4) takes a real value as in c \\in R^{C_i} and c\\in R^{2C}. However, c is the index of a channel according to the context of the paper, and could take an integer value.\n(4) Section 3.2.1 could be modified to Section 3.3.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BQKAl4DrlB", "reviewer": "Reviewer_2iAU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on image restoration tasks in the frequency domain. Specially, a multi-branch dynamic selective frequency module is employed to select the most informative frequency to recover. Besides, the authors propose a multi-branch compact selective frequency module to enlarge the receptive field. Experimental results demonstrate that the network utilizing these two modules performs better than state-of-the-art methods on different image restoration tasks.", "summary_of_the_review": "I tend to accept this paper based on its solid contributions and novelty.", "strength_and_weaknesses": "Strengths:\n1. The authors design a useful module to decouple the feature map into different parts and select the most important ones for restoration.\n2. The network architecture is well-introduced and it is easy to reproduce the main results.\n3. The experiments show that the proposed network achieves state-of-the-art results on five image restoration tasks. The supplementary material shows more results to demonstrate the effectiveness of the proposed method. \n\nWeaknesses:\n1. It will be better if the authors can integrate and simplify Figures 1-3 properly. \n2. How to switch the high-pass and low-pass filters in the Filter Switcher automatically?\n3. More visual results or the corresponding PSNRs can be added in Figure 5 to show the superior of the proposed network. Besides, the typesetting of Figure 5 should be modified. If the space is limited, try to delete some compared results in Table1-7.\n4. Are the ablation studies all about the deblurring task? If so, add this explanation. \n5. The FLOPs should be added in Table 10 to support that more MCDF modules bring few parameters and complexities.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster