Improving the Learning Capability of Small-size Image Restoration Network by Deep Fourier Shifting
Authors: Man Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: Image restoration
Primary area: machine_vision
TL;DR: Improving the Learning Capability of Small-size Image Restoration Network by Deep Fourier Shifting Operator over Resource-limited Scene

Abstract: State-of-the-art image restoration methods currently face challenges in terms of computational requirements and performance, making them impractical for deployment on edge devices such as phones and resource-limited devices. As a result, there is a need to develop alternative solutions with efficient designs that can achieve comparable performance to transformer or large-kernel methods. This motivates our research to explore techniques for improving the capability of small-size image restoration standing on the success secret of large receptive filed.

Targeting at expanding receptive filed, spatial-shift operator tailored for efficient spatial communication and  has achieved remarkable advances in high-level image classification tasks, like $S^2$-MLP and ShiftVit. However, its potential has rarely been explored in low-level image restoration tasks. The underlying reason behind this obstacle is that image restoration is sensitive to the spatial shift that occurs due to severe region-aware information loss, which exhibits a different behavior from high-level tasks. To address this challenge and unleash the potential of spatial shift for image restoration, we propose an information-lossless shifting operator, i.e., Deep Fourier Shifting, that is customized for image restoration. To develop our proposed operator, we first revisit the principle of shift operator and apply it to the Fourier domain, where the shift operator can be modeled in an information-lossless Fourier cycling manner. Inspired by Fourier cycling, we design two variants of Deep Fourier Shifting, namely the amplitude-phase variant and the real-imaginary variant. These variants are generic operators that can be directly plugged into existing image restoration networks as a drop-in replacement for the standard convolution unit, consuming fewer parameters. Extensive experiments across multiple low-level tasks including image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring demonstrate consistent performance gains obtained by our Deep Fourier Shifting while reducing the computation burden. Additionally, ablation studies verify the robustness of the shift displacement with stable performance improvement.

Improving the Learning Capability of Small-size Image Restoration Network by Deep Fourier Shifting
Man Zhou
Aerospace Information Research Institute, Chinese Academy of Sciences University Science and Technology of China
Abstract
State-of-the-art image restoration methods currently face challenges in terms of computational requirements and performance, making them impractical for de- ployment edge devices such phones and esource-limitee devices. As result, there is a need to develop alternative solutions with efficient designs that can achieve comparable performance to transformer or large-kernel methods. This mo- tivates our research to explore techniques for improving the capability of small-size image restoration standing the success secret of large receptive filed.
Targeting at expanding receptive filed, spatial-shift operator tailored for efficient spatial communication and has achieved remarkable advances in high-level image classification tasks, like 52-MPP [ and SiiftVit (2). However, its potential has rarely been explored in low-level image restoration tasks. The underlying reason behind this obstacle is that image restoration is sensitive the spatial shift that occurs due to severe region-aware information loss, which exhibits different behavior from high-level tasks. To address this challenge and unleash the potential of spatial shift for image restoration, we propose an information-lossless shift- ing operator, i.c., Deep Fourier Shifting, that is customized for image restoration. To develop proposed operator, we first revisit the principle of shift operator and apply it to the Fourier domain, where the shift operator can be modeled informmtiionllosleee Fourier cycling manner. Inspired by Fourier cycling, we design two variants of Deep Fourier Shifting, namely the amplitude-phas variant and the real real-imaginany variant. These variants generic operators that be directly plugged into existing image restoration networks as a drop-in replacement for the standard convolution unit, consuming fewer parameters. Extensive experi- ments across multiple low-level tasks including image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring demonstrate consistent performance gains obtained by our Deep Fourier Shifting while reducing the computation burden. Additionally, ablation studies verify the robustness of the shift displacement with stable performance improvement.
1 Introduction
The spatial shift operator 33 is a technique that shifts channels from a pixel to its adjacent pixels. is celebrated for its efficient facilitation of spatial information exchange. Due to its parameter-fin free nature and computational efficiency, this operator has found widespread application as a substitute for standard convolution units, particularly in high-level image classification tasks. representative work, 22- 52-MLPP D. integrates the spatial shift operator into a ccannel-mixing MLP framework. This alternative to token-minn moken-mixing MLPs, effectively mitigating intrinsic overfitting
approach serves issues and significantly enhancing recognition accuracy. Shift [2] explores the role of the self- attention mechanism in Vision Transformers (ViTs) for high-level tasks. It proposes a replacement using modified spatial shift operation 4 that facilitates the exchange of a subset of channels
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

fill 0
38.5
Original Fcycle-ab
Shift-sa
58.0
W
discard
37.5
H
group
nsm0
ns
mmmm
ns512
ns=18
(a) Spatial shift Operator
(c) with varying numbers (ns)
cycling theory
24
Original Fcycle-ab
Shrit-sa
23
22
group
DFT
F Fourier
IDFT
21
cycling
20
19
19.0
9.5
Sec. 3.2 Fcycle / Feycle-ab
n=0
n=1
nF2
[b) Fourier Shift/Cycling Operator
with varying displacement
Figure 1: Comparison between the spatial shift operator and the proposed deep Fourier shift operator. (a) Traditional spatial shift operator involves spatial shift mechanism that moves each channel of the input tensor in a distinct spatial direction, thus suffering from severe region-aware information loss and conflicting with the requirements of image restoration tasks. (b) Deep Fourier Shifting/Cyeee /Cyclig operator is a more ingenious information-lossles operator, which is tailored for image restoration tasks. (c), (d) Deep Fourier shifting achieves more stable performance gain than the spatial shifting mechanism with varying "ns" shift displacements and "n" basic units over image de-noising task where the cut-off is for compressing the vertical axis scale better illustrate the contrast effect clearly.
between neighboring features. Despite these impressive achievements, the utility of spatial shift operators in low-level image restoration tasks remains unexplored.
The main difficulty in introducing the spatial shift mechanism to low-level vision tasks lies in its inherent conflict with the objectives of image restoration tasks, as depicted in Figure | In particular, this mechanism operates by moving each channel of its input tensor in distinct spatial directions, leading to mixing of spatial information across channels. Notably, image restoration is fundamentally standard regression problem where both features and channels play critical roles in determining the final output. This contrasts with the behavior observed in high-level vision tasks. A key limitation the spatial shift operator its inherent loss region-aware information, where the regions subject to shifting are filled with zero values. Consequently, image restoration tasks, which are particularly sensitive this kind information loss, can experience decline performance due to spatial shifting.
In response to these challenges. we introduce a novel deep shifting operator specificallyy crafted for low-level image restoration tasks, named Deep Fourier Shifting. This operator revisits and enhances the fundamental principles of the traditional shift operator by extending its application into the Fourier domain. Here, the shift operation is reformulated as an information-llosless process executed through Fourier cycling. Deep Fourier Shifting composed of two distinct variants, each featuring three key elements: a 2D discrete Fourier transform, Fourier cycling rules, and 2D inverse Fourier transform. This operator designed to be generic, and seamlessly integrable into existing image restoration architectures as replacement for standard convolution units, offering the added advantage of reduced parameter usage. To assess the effectiveness of Deep Fourier Shifting, undertake comprehensive experiments across spectrum of low-level image restoration tasks. These include image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring Our experiments consistently reveal performance improvements achieved through the integration of Deep Fourier Shifting, while alleviating computational burden. Furthermore, conduct ablation studies to evaluate the robustness of Deep Fourier Shifting against varying shift displacements. These studies demonstrate its ability to consistently enhance performance, underscoring its stability and effectiveness.
Our results suggest that Deep Fourier Shifting a promising tool for image restoration, showing potential for varied real-worke applications. We hope that Deep Fourier Shifting could contribute

---

Row
2
H
B
Row
IDFT
Row
DFT group
Row
thecretically odjacent
DFT
IDFT
w
Discreta Time-Domain
W
0
IDFT
Column
Column
Fourier cycling
(b)
Figure 2: (a) The information-lostles, cycling mechanism. The discrete Fourier transform of signal exhibits period-extendee and cycling properties. Specifically, in the Fourier domain, the two pixels in sequence beginning and end may not appear adjacent, but due the period property, they are actually considered adjacent, as indicated by the upper right corner. This inherent period-extended and cycling behavior of the Fourier transform enables that is information-lost lossless,, making it well-suited for image restoration tasks. Consider the Fourier transform of discrete time-domain signal, represented as ( 3). may appear that the values and 5 are not adjacent within the main period. However, owing to the property of period extension, the 3 from the previous period and the from the current period are theoretically considered adjacent. It is reasonable to move the removed area from the end to the front, meeting the cycling mechanism. (b) Our deep Fourier shifting operator. Our operator borrows the principle of the spatial shifting mechanism and models the shifting mechanism in information-lossles Fourier cycling rules. The cycling is coded as 2D queue rolling.
model the shifting mechanism in manner
advancements in neural network designs for image restoration, particularly in improving spatial communication interactions.
Deep Fourier Shifting
Building the principle of the shift operator, extend its application the Fourier domain. In this domain, the shift operator is conceptualized an information-loosless operation, which we term Fourier cycling. To substantiate this approach, we present theorem along with its corresponding proof. Additionally, we introduce two distinct variants of Deep Fourier shifting: i) the magnitude and phase variant, and ii) the real and imaginary variant, which are derived from the transformation rules we have identified within the Fourier domain.
Definitions. f(x, y) Hxxxx is the spatial signal and F(u, v) Hxx denotes its Fourier transform where (I, y) and (u, u) represent the space coordinates and Fourier spectrum, respectively.
Theorem. The Fourier transform of discrete signal is period-extendel and cycling: F(u, v) F(u + nH, v) = F(u,u v + mW) = F(u + nH, u mW) where u = 0, 1, H 1, 0,1,2 W and n, N. N is the of positive integers starting from zero.
2.1 Proof: The Fourier Transform of Discrete Signal is Period-Extended and Cycling
We show the periodicity and eycling properties of the Fourier transform of discrete signal, as illustrated Figure 2 a). Note that the Fourier transform Flu, v) of f(z y) is expressed as
H-1 W-I-11
F(u,v) v)
1
Σ
f(, 12 -2om W)
(I)
HW
4=0
Then, we show the periodicity of F(u,u) v) E R with H and W. means F(u, v) F(u nH, v) F(u,v + mW) F(u nH, + mW) where 1, 2,

---

= 0,1,2..... W - and m E N that records the set of non non-negative integers. We take the v) =F(u + nH, mW) for example and recall Eq.
F(u nH, + mW)
f(x 52x( (utalla (++
HW
(2)
H-1W-1
HW E
Σ
2/am f(2, 12 e22mm 2jan
F(u, v)e 2jam 2580
where for any integer 2, it has 2j 1.
Further, e 2jan and e 2jam = 1 for m E N. Therefore,
F(u - nH, v + mW) = F(u, oje 2jam 2jan F(u,)))
(3)
Similarly, we can prove the periodicity of Flu, v) as well
F(u, = Fu + nH, v mW)
(4)
= nH, u
Furthermore, deep Fourier transform can be expressed in Cartesian and polar coordinates by an equivalent form as
F(u, v) A =
(5)
The period-cxtended and eycling property holds over the amplitudo-phase and real-iraaaeeee aall-maagn format.
2.2 Architectural Design
Recall the Theorem-1, we propose two deep Fourier shifting variants: amplitude-phan phase variant and real-imaginary variant.
Amplitude-phase shifting variant. This variant illustrated in Figure bb. The pseudo-code shown in Figure 3 (left). Given image X E R Wxc we first adopt the Fourier transform FFT (X) to obtain its amplitude component A and phase component P. We then evenly split the generated and P in 4 folds A_g and P_g by the channel dimension, perform the Fourier cycling over A_g and P P_g in both the H and W dimensions:
A_g [0] rorch.roll(A_ggg.. dim-1,
A_g [1] torch. borch.rollAA. ro.1((_g111]] dim- -8)
(6)
A_g [2] vorch.roll(A_g[11. dim=2, s)
A_g[3] [3] torch.roll(A_g[0) dim-2, -s);
where torch.rolle (.) accounts for the cycling function by dim parameter for shifting dimension and for shifting displacement. The transformed A_g and P_g are then fed into two independent convolution modules with 1 x kernel and followed by the inverse Fourier transform iFFT(.) to project the shifting ones back spatial domain.
Real-imaginary shifting variant. The pseudo-code for the real-imaginary shifting variant is pre- sented the right side of Figure 3 In this variant, we perform Fourier cycling separately on the real component a and the imaginary component b, while keeping the remaining processing steps the shifting variant.
same as the amplitude-pha
Concerning two variants, the first one entails trigonometric function calculations, wherein minor numerical alterations could potentially result in computational instability in engineering applications. However, it offers more accurate physical interpretations over amplitude-phase operation from signal processing perspective.

---

def DFS_AP(X):
def DFS_ab(X)
# X: input with shape Di, C, H, w]
# X: input with shape Dr, C, H, W] and the real and imaginary part a+bj FXT(X)
# A and P the amplitude and phase A.e(jP} FX)
# Fourier shifting transform rules dim=1)
Fourier shifting transform rules a_g torch.spilt(a, 4, dim=1)
A_g torch.
P-g torch.spilt(P.. dim=1)
b_g torch.spilt(b, 4, dim=1)
A_fc Fourier-cycling(4__
a_fc Fourier-cyclingga..
P_fc Fourier-cycling(t
b_fc = Fourier-cycling (b_ggbgg)g)))))
A_fc Convs_ix1 (A_fc)
A_fc = Convs_ixi (a_
P_fc Convs_1xi (P_fc)
P_fc Conva_ixi (b_fc)
# Inverse Fourier transform
Inverse Fourier transform Y iFFT(a_fc, b_fc)
Y - 1FFT(A_fc, P_fc)
Return Y # #[,, C, W]
Return Y [N, C, H,
Figure 3: Pseudo-code the two variants of the proposed deep Fourier shifting, The left the amplitude-phase variant while the right is the real-imaginary variant.
Table 1: Quantitative comparisons on low-light image enhancement. The arrow denotes the generalization setting by training on the data before the arrow and testing directly on the data after the arrow.
LOL
Huawei
Huawei
LOL
Model
Config
#Parus
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Original
19.7931
0.8361
17.7929
0.6247
20.1549
0.6851
18.0856
0.7543
0.55M
Shift-sa
19.7072
0.8343
17.6221
0.6071
20.2165
0.6873
17.9112
0.7532
0.4IM
DRBN
Feycle-AP
22.4274
0.8448
19.3252
0,6472
20.5855
0.6872
18.8666
0.7587
0,41M
Feyele-ab
22.2054
0.8429
19.3125
0.6431
20.6651
0.6876
19.1535
0.7681
0.41M
Original
20.1062
0.7895
16,5874
0.5925
20. 1742
0.6659
18.5468
0.7441
7.76M
Shift-sa 20.0148
0.7911
16.8214
0.5911
20.1517
0.665
18.4998
0.7434
7.53M
SID
Feycle-AP 22.8565
0.8019
19.1707
0.6238
20,9068
0.6708
18.8161
0.7494
7.53M
Feycle-ab
22.6313
0.7995
19.2471
0.6242
20.9271
0.6691
18.5741
0.7443
7.53M
Table 2: Comparisons image denoising.
Table 3: Comparisons on image deblurring.
DNCNN
DeepDeblur
Dataset
Metric
Dataset Metric
Original Shift-sa Feycle-AP Feycle-ab
Original Shift-sa Feycle-AP Feycle-ab
PSNR
37.1992
37.2247
37.6891
38.1837
PSNR
28.9423 28. 9037
29.2123
29.1939
SIDD
GoPro
SSIM
0.8954
0.8980
0.9013
0.9066
SSIM
0.8716
0.8712
0.8777
0.8765
PSNR
38.42
38.69
38.65
PSNR
26.9770
26.9991
27.2860
27.2547
DND
HIDE
SSIM
0.8974
0.8985
0.942
0.949
SSIM
0.8468
0.8476
0.8541
0.8536
#Paras
1.51M
1.43M
1.43M
1.43M
#Paras
11.72M
10.61M
10.61M
10.61M
Experiments
3.1 Experimental Settings
Image enhancement. We evaluate our Fourier shifting operator two popular image enhancement benchmarks: LOL [5] and Huawei [6]. The LOL dataset consists of 500 www-nnormal-ligh image pairs. Following the original setting, use 485 pairs for training and 15 pairs for testing. The Huawei dataset contains 2480 paired images, with 2200 pairs for training and 280 pairs for testing. We compare with the representative approaches SID 7 and DRBN g
Image deblurring and denoising. For the image deblurring task, we employ DeepDeblur 9 in our experiments. We use the GoPro dataset 9 for training. To demonstrate the generalizability of our operator, we also apply the model trained on the GoPro dataset directly to the test images of the HIDE dataset [10]. For image denoising, use the SIDD dataset [ the training benchmark.
5

---

Table 4: Quantitative comparisons guided image super-resolution.
WorldView-I1
Gaol'en2
WorldView-II
Model
Config
PSNR
SSIM
SAM
ERGAS
PSNR
SSIM
SAM
EGAS
PSNR
SSIM
SAM
EGAS
Original
40.8172
0.9630
0.0257
1.0555
42.1699
0.9569
0.0192
0.9565
29.68
0.907
0.085
3,426
Shift-sa
40.8791
0.9631
0.0255
1.0511
42.2107
0.9577
0.0185
0.9549
29.32
0.897
0.103
3.734
PANNET
Fcycle-AP
41,2633
0.9650
0.0242
1.0000
42.6361
0.9680
0.0173
0.9005
30:46
0,915
0.078
3.253
Reycle-ah
41.3184
0.9671
0.0238
1.0032
42.5594
0,9648
0.0185
0.9118
30.55
0.918
0.077
3.187
Original
41.4967
0.9692
0.0232
0.9781
47.1069
0.9883
0.0106
0.5626
30.59
0,924
0.0741
3.0798
Shift-sa
41.5227
0.9695
0.0230
0.9766
47.1371
0.9886
0.0098
0.5617
29.87
0.912
0.082
3.325
MuiNet
Feycle-AP
41.6686
0,9712
0.0228
0,9598
47.4225
0,9890
0.0101
0.5453
30.71
0.926
0.0737
3.0767
Feycle-ab
41.6579
0.9701
0.0228
0.9611
47.3489
0.9887
0.0104
0.5479
30.74
0.926
0.0737
3.0764
Performance evaluation is conducted on the remaining validation samples from the SIDD dataset and the DND benchmark dataset [12].
The selected baseline for comparison is DnCNN LB
Guided image super-resolution. We adopt the pan-sharpening task as a representative task of guided image super-resolution. The WorldView II and GaoFen2 datasets [14] are used in our experiments.
The baselines are state-of-the-art INNformer [13] and SFINet 14]
In evaluating the performance of different approaches, we use image quality assessment metrics such as the relative dimensionless global error in synthesis (ERGAS) [6), the peak signal-to-nois ratio (PSNR), structural similarity index (SSIM), the spectral angle mapper (SAM) [17].
3.2 Implementation Details
Based on the above competitive baselines, we perform the comparison over the following configura- tions by replacing the standard convolution with the spatial or Fourier shifting operator:
1) Original: the baseline without any changes;
2) Fcycle-AP: replacing the original model's standard convolution operator with the amplitude-
phase variant of Deep Fourier shifting;
3) Feycle-ab:
replacing the original model's standard convolution operator with the real- imaginary variant of Deep Fourier shifting;
4) Shift-sa: replacing the variants of Deep Fourier shifting in the settings of 2)/3) with the spatial shifting operator;
5) Shift-ns: replacing "ns" standard convolution operator with the spatial shifting or Fourier
shifting-ab/AP operator;
6) Shift-n:
replacing the spatial shifting or Fourier shifting-at/AP operator in the settings of 5) with varying shifting displacement "n".
3.3 Comparison and Analysis
Quantitative comparison. We evaluate model performance across various configurations detailed in the implementation section (Section 3.2p. The quantitative results of this analysis are systematically presented in Tables I and 4 In these tables, the best and second-best results are highlighted in bold and underlined, respectively. Values that fall below the established baseline are marked with gray background for clear distinction. We observe uniform trend of performance enhancement across all the tasks and datasets tested when our two novel deep Fourier shifting variants are integrated. This improvement is particularly noteworthy it is achieved with reduced computational costs, suggesting the efficiency and effectiveness of our approach. For instance, in the low-light image enhancement baseline SID [.. our "Feycle-AP" and "Feycle-ab" models surpass the "Original" model by achieving higher PSNR values of 2. 2.75dB/2.53dl and 2.6dB/2.7BB on the LOL and Huawei datasets, respectively. In contrast, the integration of the spatial shifting operator into the SID baseline results in a slight reduction in PSNR by 0.1dB/0.02dB on the LOL and Huawei datasets, as evident in the 3rd and 7th rows. These findings validate the superiority of our proposed Fourier shifting approach, demonstrating its suitability and effectiveness in addressing the unique challenges of low-level image restoration tasks. Additionally, the integration of our proposed methods has shown to enhance the training performance, in Figure
We followed the official code reproduce the results https:///ttttee wttp:///....ooec/c//aaaaa/aa

---

(a) Input image
(b) SID
(c) Shift
(d) Feyele-ab
(c) Feycle-AP
(u) Inpul image
(b) DRBN
(c) Shift-sa
(d) Feycle-ab
(c) Feycle-AP
Figure Visual comparison over image enhancement.
Features Ampiitade Components
Deep
1.4
Foycle-AP shallow layer
Foycle-AP
Shift-sa of shallow layer D
Mutual I Information
layer
0.6
0.4
Index
Feature Amplitude Components Shift-sa the Deep
we compare mutual information levels before and after employing Feycle-ab and Shift-sa operators the LOL test set, respectively. Our operator exhibits significantly higher mutual information than Shift-sa, showcasing its efficacy in information preservation. Right: visualize feature maps and their amplitude components before and after operations. This demonstrates that our Fcycle-AP promotes frequency information domain.
Figure 6: The effectiveness information preservation. Left:
23.
Original
22.3
Spatiariaa
Qualitative comparison. Due space con- straints, we present only a subset of the visual results in Figure which effectively illustrate the efficacy of our proposed deep Fourier shifting op- erator. Additional visual results are available in the supplementary materials. As demonstrated in these figures, the integration of the deep Fourier shifting operators (Feycle-ab and Fcycle-AP) with the original baseline models yields results that visually more appealing compared to the baselines and the spatial shift operator (Shift-sa). Models enhanced with our operators exhibit superior ability to restore fine texture details and mitigate degradation effects. In contrast, models using the spatial shift operator tend to produce significant artifacts.
10.0
10.3
Orighne
Prycheat
203
15.9
13.3
landioona 0C Muaoni
Figure 4: The proposed operators improve the training performance. It shows the training PSNR on the image enhancement task on the LOL and Huawei datasets in the top and bottom.
Information preservation. offer deeper un- derstanding of the efficacy of our frequency cy- cling mechanism, we begin by comparing the mu- tual information of features before and after the application of our proposed Fcycle-AP operator the LOL test set. The statistical analysis, illustrated in Figure 6 (left), demonstrates signifi-

---

Feycle-ab
Feycle-AP
Shift-sa
Model
Config
#Paras
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Original
#0
37.1992
0.8954
37.1992
0.8954
37.1992
0.8954
1.51M
#2
37.2309
0.8958
37.3781
0.8983
37.2247
0.8980
1.43M
#4
37.3981
0.8974
37.4832
0.8981
37.4256
0.8994
1.35M
Shift-ns
#6
37.6964
0.9011
37.5891
0.9013
37.5091
0.8993
1.28M
#12
38.0837
0.9064
37.4676
0.9003
37.2996
0.8983
1.06M
#18
38.2810
0.9081
37.5251
0.9044
23.6867
0.3340
0.84M
#1
37.2309
0.8958
37.3781
0.8983
37.2247
0.8980
1.43M
#2
37.2663
0.8961
37.3551
0.8978
37.3379
0.8995
1.43M
Shift-n
#3
37.4277
0.8985
37.3139
0.8974
37.4009
0.8998
1.43M
#4
37.3781
0.8977
37.4754
0.9001
37.1247
0.8909
1.43M
Table 5: Ablation studies of image denoising network, DNCNN.
cantly higher level of mutual information as outlined in 18] for our method when compared to the conventional shifting operation in the spatial domain. This notable increase in mutual information underscores the strength of our Fourier cycling mechanism in preserving information and minimizing information loss.
Moreover, an analysis of the features and their corresponding amplitude components, as depicted in 6 (right), reveals distinct grid effects in the feature maps generated by the Shift-sa operator. In contrast, our method substantially mitigates these grid effects. This achieved by facilitating interactions among frequency information and improving spatial communication. Together, these observations compellingly show the efficiency of our proposed Fourier cycling mechanism in both preserving information and enhancing frequency interaction.
3.4
Ablation Studies
With varying "n² To assess the robustness of our method, we conduct a comparative analy- sis with the image enhancement baseline SID and the image denoising baseline DNCNN. This comparison involved varying the shifting dis- placement "n". The corresponding quantita- tive results presented in Figure 7 and Ta- ble 5 The results clearly show that integrat- ing our proposed Fourier shifting solutions into these baseline models consistently yields better performance than the original baselines alone. Conversely, when spatial shifting is employed within these baselines, there marked de- cline in performance compared to their original versions, particularly larger shifting displace- ments (e.g., "n=4"). These findings not only validate the efficacy of our approach but also highlight its distinct robustness over traditional spatial shifting.
Shift-sa
FcycleAP
Foycleab
23.0
22.0
21.5
21.0
20.5
20.0
1.0
2.0
3.0
4.0
Shifton
19.5
(a) LOL
19.0
10.0
17.5
17.0
16.5
With varying "ns". We examine the robustness of our deep Fourier shifting variants by com- paring them with the image de-noising bascline DNCNN, For this comparison, we substitute the variable "ns" standard convolution units in the baseline with both the spatial shifting operator or our proposed deep Fourier shifting variants, as detailed in Section 3.2 The results of this evaluation are provided in Table We highlighted the best and second-best results in bold and under- line, respectively. visual depiction of these results is provided Figure (c). A key observation from these results is that as "ns" increases, models incorporating proposed operators not only
16.0
Shift-n
(b) Generalization Huawei
Figure 7: The effect of shifting displacement shift-n on SID.

---

exhibit consistent performance improvements but also achieve these enhancements with fewer model parameters. In contrast, models using the spatial shifting operator display a notable decrease in performance, particularly when "ns" is increased to 18. This trend demonstrates the robustness and efficiency our proposed Fourier shifting solutions. To further elucidate this point, values falling below the baseline are marked with gray background, highlighting their relevance and impact.
4 Related Work
Spatial-shifting operator. The spatial-shift operator, initially proposed by Wu et al. []. enhances efficient spatial information communication. has gained traction high-level image classification tasks []] where they innovated upon MLPMixer [99] They replaced the spatial-wise token mixer with a spatial-shift operation, enabling inter-patch communication and effectively addressing overfitting challenges associated with spatial-specific token-mixing MIPs. Similarly, Wang et al. [2] explored the role of attention mechanisms in Vision Transformers (ViTs). They introduced partial shift operation, exchanging subset of channels among adjacent features. This simple yet effective operation led to the development of ShiftViT, a backbone network that replaces traditional attention layers in VITs with shift operations. These approaches illustrate the adaptability and versatility of spatial-shift operators enhancing deep learning models for image analysis.
Deep Fourier transform over image restoration. There has been a growing interest in integrating the Fourier transform refine deep learning- -bsedd image restoration models. notable example is the work of Zhou et al. 20, which re-examines the interplay between spatial and Fourier do- mains. This study uncovers transformation rules applicable to various resolution features within the Fourier domain, leading to the development of theoretically sound Deep Fourier Up-Sampling method applicable across multiple restoration tasks. Similarly, the research in Zhou et al. [[4] investigates the degradation dynamics of guided image super-resolution. The authors propose a novel apatial-frequene, dual-domain integration network tailored to this specific task. The idea leads to a deep Fourier-batt based exposure correction network designed to tackle exposure-relatee issues. These pioneering studies harness the synergistic potential of deep learning and the intrinsic attributes of the Fourier transform.
5 Limitations
Our study acknowledges that there is room for more comprehensive experiments and exploration of representative baselines on broader computer vision tasks. It is important to note that this work represents the initial endeavor to delve into the utilization of shifting mechanisms and the development of tailored deep Fourier shifting operators specifically for low-level image restoration tasks. Moreover, focus extends beyond designing a aadg-aay module for integration into existing networks to achieve performance improvements. We aim to provide a powerful and efficient spatial communication interaction choice by offering an alternative to the basic convolution operator pool when constructing new models from scratch.
Broader Impact
Our research has the potential to facilitate efficient image restoration on edge devices and resource- limited platforms, democratizing access high-quality capabilities. This benefits critical domains like mobile photography, remote sensing, and medical imaging. Additionally, reduced computational requirements contribute energy efficiency, extending battery life and reducing environmental impact. We emphasize responsible development, evaluation, and deployment to ensure equitable and ethical use of our methods.
Conclusion
We have presented Deep Fourier Shifting, a shifting operator grounded in solid theoretical principles, specifically tailored for image restoration tasks. This operator, characterized by its information- lossless nature, leverages the Fourier cycling approach shift operations. A key feature of our Deep Fourier Shifting is its versatility; it can be seamlessly integrated as a replacement for standard convolution units in existing image restoration networks, offering the added benefit of reduced parameter usage. Our experimental evaluations have consistently demonstrated the enhancements in performance attributable to the incorporation of our Deep Fourier Shifting.

---

References
[1] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. S2-mlp: Spatial-shift mlp architecture for vision. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 297-306, January 2022.
[2] Guangting Wang, Yucheng Zhao, Chuanxin Tang, Chong Luo, and Wenjun Zeng. When shift operation meets vision transformer: An extremely simple alternative to attention mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2423-2430,
2022.
[3] Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gho- laminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9127-9135, 2018
[4] Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video under- standing. In Proceedings of the IEEF/CVF international conference computer vision, pages 7083-7093, 2019.
[5] Wenhan Yang Jiaying Liu Chen Wei, Wenjing Wang. Deep retinex decomposition for low-light enhancement. In British Machine Vision Conference. British Machine Vision Association, 2018.
[6] Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu Zou, Fang Lin, and Songchen Han. R2rnet: Low-light image enhancement via real-low to real-normal network. arXiv preprint arXiv:2106.1455 14501, 2021.
[7] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3291- 3300, 2018.
[8] Wenhan Yang, Shiqi Wang, Yuming Fang. Yue Wang, and Jiaying Liu. From fidelity to percep- tual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3063-33 3072, 2020.
[9] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883-3891, 2017.
[10] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In IEEE International Conference on Computer Vision, 2019.
[11] AbdeIrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1692-1700, 2018.
[12] Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1586-1595, 2017.
[13] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions on image processing, (6(7))3142-3155 2017.
[14] Zhou Man, Huang Jie, Yan Keyu, Yu Hu, Fu Xueyang, Liu Aiping, Wei Xian, and Zhao Feng. Spatial frequency domain information integration for pan-sharpening. In Proceedings of the European Conference on Computer Vision (ECCV), pages 234-250, 2022.
[15] Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. Effective pan- sharpening with transformer and invertible neural network, IEEE Transactions on Geoscience and Remote Sensing, 60:1-15, 2022.
10

---

[16] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce. Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data fusion contest. IEEE Transactions on Geoscience and Remote Sensing, 45(10)3001-3302 2007.
[17] A. F. Goetz J. R. H. Yuhas and J. M. Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. Proc. Summaries Annu. JPL Airborne Geosci. Workshop, pages 147-149, 1992.
[18] Tyrone E. Duncan. On the calculation of mutual information. SIAM Journal Applied Mathematics, 19(1):215-220, 1970.
[19] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, al. Mlp- mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261-24222 2021.
[20] Hu Yu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy, Deyu Meng, Chongyi Li, et al. Deep fourier up-sampling. Advances in Neural Information Processing Systems, 35:22995- 23008, 2022.

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Section 2.
Guidelines:
12

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Codes will be released after acceptance.
13

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
/ //nips. wwtpcccppppcccccccccccc.ccccccle wtwp:l/ccggiideeeeeeeeeeeeeeeeee wwwoodeeuiissiooooooioooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification:
The detailed descriptions are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
14

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///aaaaaa...ppppaaaa/aaaaaa
Answer: [Yes]
Justification:
We have read and followed the NeurlPS Code of Ethies
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader Impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
15

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswwthcode cwm/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
16

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
17


Reviews:
[{"review_id": "mbvp9ob1xk", "reviewer": "Reviewer_hJxw", "summary": "This paper explores the use of a spatial-shift operator for image restoration tasks, addressing the issue of information loss identified through experimental analysis. The authors propose a shift operator in the Fourier domain, leveraging the periodicity and cycling properties of the Fourier transform to develop a theoretically information-lossless operator that enhances performance. Two variants of the operator, magnitude-phase and real-imaginary, are considered. Experimental results across various image restoration tasks demonstrate superior performance compared to baseline methods and the spatial-shift operator. Interestingly, one could replace multiple spatial shifts with the proposed Fourier-shifts to reduce the number of parameters and thus potentially reduce memory requirements.", "questions": "In Figure 6-right, what is the cause of the artifacts present before the application of Shift-sa? Shouldn’t the two feature maps be identical prior to the application of either of the two operators?\n\nMinor issues:\n\n- In Table 4, the ERGAS metric value for the Shift-sa method on the WorldView-II dataset appears to be erroneously formatted in bold.\n    \n- Table 5 does not display any gray backgrounds in my PDF viewer. The statement on Line 233, “To further elucidate this point, values falling below the baseline are marked with a gray background, highlighting their relevance and impact,” ([pdf](zotero://open-pdf/library/items/9U2B2N4L?page=9)) may be misplaced.", "limitations": "No comments", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper presents a conceptually straightforward and theoretically sound idea.\n\n2) The proposed operator enhances the performance of image restoration tasks. Simultaneously, depending on its design, it could potentially reduce computational complexity.", "weaknesses": "The lower portion of Table 5 indicates that the number of parameters remains constant when multiple Fourier-shift operators are substituted with each spatial-shift operator. However, crucial metrics such as runtime and FLOPs are not included. Solely relying on the number of parameters does not provide a comprehensive understanding of the ablation’s suitability for the ultimate objective of deployment on edge or mobile hardware.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1, Complexity.**\n\nThank you for your feedback. In Table 5, we compared the performance of our proposed Fourier shifting operator against the spatial shift operator and the baseline 3x3 convolution. Replacing convolutions with Fourier shifting not only reduced the number of parameters by half but also improved network performance, whereas the spatial shift operator led to a significant 14 dB performance drop. Additionally, we evaluated runtime and FLOPs to assess suitability for edge devices. Our experiments show that the DnCNN-18 model with our Fourier shifting configuration achieves an average runtime of 7 milliseconds and 1.5 GFLOPs per 256x256 image, compared to 15-20 milliseconds and 3.9 GFLOPs for the baseline on an NVIDIA GTX 1080 GPU. This indicates a substantial reduction in both runtime and FLOPs, highlighting the efficiency of our method for deployment on resource-constrained hardware. We will update Table 5 to include these metrics for a comprehensive evaluation. The lower portion of Table 5 uses a controlled variable approach to ensure consistent parameters while evaluating the network's robustness to shifting size. It is important to note that the spatial-shifting configurations were not aligned with our method's parameters throughout. Our results demonstrate that our network exhibits high robustness to varying shifting sizes.\n\n**2, artifacts.**\n\nThank you for your feedback. Firstly, the Fourier transform is an efficient tool that amplifies image degradations in the Fourier domain, and our learnable parameters act as filters to eliminate these artifacts. Secondly, the artifacts arise from two aspects: (1) features are extracted from low-light degraded inputs, which naturally reflect these degradations, and (2) the testing baseline network inherently uses down-sampling operators to achieve multi-scale features, where the down-sampling inherently causes frequency truncation, leading to frequency aliasing and ringing effects. Additionally, to verify the robustness of our experiments, we analyzed both shallow and deep features in the network, finding that similar artifacts persist and even accumulate, whereas our Fourier shifting can eliminate these, resulting in cleaner features. Our frequency domain approach inherently handles these degradations. The input discrepancies are due to the network being trained from scratch after incorporating our operator, resulting in natural feature changes. Our network effectively reduces degradations both before and after the application of shift-sa, while the degradation persists with shift-sa. Finally, we used mutual information to measure information loss caused by shifting, which validates the effectiveness of our method.\n\n**3,minor issue.**\n\nThank you for your detailed review and valuable feedback. We will carefully review the entire manuscript to correct any typos and grammar errors to improve the overall clarity and quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kw0z2F1twV", "reviewer": "Reviewer_3jgW", "summary": "The authors in this paper aim at exploring more into image restoration task via the concept of spatial shift operation that facilitates efficient spatial communication and has achieved significant advancements in several high-level vision tasks.  As per their observation, since the image restoration is more spatial shift sensitive, they propose an information lossless shifting operator, i.e. Deep Fourier Shifting. Additionally in  the case of the popular spatial shift operators, the regions subjected to shifting are filled with zeros which may lead to loss of information and a decline in the overall restoration performance. \n\n\nThe main  contribution of the authors lies in  proposing a novel deep shifting operator for low-level image restoration tasks, named Deep Fourier Shifting. This operator basically revisits and enhances the fundamental principles of the traditional shift operator by extending its application into the Fourier domain.", "questions": "1. How is the mutual information shown in FIgure 6 (left ) calculated?\n2. The feature maps in Fig 6(right) does not clearly show the removal of grid effects, is it possible to show some extra images.", "limitations": "The limitations could also be framed in this way that they did not show much on real world tasks or any experiments on edge devices.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors delve into exploring the concept of shift operator for low level vision tasks which seems to be interesting.\n2. Extensive experiments across multiple low-level tasks including image denoising, low-light image enhancement, guided image super-resolution, and image de-blurring demonstrate consistent performance gains obtained by our Deep Fourier Shifting while reducing the computation burden.", "weaknesses": "1. The contribution seems to be somewhat limited, the extension of spatial shifting into fourier domain and then proving its efficacy seems to be not a strong contribution.\n2. The content in the paper does not go well with the title, \"improving the small size restoration network by ....\". If the reviewer understand correctly, there does not seem to be much focus on this concept or on this experiment. like if we look at the representative network for any 1 task, say low light image enhancement e.g SID, then there should be proper explanation as to how the proposed concept is working on it in terms of parameters, flops, inference time.\n3. There are no real world datasets actually being considered as is claimed in the introduction part, as the results for deblurring, low light seem to be only on synthetic datasets.\n4. Regarding the implementation detail part, it is very confusing to understand this part as what exactly is the baseline or the original model hasn't been explicitly mentioned.\n5. Typos: The word field for receptive field has been incorrectly written as filed in many places.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1, Fig.6.**\n\nThank you for your feedback. Firstly, the Fourier transform is an efficient tool that amplifies image degradations in the Fourier domain, and our learnable parameters act as filters to eliminate these artifacts. Secondly, the artifacts arise from two aspects: (1) features are extracted from low-light degraded inputs, which naturally reflect these degradations, and (2) the testing baseline network inherently uses down-sampling operators to achieve multi-scale features, where the down-sampling inherently causes frequency truncation, leading to frequency aliasing and ringing effects. Additionally, to verify the robustness of our experiments, we analyzed both shallow and deep features in the network, finding that similar artifacts persist and even accumulate, whereas our Fourier shifting can eliminate these, resulting in cleaner features. Our frequency domain approach inherently handles these degradations. The input discrepancies are due to the network being trained from scratch after incorporating our operator, resulting in natural feature changes. Our network effectively reduces degradations both before and after the application of shift-sa, while the degradation persists with shift-sa. Finally, we used mutual information to measure information loss caused by shifting, which validates the effectiveness of our method.\n\n**2, MI.**\n\nIn Figure 6 (left), mutual information is calculated by first extracting feature maps before and after applying the shift operator and averaging the channel dimensions of these feature maps. These averaged values are then used as the variables for mutual information computation. Joint and marginal histograms of the pixel intensities are converted into probability distributions. This measure assesses the amount of shared information between the feature maps, indicating how well the shift operator preserves feature information.In terms of the index representing the network depth, we have tested multiple stages of the network, ranging from shallow to deep layers, to demonstrate the completeness of information retention. Relevant experiments have also confirmed that our proposed shifting method effectively preserves information throughout the network. Our operator exhibits significantly higher mutual information than Shift-sa, showcasing its efficacy in information preservation.\n\n**3, real-world.**\n\nThank you for your feedback. Due to time constraints, we initially conducted experiments on real-world remote sensing satellite scenes, which involve more complex degradations compared to the simulated scenarios with simple down-sampling and blurring. The results were as follows:\n\n- **PANNET**:\n  - D_{\\lambda} = 0.0737 \n  - D_s = 0.1224\n  - QNR = 0.8143\n\nHowever, with our newly designed network, the updated metrics are:\n\n- **New Network**:\n  - D_{\\lambda} = 0.0716 \n  -  D_s = 0.1215\n  - QNR = 0.8156\n\nHere, lower values of D_s and  D_{\\lambda}, along with a higher QNR, indicate better performance. These improvements demonstrate that our approach continues to perform effectively in real-world scenarios. We will update the paper to reflect these results and emphasize the effectiveness of our method in practical applications.\n\n**4, comlexity.**\n\nWe understand that the paper’s content might not fully align with the title. To address this, we will revise the paper to better demonstrate how our method enhances small-size restoration networks. This revision will include a thorough discussion of its impact on parameters, FLOPs, and inference time, particularly for tasks such as low-light image enhancement using networks like SID. We will ensure that the paper clearly explains the application and evaluation of our proposed concept in this context. We also acknowledge that the implementation details might be unclear. To clarify, we will explicitly define the baseline or original model used for comparison, including a detailed description of the model architecture, configurations, and any modifications. Due to time constraints, we tested our method on the DnCNN model. In Table 5, we compared our Fourier shifting operator with the spatial shift operator and the baseline 3x3 convolution. Our Fourier shifting approach not only halved the number of parameters but also improved network performance, whereas the spatial shift operator caused a significant 14 dB performance drop. Additionally, we assessed runtime and FLOPs to evaluate suitability for edge devices. Our results show that the DnCNN-18 model with Fourier shifting achieved an average runtime of 7 milliseconds and 1.5 GFLOPs per 256x256 image on an NVIDIA GTX 1080 GPU, compared to 15-20 milliseconds and 3.9 GFLOPs for the baseline. This substantial reduction in both runtime and FLOPs highlights the efficiency of our method for resource-constrained hardware.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NtVjRYTzIG", "reviewer": "Reviewer_H1Gj", "summary": "This paper addresses challenges in current image restoration methods, which are often too computationally demanding for edge devices. It proposes Deep Fourier Shifting, a novel approach inspired by spatial-shift operators adapted for low-level image tasks. By leveraging the Fourier domain and ensuring information preservation through cycling, Deep Fourier Shifting introduces two variants—amplitude-phase and real-imaginary—designed to replace conventional convolution units in existing networks with fewer parameters. Extensive experiments across denoising, low-light enhancement, guided super-resolution, and de-blurring tasks demonstrate consistent performance gains and reduced computational overhead, validating the robustness and efficiency of the proposed approach.", "questions": "See the above weaknesses part.", "limitations": "The author has discussed the limitations of the method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper is well-structured and effectively organized. It provides a clear introduction to the challenges faced by current image restoration methods on edge devices due to computational limitations.\n\n2.The introduction of deep fourier shifting as a technique for small-size image restoration networks is both feasible and holds practical significance. By leveraging spatial-shift principles in the Fourier domain, the method addresses the computational constraints of edge devices while maintaining or improving restoration performance. This approach not only enhances the efficiency of image restoration tasks but also facilitates practical deployment in real-world applications where computational resources are limited.\n\n3.This paper demonstrates promising results for the deep fourier shifting method in various small-size image restoration tasks, including denoising, low-light enhancement, guided super-resolution, and de-blurring. Experimental evaluations consistently show performance improvements with reduced computational overhead compared to traditional convolutional approaches.", "weaknesses": "1.While the proposed method shows promise for deployment on edge devices due to reduced computational requirements, potential practical implementation challenges should be addressed. These might include considerations such as memory usage, real-time processing capabilities, and adaptability to different hardware platforms. Discussing these aspects would enhance the manuscript's practical relevance and feasibility in real-world applications.\n\n2.While the Deep Fourier Shifting method shows promising results across multiple low-level image restoration tasks, including denoising, low-light enhancement, and super-resolution, it would be valuable to explore its performance across a broader range of image types and degradation levels. Assessing how well the method handles complex real-world scenarios with varying textures, structures, and noise characteristics would strengthen the manuscript's applicability.\n\n3.I noticed that the references primarily include works up until 2022. To ensure the manuscript reflects the latest advancements in the field, I recommend incorporating relevant studies published in 2023 and onwards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1, types.**\n\nWe appreciate the suggestion and will expand our experiments to include various types of images with different textures, structures, and noise characteristics. This will provide a more comprehensive assessment of how Deep Fourier Shifting performs under a wider array of conditions, enhancing the manuscript’s applicability and robustness.\n\n**2, Complexity.**\n\nWe appreciate the feedback on the practical implementation of our proposed method on edge devices. To address potential challenges, we will discuss key aspects such as memory usage, real-time processing capabilities, and adaptability to different hardware platforms. This will include information on memory requirements, real-time processing metrics like latency and throughput, and evaluations on various hardware configurations, including CPUs, GPUs, and specialized edge processors. By addressing these aspects, we aim to enhance the manuscript's practical relevance and demonstrate the feasibility of deploying our method in real-world edge device applications. Due to time constraints, we tested our method on the DnCNN model. In Table 5, we compared our Fourier shifting operator with the spatial shift operator and the baseline 3x3 convolution. Our Fourier shifting approach not only halved the number of parameters but also improved network performance, whereas the spatial shift operator caused a significant 14 dB performance drop. Additionally, we assessed runtime and FLOPs to evaluate suitability for edge devices. Our results show that the DnCNN-18 model with Fourier shifting achieved an average runtime of 7 milliseconds and 1.5 GFLOPs per 256x256 image on an NVIDIA GTX 1080 GPU, compared to 15-20 milliseconds and 3.9 GFLOPs for the baseline. This substantial reduction in both runtime and FLOPs highlights the efficiency of our method for resource-constrained hardware.\n\n**3, onwards.**\n\nThank you for your observation. We will update the manuscript to include recent studies and advancements published in 2023 and onwards. Due to time constraints, we have conducted additional experiments using the recent work LFormer [1] presented at ACM-MM 2024. Specifically, we replaced the convolution operator in feature extraction with Fourier shifting to evaluate its impact on guided image super-resolution on world-view-II and Gaofen-2 satellites\n|     |   | WV2  |   |   | |   | GF2  |   |\n|--|---|-----|----------|------------|-----------|-------------|----------|------------|\n| Method    | SAM  | ERGAS  | Q8  | PSNR  | SAM  | ERGAS  | Q4  | PSNR  |\n| LFormer   | 2.8985    | 2.1645      | 0.9193   | 39.0748    | 0.6481    | 0.5778      | 0.9851   | 44.1958    |\n| LFormer+  | 3.1576    | 1.7046      | 0.8630   | 39.2988    | 0.5325    | 0.4878      | 0.9904   | 45.4331    |\n\nIncorporating our proposed Fourier shifting design into the LFormer framework has demonstrated significant improvements in performance. The results indicate that our method not only enhances the accuracy of guided image super-resolution but also improves the efficiency of the feature extraction process. This advancement is evident from the substantial gains in quantitative metrics, underscoring the effectiveness of our approach in optimizing and refining the image restoration capabilities of the model. Moving forward, we will incorporate additional methodologies to further validate and enhance the efficiency of our approach.\n\n[1] Linearly-evolved Transformer for Pan-sharpening, ACM-MM 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NtqFptcbSD", "reviewer": "Reviewer_GBwc", "summary": "The authors introduce a theoretically sound Fourier shifting operator designed to enhance the learning capability of small-size image restoration models. This work represents the first comprehensive attempt to model the shift mechanism within the Fourier domain. The proposed operator is versatile and can be seamlessly integrated into existing image restoration networks, offering a flexible plug-and-play solution.", "questions": "Please see the weakness part.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, The proposed deep Fourier shifting operator is theoretically plausible and enhances learning by transforming spatial shifting into an information-preserving Fourier cycling manner. This novel approach models the shift mechanism in the Fourier domain comprehensively.\n\n2, Integrating Fourier shifting into existing image restoration methods improves performance and reduces model parameters, demonstrating practical effectiveness and efficiency.\n\n3, The paper is well-organized and easy to follow. The clear presentation and structure help us understand the progression from problem formulation to the proposed solution.", "weaknesses": "1, The authors design two Fourier shifting variants. While both improve performance, neither consistently outperforms the other across different baselines. It would be helpful if the authors could suggest which variant to choose for different baselines or application scenarios, and explain the differences between the variants.\n\n2, In Fig.7, the authors present the generalizability of the shifting displacement effect from LOL to Huawei. I wonder why the proposed Fourier shifting has better generalization ability, and the authors are encouraged to provide some underlying working mechanisms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1,generalization.**\n\nFourier shifting exhibits enhanced generalization due to its operation in the frequency domain, which captures global image information and maintains feature integrity while minimizing domain-specific artifacts. This approach is particularly advantageous for managing low-light degradation, a global phenomenon effectively addressed in the Fourier domain. By incorporating this domain-specific knowledge, our method benefits from a natural prior that significantly improves adaptability across diverse datasets. Moreover, Fourier shifting is more parameter-efficient and reduces artifacts compared to spatial shifting, which often suffers from issues like information loss and frequency aliasing. This leads to more robust feature extraction and consistent performance across varied scenarios. Our revised paper will thoroughly discuss these advantages, including the Fourier domain's role in addressing global degradations and how it contributes to the superior generalization of our method, ensuring better results across different real-world applications.\n\n**2,variants.**\n\nLike the spatial convolution techniques, our proposed “Fourier Up-Sampling” is a drop-in alternative for convolution operator, also the general strategy and each variant thus does not prefer a specified vision task. The two Fourier shifting variants each have specific strengths and potential drawbacks. The Amplitude-Phase Variant processes amplitude and phase components separately, which is beneficial for tasks requiring precise phase preservation. However, this variant involves trigonometric functions, which can introduce numerical instability due to the amplification of small deviations in angle, affecting precision-sensitive applications. On the other hand, the Real-Imaginary Variant processes the real and imaginary parts separately, offering greater numerical stability and being suitable for tasks involving complex textures or fine-grained restoration. In our revised paper, we will provide detailed guidance on choosing between these variants based on the task requirements, baseline characteristics, and potential numerical stability issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agree about the originality and the PSNR performance of the proposed solutions.

As pointed out by reviewer 3jgW, this paper did not mention any results regarding the speed or floating point operations. During rebuttal authors have provided results on DNCNN-18, which shows a significant improvement on FLOPS. During the discussion period reviewers discussed that this results should be scalable to other architectures as well. 

Re viewer RB41 pointed out to some old references in the experiments. The authors provided comparisons with more recent work during the rebuttal.
Experiments for pan sharpening show results for PANNET (2017) and MutNET (2022), but the citations in text mentions two other works in the field. Authors should fix that in their table/text.

In the discussion period with AC, the reviewers came to the consensus that the paper should be accepted.
Decision: Accept (poster)