FouriDown: Factoring Down-Sampling into Shuffling and Superposing
Authors: Qi Zhu
Man Zhou
Jie Huang
Naishan Zheng
Hongzhi Gao
Chongyi Li
Yuan Xu
Feng Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: Image restoration
Down-Sampling
Fourier transform
TL;DR: The work revisits the spatial down-sampling and proposes a new framework for learning dynamic frequency interaction from down-sampling in a unified Fourier function.

Abstract: Spatial down-sampling techniques, such as strided convolution, Gaussian, and Nearest down-sampling, are essential in deep neural networks. In this study, we revisit the working mechanism of the spatial down-sampling family and analyze the biased effects caused by the static weighting strategy employed in previous approaches. To overcome this limitation, we propose a novel  down-sampling paradigm in the Fourier domain, abbreviated as FouriDown, which unifies existing down-sampling techniques. Drawing inspiration from the signal sampling theorem, we parameterize the non-parameter static weighting down-sampling operator as a learnable and context-adaptive operator within a unified Fourier function. Specifically, we organize the corresponding frequency positions of the 2D plane in a physically-closed manner within a single channel dimension. We then perform point-wise channel shuffling based on an indicator that determines whether a channel's signal frequency bin is susceptible to aliasing, ensuring the consistency of the weighting parameter learning. FouriDown, as a generic operator, comprises four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform. These components can be easily integrated into existing image restoration networks. To demonstrate the efficacy of FouriDown, we conduct extensive experiments on image de-blurring and low-light image enhancement. The results consistently show that FouriDown can provide significant performance improvements. We will make the code publicly available to facilitate further exploration and application of FouriDown.

FouriDown: Factoring Down-Sampling into Shuffling and Superposing
Qi Zhu 1 Man Zhou 233- Jie Huang 1 Naishan Zheng Hongzhi Gao
Chongyi Li 4 Yuan Xv 3, Feng Zhao
University of Science and Technology of China,
LS-Lab, 'Nayyang Technological University, *Nankai University
fzqerafta, aajorf mazheng, hongzhigao) uatc.edu.. en,
{man. zhou, xu. yuan) wnw.iedu.uuuuu.........ee edu. 1ichongyi258gmail com, com, fzhao9560ustc. edu. cn
Abstract
Spatial down-sampling techniques, such as strided convolution, Gaussian, and Nearest down-sampling, are essential in deep neural networks. In this paper, we revisit the working mechanism of the spatial down-sampling family and analyze the biased effects caused by the static weighting strategy employed in previous approaches. To overcome the bias limitation, we propose novel hown-sampling paradigm in the Fourier domain, abbreviated as FouriDown, which unifies existing down-sampling techniques. Drawing inspiration from the signal sampling theorem, we parameterize the ma-paareettrr static weighting down- down-sampling operator as a learnable and context context-adaptive operator within unified Fourier function. Specif- ically, we organize the corresponding frequency positions of the 2D plane in a physically-closed manner within a single channel dimension. We then perform point-wise channel shuffling based nel's signal frequency bin is susceptible to aliasing, ensuring the consistency of the weighting parameter learning. FouriDown, key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adapun adaptively superposing rules, and 2D inverse Fourier transform. These components can be easily integrated into existing image restoration networks. To conduct extensive experiments
indicator that determines whether chan-
general operator, comprises four
demonstrate the efficacy of FouriDown, image de-blurring and low-light image enhancement. The results consistently show that FouriDown can provide significant performance improvements. The code is publicly available to facilitate further exploration and application of FouriDown at http://www....coe.///oo/oooooooo
Introduction
Down-sampling technique [1, 2, 3] plays a vital role in deep neural networks because of its benefits in enlarging the receptive field, extracting hierarchical features, improving computational efficiency, and handling scale and translation variations. However, based on the signal sampling theorem, existing lown-sampling techniques such as strided convolution, Gaussian, and Nearest down-sampling
[1, 4, 5, 6] unavoidably reduce the sampling frequency of discrete signals, leading to unexpected frequency aliasing where high frequencies are folded into low frequencies.
To address the aliasing problem, several strategies [7, 8, 9, 10, 11, 12, 13] have been developed. They pre-process the signals applying the low-pass filtering mechanism, which aims filter high- frequency information by employing different types of low-pass designs. There are two commonly
Botth authors contributed equally this rescarch.
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Spatial Domain:
Downsampling (strides=
Previous Downsampling:
H
CxHXW
C
i Strides 2 (e.g. Conv, Pooling)
Wy [0.25, 0.25, 0.25, 0.25]
Static weighting
FouriDown(s,, wi)
Fourier Domain:
(ii Ideal low-pass -2x
(0.25, 0.00, 0.00, 0.00]
Shuffling (Sec 3.3)
Superposing
(Sec 3.2)
Guassian filter-2x
W
When strides=2,
Mi Guassian Distribution
IDFT
X'
WiX
Ours (Sec. 3.3)
Adaptive
Wi
Softmax
Conv
Rein
Cunv
DFT Discrete Fourier Transform
IDFT Inverse Discrete Fourier Tramsform
Comparison the flowcharts of different down-sampling techniques in 2x scale. The previous spatial down-sampling community stands on the static weighting templates and is not relevant with the image context. By contrast, inspired by the signal sampling theorem, we parameterize the static weighting down-sampling operator as a learnable and context-adaptive operator in a unified Fourier function.
Figure 1:
used types including the ideal low-pass filter that truncates high frequencies in the Fourier domain and the Gaussian low-pass filter that gradually attenuates frequency components near the boundary. However, the ideal low-pass filter may introduce ring artifacts due spectrum leakage, while the Gaussian low-pass filter may result in significant loss of edge information that is crucial for visual recognition tasks.
The prevailing approaches in the down-sampling family rely on a static weighting strategy, which may lead to unintended biases (See Section 4.4 for details.). As described in Figure 1, strided convolution and strided pooling variants rely on the static template w; = [0.25, 0.25, 0.25, 0.25] over the corresponding cornered positions while the ideal low-pass one exploits the Wt; [0.25, 0, 0, weighting template. (See Appendix B for proofs.) All of them are shared over all the coordinated positions and uninvolved with the feature context. It widely acknowledged that the static sampling approach, which lacks contextual relevance, is sub-optimal for visual tasks. Therefore, both bridging different samp-sgn approaches and achieving an optimal approach are desirable, shown in Figure 1 where we focus on unifying the down-same down-sampling modeling rules in learnable and context-adaptive parameterized function Fourier domain.
In this study, we delve into the working mechanism of the spatial down-sampling family and ana- lyze the biased effects resulting from the static weighting strategy used in existing down-sampling approaches. To solve the bias problem, we propose a novel down-sampling paradigm called Fouri- Down, which operates in the Fourier domain and adapts the feature sampling based on the image context. Inspired by the signal sampling theorem, we parameterize the non-parameter static weighting down-sampling operator
learnable and context-adaptive operator in unified Fourier function. Furthermore, drawing from this insight, we organize the corresponding frequency positions of the 2D) plane, ensuring that they are physically closed in single channel dimension. We then perform point- wise channel shuffling based on an indicator that determines whether channel's signal frequency bin prone aliasing, thereby maintaining the consistency of the weighting parameter learning. Fouri- Down, as a generic operator, comprises four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform. These components can be readily integrated into existing image restoration networks, allowing for plug-and-pl approach. To verify its efficacy, we conduct extensive experiments across multiple computer vision tasks, including image de-blurring and low-light image enhancement. The results demonstrate that FouriDown consistently outperforms the baselines, showcasing its capability of performance improvement.

---

In conclusion, this work propose a novel and unified framework for the research of down-sampling, which have the following contributions.
1) We provide the first exploration of the aliasing problem in deep neural networks, analyzing it from spectrum perspective.
2) To achieve dynamic frequency aliasing, we introduce a unified approach to down-sampling strategies within the Fourier function. Additionally, we propose learnable and context-adaptive down-sampling operator based on the Nyquist signal sampling theorem.
3) Our proposed down-sampling approach serves as plug-and-ple and-play operator, consistently enhancing the performance of image restoration tasks, such as low-light enhancement and image deblurring.
2 Related Work
2.1 Traditional Down-Sampling
Downsampling is an important and common operator in computer vision [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25J, which benefits from enlarging the receptive field and reducing computational costs. So many models incorporate downsampling allow the primary reconstruction components conducting at lower resolution. Moreover, with the emergence of increasingly compute-intensive large models, downsampling becomes especially crucial, particularly for high-resolution input images.
Previous downsampling methods often utilized local spatial neighborhood computations (e.g., Bilin- ear, Bicubic and MaxPooling), which show decent performances across various tasks. However, these computations are relatively fixed, making it challenging to maintain consistent performance across different tasks. To address this, some methods made specific designs make the downsampling more efficient in specific tasks. For instance, some works [12, 11, 10, 7jintroduce the Gaussian blur kernel before the downsampling convolution combat aliasing for better shift-invariance in classification tasks. Grabinski et al. [26, 27] equip the ideal low-pass filter or the hamming filter into downsampling to enhance model robustness and avoid overfitting.
2.2 Dynamic Down-Sampling
Due to the development of data-driven deep learning, in addition to traditional down- down-sampling, some other works [28, 29, 30, 31, 32, 33] introduce dynamic downsampling to adaptively adjust for different tasks, thereby achieving better generalizability. For instance, Pixel-Shuffle [28] enables dynamic spatial neighborhood computation through the interaction between feature channels and spaces, restoring finer details more effectively. Kim et al. [29] proposes task-aware image downsampling to support upsampling for more efficient restoration.
In addition to dynamic neighborhood computation, dynamic strides have also gained widespread attention in recent years. For instance, Riad et al. [30] posits that the commonly adopted integer stride of 2 for downsampling might not be optimal. Consequently, they introduce learnable strides to explore a better trade-off between computation costs and performances. However, the stride is still spatially uniformly distributed, which might not be the best fit for images with uneven texture density distributions. To address this issue, dynamic non-unifon unn-ufformm sampling garners significant attention [31, 32, 33J. For example, Thavamani et al. [31] proposed saliency-guided non-uniform sampling method aimed at reducing computation while retaining task-relevant image information.
In conclusion, most of recent researches focus on dynamic neighborhood computation or dynamic stride for down-sampling, where the paradigm can be represented Down(s), where denotes the stride. However, in this work, we observe that the methods based on this downsampling paradigm employ static frequency aliasing, which may potentially hinder further development towards effective downsampling. However, learning dynamic frequency aliasing upon the existing paradigm poses challenges. To address this issue, we revisit downsampling from a spectral perspective and propose a novel paradigm for it, denoted as Fouri Down w). This paradigm, while retaining the stride parameter, introduces a new parameter, w, which represents the weight frequency aliasing during downsampling and is related to strides. Further, based on this framework, we present elegant and effective approach to achieve downsampling with dynamic frequency aliasing, demonstrating notable performance improvements across multiple tasks and network architectures.
3

---

(a) I-Dimension: F(t) and F'
(b) 2-Dimension: F(u,y) and (u,y)
4g
0g
(Better in
Dg
2
Amp
Before Downsanpling
201
Qg
11:
Po
42(44
Ps Mag
11
Amp
After Dowasanpling
6p
Figure 2: The visualization of the shuffling and superposing theory in the 1-D and 2-D signals.
3 Method
Definitions. f(x, y) E Hxx the 2-D discrete spatial signal and the sampling rates in x and y axis are Sig and Sy, respectively. F(u,v) gulxxXG is the Fourier transform of f(x, y), where the maximum frequencies in u and v axis are respectively denoted as umax and Umaz- Moreover, f' y) E * xC is 2-strided down-sampled f(x.y) and its Fourier transform F" (u,
11 Shuffling and Superposing. The spatial down-sampling typically results in shrinkage of the tolerance for the maximun frequency of the signal. Specifically, high frequencies will fold back into new low frequencies and superpose onto the original low frequencies. To illustrate with 1- dimensional signal, the high and low frequency superposition in the down samplin can be formulated
(u) F(F( + Px 2x)) when u E (0, 2
(I)
where S a superposing operator. Note that the high frequency is F(u + (x) considering positive directions, while the low frequency is F()) considering positive directions instead.
Theorem-2. Static Averaging Superposing, For an image, the spatial down-sampling operator with 2 strides can be equivalent to dividing the Fourier spectrum into 2 x 2 equal parts and superposing them averagely by 1 factor
Flu, v) = (u. FOD (u. F1,,0 + F(.i)
(2)
where (u, v) is a sub-matrix of F(u, v) by equally dividing F(u, v) into 2 2 partitions and i.j E {0,1). 1}. Given that Downg is 2-strided down- nown-sampling operator and IDFT inverse discrete Fourier transform, we have
Has)(u i=
Down (f(x, IDFT
1
The proof of the above theorem can be found in the Appendix, and examples 1-D and 2-D signals can also be referred in Figure 2(a) and (b).

---

Cluster superposing groups (Theorem I and Theorem 2)
Band Aligament
Spectral Shuffle
Split
16 patches
D>
P
CxHXW
0
2
Mtu, x)= uuu 1o. )
R e (0 2
Rinchuu e B M(u.)), 1 m
D2 Min.))x 5 Aue
4xCx
Figure 3: Overview of the proposed spectral shuffle.
Algorithm 1 Pseudo-code of FouriDown.
Data: Input: E X Set of real numbers: R. Set complex numbers: C.
Result: y E RNNCX
Ifst + FFT(z) E CNXCXIXW // Fast Fourier transform
Fseaffle Shuf Shufffe(xzss) CNxxxCx#*
Tweight f- Soo (ooo:mmmaaooossss/hhhfffeeeeee groups = C)) E CNXIXC***
I Ssuperposed + Sum(Iweightt Zzhuf fle; dim 1) xxXx
y + iFFT (FSuperppop)e) // Inverse Fourier transform
3.1 Architecture Design
In this work, we argue that the the static superposing strategy like the stride-basee down-sampling in Theorem-2 might lead biased effects. Motivated by adaptively learning ability of CNNs, aim to parameterize the non-parameter static superposing step as a learnable and context-adaptive operator in the Fourier domain.
Definition-2 Schuffle-Invarianne Given an operator =(.) that is shuffle-invariant and 01, 02, 03, 01 as different components, the shuffle-nvvrrian is defined 2(01, 02, 03, 04) = (shuffle 03, 01)), where shuffle is shuffling the order of input components arbitrarily.
Note that the average operator in Theorem-2 is shuffle-invarian. For example, Aver(a, c. = Aver (b, a, c, d)). However, different from averaging. the convolution operator, which is sensitive to the input order, does not have this property.
To alleviate this problem, design spectral-shuffle strategy that first performs shuffling according to Theorem-1 and then aligns across different frequency bands, as shown in Figure 3. Specifically, we initially spilt the original spectrum F(u, v) into patches equally. Then, according Thcorem-L, we classify these patches into 4 group. where each group is pixel-wise matched frequency bin for superposing. However, the energy distribution each group is different. Considering the shuffle- variance of convolution operators, we reorder the intra-group sequence for inter- groug alignment. The alignment motivated by wavelet theory, where antra-group frequencies are reordered according to low- oowffrequency and high-frequency in horizontal direction, high-frequency in vertical direction, and high-frequency diagonal direction. Then, the aligned groups are sorted orderly on channels for pixel-wise matching in the channel dimension. Finally, we perform adaptively weighted superposition on channels by learned weights for the down-sampling results. The main implementation is depicted in Algorithm I. Code will be public.
5

---

Experiments and Discussion
To validate the efficacy our proposed FouriDown, we execute comprehensive experiments across several computer vision tasks and conduct exhaustive ablation studies to evaluate its resilience.
4.1 Experimental Settings
Image enhancement. For image enhancement, assess our FouriDown model using the LOL [34] and Huawei [35] benchmarks. The LOL. dataset contains 500 image pairs (485 for training, 15 for testing), and the Huawei dataset contains 2480 paired images (2200 for training, 280 for testing). We compare our results with two established baselines, SID [36] and DRBN [37].
Image deblurring. For image deblurring, we utilize DeepDeblur [38] and MPRNet [39] on the DVD dataset [40], which includes 2103 training and 1111 test pairs. We further validate our model's generalizability using the HIDE dataset [41].
Image denoising. In the context of image de- de-noising, our training involves the SIDD dataset [42]. Subsequent performance assessments are carried out on the remaining validation samples from the SIDD dataset and on the DND benchmark dataset [43]. For comparative analysis, we choose baselines such as MIRNet [44], and Restormer [45].
Image dehazing. For image dehazing, employ RESIDE dataset for evaluations. We also use two different network designs MSBDN [46] and GridNet [47] with our proposed operator for validation.
4.2 Implementation Details
Regarding the above competitive baselines, we perform the comparison over the following configu- rations by replacing the down-sampling operator, such strided convolution and strided pooling), with the proposed FouriDown operator. Additionally, we also perform comparisons with previous anti-aliasing down-sampling methods, including Gaussian filter [7] and "ideal" Low-Pass Filter (LPF) [13], which conduct the static modulation on the spectrum.
1) Original: The original down-sampling in the baseline;
2) Gaussian: Following [7]. equipping the Gaussian filter in all channels before the original down-sampling for anti-aliasing;
3) LPF: Following [13], equipping the "ideal" Low-Pass Filter in all channels before the
original somn-sa operator for anti-aliasing;
4) Ours: Replacing our proposed FouriDown with the original down-sampling operator;
Table
Image enhancement comparison.
Table 2: Image deblurring comparison.
LOL
Huawei
DVD
HIDE
Method
Config
Method
Conlg
PSNR
PSNR
SSIM
Original
19.92
0.7712
20.21
0.6742
Original
29.32
0.8817
29.60
0.8849
Gaussian
20.21
0.8146
20.66
0.6955
Gaussian
29.36
0.8823
29.62
0.8892
DRBN
DeepDeblur
LPF
18.91
0.7441
20.34
0.6812
LPF
29.19
0.8751
29.51
0.8851
Ours
21.64
0,8513
21,46
0.7213
Ours
29.44
0,8856
29.70
0.8904
Original
21.46
0.8584
20.38
0.6931
Original
30.12
0.8958
30.04
0.8945
Gaussian
21.78
0,8612
20.52
0.6926
Gaussian
30.23
0.8922
30.06
0.8966
SID
MPRNet
I.FF
0.8918
0.8937
Ours
30.31
30.25
0.9102
Table 3: Image de-noising comparison.
Table 4: Image dehazing comparison.
SIDD
DND
Indoor
Outdoor
Method
Config
Method
Config
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Original
39.41
0,9171
39.67
0.9173
Original
29.77
0.9591
28.88
0.9581
Gaussian
39.43
0.9169
39.69
0.9177
Gaussian
30.09
0.9607
28.91
0.9583
Restormer
MSBDN
I.PF
39.35
0.9162
39.64
0.9167
I.PF
29.92
0.9598
29.03
0.9591
39.47
0.9174
39.73
0.9180
Ours
30.19
0.9612
29.21
0.9604
Original
39.52
0.9182
0.9146
Original
0.9616
Gaussian
39.55
0.9184
39.45
0.9148
Gaussian
30.21
0.9617
29.62
0.9622
MIRNet
GridNet
LPF
39.49
0.9179
39.35
0.9141
LPF
30.18
0.9611
29.58
0.9615
Ours
39.64
0.9186
39,56
0.9251
Ours
30.42
0.9654
29.71
0.9641

---

(a) Low-light image
(b) SID
(c) SID with FouriDown
(d) Groud truth
Figure 4: Visual comparison of SID on the LOL dataset. FouriDown enhances the global color perception ability of the original model, thereby improving the model's performance without adding extra parameters or computational overhead.
(a) Low-light image
(b) DRBN
(c) DRBN with Fouriloown
(d) Groud truth
Figure 5: Visual comparison of DRBN the LOL dataset. The more flexible frequency interaction mechanism in FouriDown reduces artifacts compared to the original methods.
4.3 Comparison and Analysis
Quantitative Comparison. To demonstrate the effectiveness of our proposed FouriDown, we conduct extensive experiments shown in Tables 1-4, The best results are in bold. Above and below the baseline are highlighted in red and blue, respectively. From these tables, although previous anti-aliasing methods may be useful for some image restoration tasks, their static weights limit their universality in other tasks. For instance, while the LPF approach performs well in dehazing, it fails to deliver effective in deblurring and low-light enhancement. In contrast, our method is proved to be effective across the majority of image restoration tasks. Specifically, we achieved an improvement of 1.82dB in low-light enhancement and 0.42dB in dehazing on LOL and Reside dataset respectively.
Further, we compare the computing costs with other methods shown in Table 5, We include results from traditional samplinnp techniques like bicubic, bilinear, pixel-unshufile. 2x2 learnable CNN (with stride=2), max-pooling, average-pooling, IPF, Gaussian and Ours. Noting that the "Original" down-sampling of the method is pointed by asterisk (**'). This will allow clearer contrast and showcase the advantages of our method not only against anti-aliasing approaches but also against these conventional down-sampling methods.
Qualitative Comparison. Due space constraints, only present qualitative comparison on the low-light enhancement task. As illustrated in Figure 4 and Figure 5, our FouriDown reduces original artifacts presented in the SID due to the more flexible frequency interactions. Then, we compare the visualizations of the feature maps and their corresponding spectra between FouriDown and other down-sampling methods (see Figure 6 and Figure 7). It can be observed that the model equipped with FouriDown generates much stronger responses to degradation-aware regions, i.e. global low- illumination in the low-light enhancement task. In contrast, the model with other down-sampling method responds weakly to these regions. The results demonstrates the effectiveness of FouriDown in capturing degradation-awan information by adaptive frequency superposition in lown-sampling. For the Gaussian method, its response to degradation is relatively large (second only to FouriDown), thus achieving performance that is also second only FouriDown. Similarly, as the LFP method has the poorest performance, its feature response of the low-light areas is also the lowest. The performance of other methods is roughly similar, their feature responses are also quite similar, indicating a similar capability to capture image degradation areas. Additionally, from the spectral comparison in Figure 7, can be observed that the Gaussian method loses a lot of high-frequency information compared to FowriDown. This leads to challenges in recovering textures in dark areas. Hence, although the Gaussian method exhibits good responses, FouriDown achieves better performances compared to More qualitative comparisons can be found in the following supplementary material.
7

---

(o) Input
(b) AverPooling
(c) Bienbic
(d) Bilinear
(e) Gauseian
(1) LPF
(g) MaxPooling
(h) PixelShufile
StrideComv
i Ours
Figure 6: Feature comparison between our FouriDown and other down-sampling methods in low-light enhancement task. Due to the unique global modeling mechanism in frequency domain, the features extracted by our method significantly achieve larger response than others.
Iuput
AverPooling
(c)
(d)
(e)
(9 LPF
(g) MaxPooling
(h) PixelShurfte
(i) StrideCony
Ours
Figure 7: Spectrum comparison of the feature maps in Figure 6. The spectrum following FouriDown obtains outstanding smooth response both high and low frequencies.
4.4 Discussions
Bias Effects by Static Superposing. As shown in Figure 8, we compare different down-sampling methods with static superposing manner, and find they have various bias effects.
Frequency Aliasing Visualization. To delve deeper into the high-low frequency interactions in down-sampling, examine the spectrum of images down-sampled by factors of 4x, 2x, and 1x. Following Theorem 1, some regions of spectrums are overlaid on the same frequency band, with smaller scales overlaying larger ones, shown in Figure 9. This alignment of same bandwidth reveals a rectangular contour at the intersections, where high-frree frequuereeesess not obeying the Nyquist theory fold into low frequencies during down sampling pointed by the yellow arrow. This suggests that significant for down-sampling to modulate frequencies, otherwise might degrade the original signal undesirably.
Other Discussions. Because space constraints, for more discussions, including extensions to Theorem-2 and revisiting of previous anti-aliasing methods in the proposed FouriDown framework, could be referred supplementary material.
5 Limitations
In this work, explore spatial down-sampling from frequency-domate perspective and optimize the static weighting of previous wwmn-samping with stride of in the frequency domain. Our modeling of down-sampling is based on using uniformly distributed impulse sequences as the sampling function, hence exploring characteristics of the sampling function in the frequency domain. However, for non-uniform down-sampling, where the sampling rate varies according to the content, our method might become limited. We hope to overcome this limitation the future work by exploring the frequency domain response of non-uniform sampling functions.

---

"aturall"
Aliaxing offect
"Ringing offectt
"Highffequrecy Iuns"
(2) Iuput
Nearest-2X
Ideal low-pass 22K
(4) Guasslan Ix
Figure 8: The comparison of results by different down-sampling manner in 2x scale. In (b), which is the nearest down-sampling of (a), some unnatural stripes in the edge of windows and walls, called aliasing effects. To relieve the occurrence of aliasing, (c) and (d) employ the "ideal low-pass filter" and the Gaussian filter before down-sampling respectively. However, these manners usually lead to ringring effects or heavy high-frequency loss inevitably.
LOL
S=1
Config
FLOPs(G)
Para(M)
PSNR
SSIM
Bicubic
21.35
0.8497
13.764
7.84
S=2
Bilinear
21.26
0.8464
13.764
7.84
Pixle-shuffle
21.41
0.8552
13.954
8.11
S-4
Stride Conv
21.36
0.8534
13.954
8.11
Max pooling *
21.46
0.8584
13.753
7.84
Average pooling
21.34
0.8481
13.754
7.84
Gaussian
21.79
0.8612
16.102
8.54
LPF 20.74
0.8124
16.102
8.54
Better zoom in.
Ours
23.28
0.8708
13.827
7.87
Table 5: The effectiveness and efficiency comparison be- tween our FouriDown and other down-sampling methods in low-light enhancement task.
Figure 9: The spectrums of different- scale images placed under the same frequency coordinate system.
6 Conclusion
In our study, we revisit the spatial down-sampling techniques and anti-aliasing strategies from a Fourier domain perspective, recognizing their reliance static high and low frequency superposing. As a result, we propose a novel approach (FouriDown) to learn a learnable frequency-conted interplay among high and low frequeeniess during down-sampling Moreover, this work is the first exploration of dynamic frequency interaction in down-sampling. The FouriDown is designed based on the signal sampling theory, so it is convenient replace most of current down-sampling and anti-aliasing techniques. Extensive experiments demonstrate the performance improvements across a variety of vision tasks.
Ultimately, we believe that down-sampling erucial research direction the future. It allows for network design at lower resolution, significantly reducing the computational overhead and enabling weighttgg models.
Broader Impact
This work further exploits the potential of down- sown-sampling in the Fourier domain and offers new perspective that frequency band shuffling and superposing for future research of down-sampling, Down-sampling techniques can potentially make the future model development more efficient and effective, beneficial for various machine learning and AI applications. Nonetheless, the efficacy of our method could be source of concern not properly utilized, especially in terms of the safety of real-world applications. To alleviate such concerns, we plan to rigorously investigate the robustness and effectiveness of our approach across a more diverse spectrum of real-world scenarios.
Acknowledgments
This work was supported by the JKW Research Funds under Grant 20-163-144L--001-0000 and the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.

---

References
[1] Yongbing Zhang, Debin Zhao, Jian Zhang, Ruiqin Xiong, and Wen Gao. Interpolation-depeneenn image downsampling. IEEE Transactions Image Processing, 20(11):3291-3299 2011.
[2] Ding-Xuan Zhou. Theory of deep convolutional neural networks: Downsampling. Neural Networks, 124:319-327, 2020.
[3] Christian Bailer, Tewodros Habtegebrial, Didier Stricker, et al. Fast feature extraction with cnns with pooling layers. arXiv preprint arXiv:1805.030% 2018.
[4] Leif E Peterson. K-nearest neighbor. Scholarpedia, 4(2):1883 2009.
[5] Michael Lindenbaum, Shaul Markovitch, and Dmitry Rusakov. Selective sampling for nearest neighbor classifiers. Machine learning, 14:125-152, 2004.
[6] Edward A Patrick and Frederic P Fischer III. generalized k-nearest neighbor rule. Information and Control, 16(2):128-152, 1970.
[7] Richard Zhang. Making convolutional networks 2019.
again. ArXiv, abs/1904.11450
[8] António H. Ribeiro and Thomas Schon. How convolutional neural networks deal with aliasing. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 2755-27555 2021.
[9] Cristina Nader Vasconcelos, H. Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin. An effective anti-aliasing approach for residual networks. ArXiv, abs/2011.10675. 2020.
[10] Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. International Journal Computer Vision, 131:67-811 2020.
[11] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11400-11410 1410, 2022.
[12] Shengju Qian, Hao-Chiang Shao, Yi Zhu, Mu Li, and Jiaya Jia. Blending anti-aliasing into vision transformer. In Neural Information Processing Systems, volume 34, pages 5416-5429 2021.
[13] Hagay Michaeli, Tomer Michaeli, and Daniel Soudry. Alias-free convnets: Fractional shift invariance via polynomial activations. ArXiv, abs/2303.08085. 2023.
[14] Tsung- Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. n Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 936-944, 2016.
[15] Naishan Zheng, Man Zhou, Yanmeng Dong, Xiangyu Rui, Jie Huang, Chongyi Li, and Feng Zhao. Empowering low-light image enhancer through customized learnable priors. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12559-
12569, October 2023.
[16) Xiang Chen, Jinshan Pan, Jiyang Lu, Zhentao Fan, and Hao Li. Hybrid cnn-transformer feature fusion for single image deraining. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 378-386, 2023.
[17) Qi Zhu, Naishan Zheng, Jie Huang, Man Zhou, Jinghao Zhang, and Feng Zhao Learning spatio-temporal sharpness map for video deblurring. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[18] Qi Zhu, Man Zhou, Naishan Zheng, Chongyi Li, Jie Huang, and Feng Zhao. Exploring temporal frequency spectrum in deep video deblurring. In Proceedings of the IEFF/CVE International Conference Computer Vision, pages 12428-12437, 2023,
10

---

[19] Qi Zhu, Zeyu Xiao, Jie Huang, and Fengmei Zhao. Dast-net: Depth-aware apatio-tempont network for video deblurring. Proceedings of the IEEE/CVF International Conference on Multimedia and Expo, pages 1-6, 2022.
[20] Naishan Zheng, Jie Huang, Qi Zhu, Man Zhou, Feng Zhao, and Zheng-Jun Zha. Enhancement by your aesthetic: An intelligible unsupervised personalized enhancer for low-light images. In Proceedings of the 30th ACM International Conference Multimedia, pages 6521-65222 2022.
[21] Naishan Zheng, Jie Huang, Man Zhou, Zizheng Yang. Qi Zhu, and Feng Zhao. Learning semantic degradation-awee aware guidance for recognition-drive unsupervised low-light image enhancement. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 3678-3686, 2023.
[22] Jie Huang, Feng Zhao, Man Zhou, Jie Xiao, Naishan Zheng, Kaiwen Zheng, and Zhiwei Xiong. Learning sample relationship for exposure correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9904-9913, June 2023.
[23] Naishan Zheng, Jie Huang, Feng Zhao, Xueyang Fu, and Feng Wu. Unsupervised underexposed image enhancement via self-illuminated and perceptual guidance. IEEE Transactions on Multimedia, 2022.
[24] Huikang Yu, Jie Huang, Yajing Liu, Qibang Zhu, Man Zhou, and Fengmei Zhao. Source-free domain adaptation for real-world image dehazing. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
[25] Wei Yu, Qi Zhu, Naishan Zheng. Jie Huang, Man Zhou, and Fengmei Zhao. Learning non- uniform sampling for ultra-high highh-hihihhiiiionn image enhancement. Proceedings of the 31st ACM International Conference Multimedia, 2023.
uniform-sammling
[26] Julia Grabinski, Steffen Jung, Janis Keuper, and Margret Keuper. Frequencyloweut pooling plug& & play against catastrophic overfitting. ArXiv, abs/2204.00491, 2022.
[27] Julia Grabinski, Janis Keuper, and Margret Keuper. Fix your downsampling asap! be natively more robust via aliasing and spectral artifact free pooling. ArXiv, abs/2307.0988 2023.
[28] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874 1874-1883, 2016.
[29] Heewon Kim, Myungsub Choi, Bee Lim, and Kyoung Mu Lee. Task-aware image downscaling. In European Conference on Computer Vision, 2018.
[30] Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour. Learning strides in convolu- tional neural networks. ArXiv, abs/2202.01653, 2022.
[31] Chittesh Thavamani, Mengtian Li, Francesco Ferroni, and Deva Ramanan. Learning to zoom and unzoom. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 008865095,.00 2023.
[32] Dmitrii Marin, Zijian He, Péter Vajda, Priyam Chatterjee, Sam S. Tsai, Fei Yang, and Yuri Boykov. Efficient segmentation: Learning downsampling near semantic boundaries. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2131-2141, 2019.
[33] Babak Ehteshami Bejnordi, Amirhossein Habibian, Fatih Murat Porikli, and Amir Ghodrati. Sal- isa: Saliency-based input sampling for efficient video object detection. ArXiv, abs/2204.02397, 2022.
[34] Wenhan Yang Jiaying Liu Chen Wei, Wenjing Wang. Deep retinex decomposition for low-light enhancement. In Proceedings of the British Machine Vision Conference, 2018.
[35] Jiang Hai, Zhu Xuan, Ren Yang. Yutong Hao, Fengzhu Zou, Fang Lin, and Songchen Han. R2rnet: Light image enhancement via real-low to real-normal network. arXiv preprint arXiv:2106.1 14501, 2021.
11

---

[36] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3291-3300, 2018.
[37] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to percep- tual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3063-3072, 2020.
[38] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3883-3891, 2017.
[39] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14821- 14831, 2021.
[40] Shuochen Su, Mauricio Delbracio, Jue Wang. Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. Deep video deblurring for hand-held held cameras. Proceedings of thelEEE Conference on Computer Vision and Puttern Recognition, pages 237-246, 2017.
[41] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In IEEE International Conference Computer Vision, 2019.
[42] Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1692-1700, 2018
[43] Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1 1586-1595 2017.
[44] Syed Waqas Zamir, Aditya Arora, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hiug-Hsuaan Yang. and Ling Shao. Learning enriched features for fast image restoration and enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:1934 2020.
[45] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5728-5739, 2022.
[46] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2157-2167, 2020.
[47] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-basel multi- scale network for image dehazing. In Proceedings of the IEFF/CVE international conference on computer vision, pages 7314-7323, 2019.
[48] Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6849-68868 0000-6877720 2019.
[49] Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics, 36:1
12, 2017.
12

---

Appendix A: Proofs of Theorem-1 and Theorem-2
Proof of Theorem-1: Shuffling and Superposing
To model the relationship between (x) and f(), we stand on their derived continuous signal g(x) by specific sampling function. Note that the sampling functions *a)()) is defined the sum of infinitely impulses units separated by AT intervals:
s(x, AT)
Σ - -(T=nAT).
(4)
Based on the Eq. (4), f()) and down-sampling (a) can be represented
f(x) =
f'(x) = (( 2A7).
(5)
According to the Fourier transform and convolution theorem, the aforementioned spatial sampling can be expressed in the Fourier domain
F(u) = G(u) * S(u, AT) = 1
(6)
00
AT Σ G(T) ----- AT dr
AT
u
AT
where G(u) and Suu, AT) are the Fourier transform of g(x) and s(x, AT). From Eq. (6), it can be observed that the spatial sampling introduces the periodicity to the spectrum and the period AT
Note that the sampling rates of f(x) and f'(x:) are Ω2 and f2 the relationship between them can be written as
1
Ω2
Ω
Ω2
(7)
AT
2AT
Before down-sampling, to focus on the following down-sampling operation, we assume that f(x) adheres the Nyquist sampling theorem, which implies that Umar S2
After down-sampling, according Nyquist sampling theorem, whole sub-frequency band is limited in (0, 9rr ). Moreover, the resulted band is the superposition of two original bands, which denoted as 2
F'(w) = S(F()) P())),
(
where û, are the high frequency above the sampling rate and the low frequency below the sampling rate, respectively, and & is superposition operator.
In the positive sub-band, where (0, fx), á and should satisfy
u E (0, Ω and à
( Umaz).
(9)
According to the aliasing theorem, the high frequency à is folded back to the low frequency:
ū (k 1)
(k + 2)
<
(10)
2
2
2
where k: = 1,,3,5.. and ä is folded results by u.
According to Eq. (9) and Eq. (10), have
Ω a, and u
Ur ( 2
Ωr
Umar
(11)
Then, according to Eq. (8) and Eq, (11), we attain
F(u) (F(()), F( u))
if (0, 8x =umax)
F'(u)
(12)
if L
Umaz: Px
13

---

According to Eq. (6), F(u) is symmetric with respect to 9z. 2
2) = F(u+ 2 F( Q
(13)
2
2
Further, P(u + 22) =0 when u E (0, 2s
2 Umar)) Upon Eq. (13), can uniform Eq. (12) as
F((u) S(F(u), F(u (2o)) when (0, Pz 4
(14)
The visualization of the aforementioned proof process is depicted in Figure 2(a). (2) the negative sub-band, where u E ( fzz ). different from (1), and should satisfy
2g P
Gz Umax: 4
4 and à E E
(15)
Similarly, we can proof F(u) the negative sub-band as well.
2 when u E
Or Qr
Fu) = S(F(u), F(u+
(16)
2
2
Combined with Eq. (14) and Eq. (16), we obtain
Qu 22,, when (0, Ωr F'(u) = S(F(u), F(u
(17)
2
2
Proof of Theorem-2: Static Superposing. According to Eq. (7) and Eq. (6), we can deduce that the amplitude of F is half that of F.
Therefore, can write Fu( in the axis as,
Q Px when u E (0, F'(u) 2 F(u) + F(u + 2
(18)
2
Upon the dual principle, we can prove ("(v) in the whole sub-band
Pz 2,0) F(u, Ly + F(u 1 4 F(u,o) + F(u+ +
u + fl
F((())))(((((
2
(19)
where (0, 2)) ) (0, )
Appendix B: Revisiting previous down-sampling manners in the FouriDown
In the main body, we introduce the FouriDown framework which can simultancously adjust the stride of down-sampling and the characteristies of frequency interaction. Moreover, in the manuscript, we prove that the operator with a stride of can be simplified to a fixed frequency weighting. i.e. averaging. However, this kind of weighting brings aliasing, hence the need for anti-aliasing methods such as ideal low-pass filters and Gaussian filters, which put focused weighting on frequencies. In this section, we revisit these anti-aliasing methods within the FouriDown framework and discover that their weighting methods remain fixed, In other words, within our framework, it's possible to realize the aforementioned anti-alian unti-aliasing methods by simply changing specific parameters as shown in Figure 10 and 11.
Appendix C: Stride Extension of Theorem-1 and Theorem-2
In the manuscript, we propose FouriDown which primarily focuses on analyzing frequency interac- tions in the case of stride=2. In this section, we extend the Theorem-2 to arbitrary integer strides resulting the following theorem. The proof for these theorems follow similar logic that detailed in the main body, and is therefore omitted for brevity.
14

---

Shuffling
Superposing
Spli into 16 putches
Wo. Wz. W3
$
IDFT
0.25, 0.25, 0.25, 0.25
"Aliasing effect
(a) Nearest-2x
DFT
shuffle
Wo W2, W3
0.25 0.00, 0.00, 0.00
"Ringing effecct
(b) Ideal low-pass -2x
Wo W2. W3
SP
Weighted superposing
DFT
Diserete Fourier Transform (DFT)
Guassian Distribution
"Highefropuency loss'
IDFT
Inverse Discrete Fourier Transform (IDFT)
(c) Guassian-22
Figure 10: The achievements of different down-sampling manners the FouriDown by simply changing specific parameters.
(1) Nearest downsampling (no interpolation
"Ringing"
Process
(2) Idcal low-pass filter
"Iligh-freqpency Loss"
Piocess (1)
Discrete Fourier Trunsform (DFT)
Itaustons (IDFT)
Guassian filter
Figure 11: Revisiting different down-sampling methods from the perspective of spectrum.
Stride Extension of Theorem-1. In the condition of strides, illustrate with I-dimensional signal, the high and low frequency superposition in the down-sampling can be formulated as
= S(F(u), k(ll))
when E (0, Gx 2 )
(20)
where S is superposing operator and k; = -,,.., 1.
Stride Extension of Theorem-2. For image, the spatial down-sampling operator with strides can be equivalent to dividing the Fourier spectrum into s x equal parts and superposing them averugely by a factor
Given that Downg is s-strided down-sampling operator and IDFT is inverse discrete Fourier trans- form, we have
15

---

Downsampling Strides 2)
Downsampling Strides x)
0g
kfg
fg
2
5
Fy
R2
k0
fy
a
11
14
1, 1
Backgrmund to walch
fg
fg
Qa
1luy 2
2
Figure 12: Illustration of theorem extension from strides=2 to strides=s. Due to visualization space limitation, is taken in the figure. The background region has no meaning, just to show more clearly.
8-18-1 ΣΣ ==
(fwn,(f(z,)) 8)) IDFT
Fu
(21)
Based above extension, the down-sampling of 3 strides could be presented in Figure 12.
Appendix D: More Qualitative comparison.
Due to the limited space, we only report the visual results of the low-light enhancement task in main manuscript. We report more visual results in the supplementary materials. As shown, integrating the FouriDown with the original baseline [36, 37, 48, 49, 39, 45, 44, 46, 47] achieves more visually pleasing results shown in Figure 13 14 15 16 17.
16

---

Low-light image
DRBN
(c) DRBN with Fouril
(d) Ciroud truth
(n) Low-light image
(b) SID
(c) SID with FouriDown
(d) Groud tuuth
Figure 13: Visual comparison the low-light enhancement task.
(a) Hazy image
(b) MSRDN
(c) MSBDN with FouriDown
Groud truth
(a) luzzy image
(b) GridNet
(c) GridNet with FouriDown
(d) Groud truth
Figure 14: Visual comparison on the dehazing task.
17

---

(a) Input image
(b) HDRNer
(c) HDRNet with FouriDown
(d) Groud truth
(a) Input image
(b) UPE
(c) UPE with FouriDown
(d) Groud truth
Figure 15: Visual comparison on the UHD-enhancement task.
(a) Noisy image
(b) Restormer
(c) Restormer with FouriDown
(d) Groud truth
(a) Noisy image
(b) MIRNet
(c) MIRNet with FouriDown
(d) Groud truth
Figure 16: Visual comparison on the denoising task.
18

---

(a) Blurry image
(b) DecpDeblur
(c) DeepDeblur with FouriDown
(d) Groud truth
(a) Bluryy image
(b) MPRNet
(c) MPRNet with FouriDown
(d) Groud truth
Figure 17: Visual comparison the deblurring task.
19


Reviews:
[{"review_id": "U3mSr7tFE1", "reviewer": "Reviewer_e5Db", "summary": "This paper presents a downsampling method for neural networks in the frequency domain.\nThe proposed downsampling consists of discrete Fourier transform, shuffling, superposing, and inverse discrete Fourier transform.\nThe shuffling shuffles the patches in Fourier domain and the superposing adds the patches with learned and adaptive weights.\nExperimental results present that the proposed method outperforms Gaussian filtering and the ideal low-pass filtering in frequency domain for neural networks on various computer vision tasks.\n", "questions": "Please see the weaknesses.", "limitations": "Please see the weaknesses.", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "Downsampling in frequency domain is an interesting topic. \nThe proposed method and the experimental results are reasonable.\nPerformance improvements on enhancement tasks are impressive.", "weaknesses": "Experiments lack comparisons and analyzes to convince the effectiveness of the proposed method.\n\nThe proposed method uses additional convolution layers than compared methods, so the computational overheads (FLOPs and latency) should be compared.\nIt is worth noting that DFT and IDFT also require additional overheads compared to the original networks.\n\nComparisons to numerous image downscaling methods (bilinear, bicubic, etc) are missing.\nIn particular, pixel-downshuffling [A] with convolution performs a similar operation to the proposed method without Fourier transform.\n\nFeature map visualization in frequency domain is missing.\nFeature maps of convolution layers usually contain high frequency edges, thus it is curious how the proposed method works in the feature space.\n\nAblation study of additional convolution layers is missing.\nSimple operations instead of the convolution layers such as max or mean will be an interesting ablation study.\n\n[A] Channel Attention Is All You Need for Video Frame Interpolation, AAAI 2020\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we appreciate your acknowledgment of the topic and methodological design of our work. Complete and meaningful of the comparisons and analysis has been complemented by visualizations and tables in the rebuttal phase. If this paper is accepted, we commit to meticulously revising and organizing the experimental and analysis sections in the revision.\n1.\t** DFT overheads **:\nFirst, following [1,2], we perform DFT using \\textit{torch.fft.fft2(x)} with few extra overhead. To further illustrate this point, we conduct a toy experiment shown in the following table. It could be seen that although the FFT and its inverse traditionally have a computational complexity of O(N log N), this part has been accelerated by pytorch library and does not occupy any FLOPs. Therefore, the consumption of time is also very little\n\n| SID          |   | FLOPs(G) | Time(s)  |   \n|--------------|---|----------|----------|\n| wo FFT/IFFT  |   | 13.753   | 0.0134   |   \n| w FFT/IFFT   |   | 13.753   | 0.0139   |   \n| w 4*FFT/iFFT |   | 13.753   | 0.0146   |  \n\n[1] Intriguing Findings of Frequency Selection for Image Deblurring. In AAAI, 2023.\n[2] Deep Fourier-Based Exposure Correction Network with Spatial-Frequency Interaction. In ECCV, 2022.\n\n2.Comparison with Traditional Downscaling Methods. Due to space limitation, in the supplementary, we recognize the importance of comparing our method with traditional techniques like nearest/bicubic/bilinear/lanczos downsampling shown in Fig 5. As depicted in Supp. Line 86-90, it can be observed that when the down-sampling factor is exactly divisible by the original image resolution, no interpolation actually takes place. In such cases, the bicubic/bilinear/lanczos methods mentioned above are equivalent to the nearest neighbor method, which appears same static weights [0.25,0.25,0.25,0.25] in FouriDown.\nFurthermore, we include comprehensive comparisons with these methods as shown in the above table. \n\n2.Pixel-shuffling and proposed spectral-shuffling:  We are aware of the pixel-downshuffling method you referred to in [A]. However, it is hard to agree with your opinion. Specifically, there are following distinct differences.\n\n(a)Domain of Operation. Our method operates in the frequency domain, leveraging the unique properties of Fourier transforms. Pixel-downshuffling operates in the spatial domain, and the nature of these domains and their inherent properties can lead to different results.\n\n(b)Distinct Shuffle Patterns. It is apparent that our shuffle mode is specifically designed for spectrum and is completely different from the pixel-shuffle[A] mode which relies on nearby sampling. To illustrate the point better, we replace our spectrum-shuffle with pixel-shuffle after FFT as evidenced in following table, which leads to bad results.\n\n| | | |  | |   \n|--------|--------------------|---|-------|---------|\n| MPRNET | FouriDown  |   | PSNR  | SSIM    |  \n|  | wo shuffle         |   | 25.43 | 0.8144  | \n|   | w pixel-shuffle    |   | 23.43 | 0.7954  |  \n|  | w spectral-shuffle |   | 30.31 | 0.8996  |   \n\n (c)White-box vs. Black-box Approach. Different from the pixel-downshuffle[A] operating as a 'black box', our shuffle criterion is derived from rigorous signal theory, ensuring a more interpretability and expansibility grounded process.\n(d)Potential Impact for FFT Developing. We believe the spectrum-shuffle isn't merely a technical novelty, it has significant implications for the develop of the Fourier theory in the low-level vision task.\nAbove all, the Fourier domain offers insights and opportunities that are not readily available in the spatial domain. By working in this domain, our method provides a novel solution for adaptive aliasing issue and potential improvements in downsampling.\n\n3.Feature map visualization. Thank you for your suggestions. We have included visualizations of the feature maps and their corresponding spectra in the PDF file attached to the rebuttal. Notably, the features extracted by FouriDown demonstrate a significantly larger response than other methods, attributing this to the unique global modeling mechanism we employ in the frequency domain. Moreover, to delve deeper into the reasons for its efficacy, we have also compared the spectral images of the features. The results show that the spectrum using FouriDown exhibits an exceptionally smooth response across both high and low frequencies.\n\n4.Max or Mean Ablation Study. Indeed, this is an interesting experiment which highlights the extensibility of the FouriDown framework. In fact, we have already conducted this ablation study earlier, but due to space constraints, it was not included. As shown in the following table, according to Theorem 2, when averaging is used, FouriDown essentially becomes equivalent to the stride=2 method. The Max approach is indeed an interesting idea, however, regrettably, it hasn't been so effective. We hope to find the scenarios where the Max method is more applicable in the future.\n|            |        |   |       |         |   \n|------------|------------|---|-------|---------|\n| DeepDeblur | FouriDown  |   | PSNR  | SSIM    |  \n|            | Max        |   | 29.32 | 0.8815  |   \n|            | Average    |   | 29.34 | 0.8817  | \n|            | Conv layer |   | 29.44 | 0.8856  |  \n"}, {"review_id": "I83DyuNAKP", "reviewer": "Reviewer_fgKa", "summary": " - This paper explores a new method for downsampling, by factoring it into shuffling and superposing. Firstly, the authors provide an understanding of the aliasing in deep neural networks from a spectrum view. \n\n - FouriDown, a unified downsampling approach with the learnable and context-adaptive down-sampling operator, is proposed. This method is made of four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform.\n\n - Extensive experiments on image de-blurring and low-light image enhancement are conducted, which consistently show that FouriDown can provide significant performance improvements.", "questions": " - What are the additional computation costs of FouriDown, compared to other downsampling methods?\n\n - Is there any difference between applying FouriDown to the image and feature level? It's interesting to see the comparisons with other downsampling methods in-between, as images and features in DNN have differences in frequency distribution.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": " - Extensive qualitative and quantitative evaluation results on scale-sensitive tasks are conducted, which show the advantage of the proposed downsampling method over previous ones.\n\n - This work first provides an exploration of the aliasing issue in neural networks from a spectrum perspective, which will facilitate a deeper understanding of the aliasing issue.\n", "weaknesses": " - Additional details, such as the computation costs and the respective results on images and features, should be provided. (See Question)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **FouriDown Computation**:\nThank you for your question. The FouriDown incorporates the Fourier transform and its inverse, which traditionally have a computational complexity of \\(O(N \\log N)\\).However, this part has been accelerated by pytorch library and does not occupy any FLOPs. Moreover, there are additional computations related to the spectral-shuffle operation and filtering in the frequency domain. For spectral-shuffle, the operations such as tensor remodeling and composition do not require additional Flops and the number of parameters. Moreover, the additional convolution layers contain very small flops and parameters. Finally, the computational cost of our algorithm compared to other downsampling methods is shown in the Table above. See attached file for more experiments.\nNote that although the shuffle process doesn't consume any parameters or FLOPs, implementing this step in software does take extra time. We will accelerate this step by CUDA and release the code in the future. We will further elucidate these computational trade-offs in the revised manuscript.\n\n2. Thank you for raising this insightful question. FouriDown is a downsampling operator applied to the feature layer that relies on convolutional parameters. The learning downsampling process is similar to pixel-unshuffling. As a result, it may not be suitable for image domain resizing, such as bicubic and bilinear. However, it is meaningful to compare the feature map visualizations and their spectrum of different downsampling methods. Please refer to Figure 1-3 in the attached file. It can be seen that the spectrum following FouriDown obtains the outstanding smooth response in high and low frequencies.\n"}, {"review_id": "jReq0B42wt", "reviewer": "Reviewer_k1g9", "summary": "This paper proposes a downsampling method in Fourier domain (FouriDown) for several low-level vision tasks and an image classification task. FouriDown is supported by theoretical background proven by the authors. While this work improves multiple vision tasks when compared to previous anti-aliasing downsampling methods, it lacks to truly demonstrate its superiority over existing other downsampling approaches due to not well-organized experiments. Moreover, exploration of other architecture is insufficient. So, despite the robust and well-presented theorems the contribution of this paper is limited.", "questions": "How about ImageNet 1K classification or object detection? Did you conduct other experiments about high-level tasks? Because the performance gains are more notable in the case of CIFAR classification than low-level tasks, focusing mainly on high-level vision recognition tasks seems better.", "limitations": "See weakness.\n\n\nI think the additional tables, figures, and description made this paper more comprehensive to be published at NeurIPS 2023. Therefore, I would like to recommend this paper to be accepted, and will change my rating from borderline reject to weak accept.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The explanation and background of this paper are theoretically robust. Why the shuffling and superposition operations of FouriDown are constructed in that manner is clear. The learnable downsampling weight, which is unlike previous anti-aliasing methods, is technically sound.\n\nMany low-level vision tasks have been improved by FouriDown. The tables showing experimental results are easy to understand (however, some issues of experimental composition exist as mentioned in weakness). In addition to low-level tasks, the results that an image classification task is also improved makes FouriDown more robust.", "weaknesses": "1. Most importantly, this work focusing only on downsampling method has limited contribution for being published at this conference. If this paper has studied other architectures (e.g., CNN’s kernel or Transformer’s self-attention) and secondarily introduced the effectiveness of FouriDown, the novel downsampling method could have delivered more importance for deep learning field. This does not mean that FouriDown is not effective. Instead, more urgent architecture designs could be considered and provided in this paper. Moreover, a study regarding how to effectively and efficiently address the hierarchical downsampled multi-scale feature maps, where downsampling itself loses informative details, is suggested to be conducted.\n2. While replacement of static superposition to learnable one is interesting, the composition of this learnable weight is not sophisticated enough to be considered as a novel idea. It just follows standard 1x1 convolution, activation function, and softmax. It should be carefully considered.\n3. Tabs.1,2,3,4,5,6 should include some factors with respect to efficiency: inference time, the number of parameters, etc.\n4. The “Original” downsampling methods in Tabs.1,2,3,4,5,6 are not presented. The authors, therefore, should present other downsampling methods that are not anti-aliasing approaches. For example, pixel-unshuffle, 2x2 learnable CNN (with stride=2), max-pooling, or other interpolation can be considered.\n5. The summation operation along subsample dimension (indicated as “4”) in the Algorithm 1 can be better demonstrated by another ablation study. As alternatives to summation, there exists average operation or weighted summation for the “4”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Recently, many recent studies have been centered on downsampling [reference] as stated in the related works, thus we believe the downsampling is sufficient as a crucial research topic. Our work also has ample contributions for the following reasons:\n(1)\tOur study is the first to revisit the deep downsampling operator from a frequency domain perspective.\n(2)\tThis approach is particularly challenging due to the significant disparity between the spatial and frequency domains (details see W2). \n(3)\tIn this work, we innovatively pinpoint that previous downsampling methods were mostly constrained by the issue of static frequency aliasing.\nTherefore, this work comprises ample theoretical motivation and presents a challenging solution from an unprecedented perspective, which aligns well with the preferences of this conference.\n\nMeanwhile, we appreciate your sincere suggestions for hierarchical multi-scale exploration. In fact, our work essentially taps into the causes of detail information loss. Moreover, in the supplementary material, we've discussed different scale relationships in FouriDown, which might provide insights for addressing this issue. We will explore it in the future. \n\n2. The design of the learnable weight is both carefully-considered and theory-robust.\nIt seems there might be some misunderstandings regarding the novelty of our work. We don’t merely propose a simple convolution layer replacement. Importantly, before the convolution operation, a pioneering and theoretically based spectral-shuffle in Figure 3 is meticulously incorporated, ensuring the feasibility of learnable superposition.\nFurther, replacing the convolution layer directly without the shuffle would result in inferior performance, as evidenced in the following table. Hence, we have rigorously considered and trialed numerous methods to realize dynamic superposition. In response to your concerns, we'd like to highlight some insights as following:\n\n|        |                    |   |       |         |   \n|--------|--------------------|---|-------|---------|\n| MPRNET | FouriDown          |   | PSNR  | SSIM    |  \n|        | wo shuffle         |   | 25.43 | 0.8144  | \n|        | w pixel-shuffle    |   | 23.43 | 0.7954  |  \n|        | w spectral-shuffle |   | 30.31 | 0.8996  |   \n\n\n(a) Designing learnable superposition is inherently challenging.\n   (i) The pronounced nonlinear discrepancy between spatial and frequency domains makes learning downsampling directly in the frequency domain via convolutional layers especially arduous. As depicted in above table, solely relying on some nonlinear activation functions is nearly inadequate to approximate the functional relationships between spatial and frequency domains. To relieve the challenging nonlinear function, we introduce spectral-shuffle, based on Theorem 1 and Theorem 2, thus offering a viable approach for realizing learnable weights via CNNs.\n   (ii) The convolution operation is typically ill-suited for spectral domains. Conventional convolution operations are primarily designed to operate in the spatial domain, capturing local features and receptive fields. However, the frequency spectrum doesn't exhibit similar biases. Most prior works [1,2] use 1x1 convolutions that share weights across the entire spectrum, essentially learning channel-wise weights. Such a design has limited the evolution of Fourier theory in deep learning. Therefore, how to extract \"pseudo-local\" spectral features using convolution operators remains a challenge. Fortunately, our work uncovers the 'pseudo-local' relationship within the downsampling spectrum, denoted as $[F(u), F(u + \\frac{\\Omega_x}{2})]$, as described in Theorem 1. Based on the derivation, we achieve a 'pseudo-local’ extraction from spectral features via convolution operators, fostering future applications and growth of Fourier theory in deep learning.\n[1] Intriguing Findings of Frequency Selection for Image Deblurring. In AAAI, 2023.\n[2] Deep Fourier-Based Exposure Correction Network with Spatial-Frequency Interaction. In ECCV, 2022.\n\n (b) Simplicity with effectiveness epitomizes elegance.\n   (i) \"Entities should not be multiplied without necessity\" – Occam's razor. Resolving existing issues in a straightforward and principled manner is intrinsically significant. While more complex implementations exist, we believe they aren't particularly necessary.\n   (ii) General algorithm is usually simple. As shown in experiments section, thanks to our simple yet potent design, the FouriDown exhibits commendable versatility across various tasks (both low-level and high-level) and diverse architectures (both CNNs and Transformers). Such a design resonates with the contemporary philosophy of General Artificial Intelligence (AGI). \n\nIn fact, according to Theorem 2, within our FouriDown framework, the average operation is essentially equivalent to the stride=2 operation, meaning it aligns with the baseline of many existing methods. The weighted summation is an adaptive weight prediction we introduced. The ablation study for average versus weighted summation is presented as follows. \n| | FouriDown    |   | PSNR  | SSIM    | \n|------------|--------------|---|-------|---------|\n| DeepDeblur  | Average      |   | 29.32 | 0.8817  |  \n|            | Weighted sum |   | 29.44 | 0.8856  | \n| MPRNET     | Average      |   | 30.11 | 0.8956  |   \n|            | Weighted sum |   | 30.31 | 0.8996  |  \n\n3.Due to time constraints, we trained the performance of YOLO v5 on the COCO subdataset and found the improvement, as shown in the table below.\n| YOLOv5    | AP   | AR    | \n|-----------|------|-------|\n| Baseline  | 23.5 | 36.8  | \n| FouriDown | 25.3 | 32.6  |   \nAs mentioned, FouriDown possesses good versatility. However, since this paper primarily focuses on addressing low-level tasks, we hope more experiments will be conducted in the future to further explore its potential in high-level tasks.\n\n\n\n\n"}, {"review_id": "yIqH7jx8do", "reviewer": "Reviewer_rAiK", "summary": "In this research, the authors re-examine the spatial down-sampling mechanism and analyze the biased impacts that arise from the static weighting strategy used in previous methods. To overcome this limitation, this work proposes a new down-sampling paradigm called FouriDown, which incorporates frequency-context adaptive superposing. This component can be seamlessly integrated into existing image restoration networks and significantly improve performance.", "questions": "1. The authors present a novel down-sampling method based on Shannon's sampling theorem. As feature maps commonly comprise multiple channels, it would be interesting to investigate the interaction patterns that emerge among these channels.\n2. Based on the diagram provided, the FouriDown method seems to involve information transfer onto the channel followed by channel contraction, similar to the approach used in pixel-shuffle down-sampling. It is crucial to conduct an analysis that specifically explores and demonstrates the advantages of the FouriDown method over pixel-shuffle down-sampling. This comparative analysis will provide valuable insights into the superiority of the proposed method.\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This work provides an in-depth exploration of the down-sampling operator, offering a novel perspective that has the potential to significantly advance future research related to operators. Furthermore, the motivation behind this work is clear and well-founded, as evidenced by the compelling depiction in Figure 2 along with the accompanying theoretical proof. Importantly, this research showcases considerable potential across diverse visual tasks.", "weaknesses": "1. To better emphasize the main content of the paper, it is suggested to move the proof of formulas to the supplementary material. Furthermore, the experimental section of the main text could benefit from incorporating the information from Section 1 in the supplementary material, which illustrates the superior extensibility of this framework over other down-sampling methods.\n2. Although the supplementary material includes numerous excellent visual results to showcase the superiority of the proposed method, it is recommended to supplement the updated version with visualizations of the down-sampled feature maps.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. Thank you for your sincere suggestion. We agree with your suggestion and will move the proof of formulas to the supplementary material. This will allow us to incorporate more relevant experiments and analysis into the main text, highlighting the superiority of our framework over other down-sampling methods.\n\nW2. Thank you for pointing this out. We perform the comparisons with visual representations of down-sampled feature maps to better illustrate the effectiveness of our method in Figure 1-3 in the attached file. In the revised version, we will incorporate them to provide a clearer understanding of how our method performs in comparison to others.\n\nQ1. In this work, the FouriDown doesn't involve interactions among channels, they operate independently, but they do interact in the spectral domain. This is analogous to the Fourier transform, where channels also operate independently when transforming feature maps. Thanks for your insightful problems and how to establish relationships between channels in the Fourier transform is an intriguing question to explore in the future.\nMoreover, while channels are processed independently during FouriDown, their interactions within convolutional networks are still related. In low-level vision tasks, channel interactions might occur between high-frequency texture information and low-frequency color information. Through our research, we find that while there's no explicit channel interaction during down-sampling, there's a manifest high-low frequency interaction occurring in the spectral domain. This could be one of the reasons why FouriDown is effective in various low-level vision tasks.\n\nQ2. Indeed, at a superficial glance, FouriDown might appear to have similarities with pixel-shuffle down-sampling in terms of channel information transfer and contraction. However, there are distinct differences:\n1. **Domain of Operation**: FouriDown operates in the spectral domain, leveraging the properties of the Fourier transform. In contrast, pixel-shuffle primarily works in the spatial domain.\n2. **Inherent Mechanisms**: The shuffle process in FouriDown is rooted in signal theory and is meticulously designed for convolution operations in the spectral domain. Pixel-shuffle, on the other hand, utilizes a more straightforward neighborhood sampling mechanism.\n3. **Potential Impact**: We believe the spectrum-shuffle isn't merely a technical novelty, it has significant implications for the develop of the Fourier theory in the low-level vision task.\nIn light of your feedback, we'll bolster our paper with a more in-depth comparative analysis between FouriDown and pixel-shuffle down-sampling. This will elucidate the specific advantages and the overarching superiority of our method.\n"}, {"review_id": "TTHmDdlzRO", "reviewer": "Reviewer_V6WG", "summary": "This paper introduces a novel down-sampling operator in the Fourier domain, specifically tailored to mitigate the common problem of frequency aliasing encountered during down-sampling. The operator's efficacy has been assessed across diverse computer vision problems, consistently yielding improved performance in these tasks.", "questions": "Please see the weaknesses", "limitations": "The authors mentioned the limitations in section 5 and addressed the potential broader impact of their work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The authors address the long-standing issue of frequency aliasing in signal processing, with a specific focus on image down-sampling. Their meticulously crafted design offers a compelling and well-founded solution to mitigate frequency aliasing.\n\n2. The authors substantiate their claims in the paper with comprehensive and rigorous theoretical proofs, providing a strong basis for their proposed approach.\n\n3. The proposed FouriDown operator serves as a plug-and-play solution, seamlessly integrating into existing networks. Furthermore, it consistently enhances the performance of the underlying models across various tasks.\n\n4. The writing in the paper is clear and the explanations are thorough, ensuring a comprehensive understanding of the presented concepts.", "weaknesses": "1. While the quantitative results are satisfactory, it would be beneficial to include additional efficiency analysis, such as providing the parameters of the FouriDown operator, in the form of a table showcasing relevant metrics. \n\n2. Although the FouriDown operator is currently limited to replacing the stride 2 down-sampling operator, it would be valuable to explore whether the authors have developed other variants capable of handling more general scenarios. \n\n3. It would be advantageous to present more visual results while condensing reducing the length of the theorem proofs. \n\n4. Figure 5 captures attention due to its intriguing nature. It would greatly benefit the readers if the authors could provide more detailed explanations accompanying the figure to enhance understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Thank you for your sincere suggestion. We incorporating a comprehensive efficiency analysis including as following and above Table in the attached file. We will add them to further demonstrate the effectiveness and efficiency of the FouriDown operator in the revised version.\n2. Indeed, while the current work primarily focuses on replacing stride 2 down-sampling, the underlying principle of FouriDown has the potential to be extended for more general scenarios. For example, we have explored different stride lengths in the down-sampling. As shown in Supp. Figure 6, we aim to generalize the method to accommodate varying strides, making it more adaptable to diverse applications. Moreover, beyond convolutional layers, it would be intriguing to see more potential architectures like transformers are performed for the application of specific scenarios. By delving into these areas, we hope to make the FouriDown operator even more versatile and better suited to a wider range of tasks.\n3. Thanks for your suggestions, we will add more visualizations in the updated version.\n4. Thank you for your suggestions. Figure 5 depicts the spectrum changes of an image as it undergoes progressive downsampling, further aiding readers in understanding the frequency aliasing process. In Figure 5, three downsampling instances are displayed, each resulting in differently sized spectral representations. These are then superimposed onto the same frequency coordinate system. According to Theorem 1, high-frequency components will fold into the lower frequencies during downsampling, so the process is illustrated by the yellow arrows in the figure. In the revised version, we will expand the figure caption and add a dedicated subsection in the manuscript. \n"}]

Author rebuttal:
Thank all reviewers for insightful comment. We recognize the importance of providing comprehensive benchmarks for our proposed method. As shown in table below, we include results from traditional downsampling techniques like bicubic, bilinear, pixel-unshuffle, 2x2 learnable CNN (with stride=2), max-pooling, average-pooling, LPF, Gaussian and Ours. Noting that the “Original” downsampling of the method is pointed by asterisk (‘*’). This will allow a clearer contrast and showcase the advantages of our method not only against anti-aliasing approaches but also against these conventional downsampling methods. 
Note that although the shuffle process doesn't consume any parameters or FLOPs, implementing this step in software does take extra time. We will accelerate this step by CUDA and release the code in the future. We will further elucidate these computational trade-offs in the revised manuscript.

| Method | Config          |   | LOL    |        | FLOPs(G) | Time(s) | Para(M)  |
|--------|-----------------|---|--------|--------|----------|---------|----------|
|        |                 |   | PSNR   | SSIM   |          |         |          |
| SID    | Bicubic         |   | 21.35  | 0.8497 | 13.764   | 0.0131  | 7.84     |
|        | Bilinear        |   | 21.26  | 0.8464 | 13.764   | 0.0136  | 7.84     |
|        | Pixle-shuffle   |   | 21.41  | 0.8552 | 13.954   | 0.0138  | 8.11     |
|        | Stride Conv     |   | 21.36  | 0.8534 | 13.954   | 0.0144  | 8.11     |
|        | Max pooling *   |   | 21.46  | 0.8584 | 13.753   | 0.0134  | 7.84     |
|        | Average pooling |   | 21.34  | 0.8481 | 13.754   | 0.0128  | 7.84     |
|        | LPF             |   | 21.79  | 0.8612 | 16.102   | 0.0149  | 8.54     |
|        | Gaussian        |   | 20.74  | 0.8124 | 16.102   | 0.0137  | 8.54     |
|        | Ours            |   | 23.28  | 0.8708 | 13.827   | 0.0176  | 7.87     |

| Method | Config          |   | DVD     |        | FLOPs(T) | Time(s) | Para(M)  |
|--------|-----------------|---|---------|--------|----------|---------|----------|
|        |                 |   | PSNR    | SSIM   |          |         |          |
| MPRNET | Bicubic         |   | 29.8302 | 0.8815 | 1.398    | 0.5258  | 15.74    |
|        | Bilinear        |   | 29.8795 | 0.8822 | 1.398    | 0.5422  | 15.74    |
|        | Pixle-shuffle   |   | 29.8202 | 0.8816 | 1.399    | 0.575   | 15.93    |
|        | Stride Conv *   |   | 30.12   | 0.8958 | 1.399    | 0.5082  | 15.93    |
|        | Max pooling     |   | 29.8038 | 0.881  | 1.398    | 0.5402  | 15.74    |
|        | Average pooling |   | 29.8716 | 0.8825 | 1.398    | 0.5414  | 15.74    |
|        | LPF             |   | 30      | 0.8918 | 1.416    | 0.5004  | 16.26    |
|        | Gaussian        |   | 30.23   | 0.8922 | 1.416    | 0.5491  | 16.26    |
|        | Ours            |   | 30.31   | 0.8996 | 1.398    | 0.5974  | 15.93    |


Meta-review: This submission has received five reviews with four reviews that recommend acceptance and one reviewer with a strong rejection vote. 

All reviewers appreciate the problem, this is relevant as a basic building block for network architectures. The experiments are thorough and comprehensive, several image processing tasks with several baselines each, the results show an increased performance throughout. This includes image classification and image processing experiments, on the classification part only small models and CIFAR datasets. 


We checked carefully the concerns of reviewer e5DbChecking carefully the raised concerns of reviewer e5Db. However after the rebuttal and the discussion we find that the concerns are mainly answered, eg. the experimental evidence is found, the rebuttal states computation time and other baseline results were already in the initial draft. The related work that was raised in the discussion is not a critical miss for this paper. Therefore we regard the strong reject rating as too drastic given the arguments. 

There are certainly points that can and should be improved, the rebuttal and discussion surfaced some. Also many typos need correct, e.g.  (Guassian, Groud truth) also maybe move proof to the appendix/supplementary? However the results are convincing, the method is found novel from all reviewers, therefore the recommendation is to accept the work.
Decision: Accept (poster)