Title: FouriDown: Factoring Down-Sampling into Shuffling and Superposing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image restoration

Down-Sampling

Fourier transform
Abstract: Spatial down-sampling techniques, such as strided convolution, Gaussian, and Nearest down-sampling, are essential in deep neural networks. In this study, we revisit the working mechanism of the spatial down-sampling family and analyze the biased effects caused by the static weighting strategy employed in previous approaches. To overcome this limitation, we propose a novel  down-sampling paradigm in the Fourier domain, abbreviated as FouriDown, which unifies existing down-sampling techniques. Drawing inspiration from the signal sampling theorem, we parameterize the non-parameter static weighting down-sampling operator as a learnable and context-adaptive operator within a unified Fourier function. Specifically, we organize the corresponding frequency positions of the 2D plane in a physically-closed manner within a single channel dimension. We then perform point-wise channel shuffling based on an indicator that determines whether a channel's signal frequency bin is susceptible to aliasing, ensuring the consistency of the weighting parameter learning. FouriDown, as a generic operator, comprises four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform. These components can be easily integrated into existing image restoration networks. To demonstrate the efficacy of FouriDown, we conduct extensive experiments on image de-blurring and low-light image enhancement. The results consistently show that FouriDown can provide significant performance improvements. We will make the code publicly available to facilitate further exploration and application of FouriDown.

FouriDown: Factoring Down-Sampling into Shuffling and Superposing

Abstract

Spatial down-sampling techniques, such as strided convolution, Gaussian, and Nearest down-sampling, are essential in deep neural networks. In this paper, we revisit the working mechanism of the spatial down-sampling family and analyze the biased effects caused by the static weighting strategy employed in previous approaches. To overcome the bias limitation, we propose a novel down-sampling paradigm in the Fourier domain, abbreviated as FouriDown, which unifies existing down-sampling techniques. Drawing inspiration from the signal sampling theorem, we parameterize the non-parameter static weighting down-sampling operator as a learnable and context-adaptive operator within a unified Fourier function. Specifically, we organize the corresponding frequency positions of the 2D plane in a physically-closed manner within a single channel dimension. We then perform point-wise channel shuffling based on an indicator that determines whether a channel's signal frequency bin is susceptible to aliasing, ensuring the consistency of the weighting parameter learning. FouriDown, as a general operator, comprises four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform. These components can be easily integrated into existing image restoration networks. To demonstrate the efficacy of FouriDown, we conduct extensive experiments on image de-blurring and low-light image enhancement. The results consistently show that FouriDown can provide significant performance improvements. The code is publicly available to facilitate further exploration and application of FouriDown at https://github.com/zqcrafts/FouriDown .

1 Introduction

Down-sampling technique [1, 2, 3] plays a vital role in deep neural networks because of its benefits in enlarging the receptive field, extracting hierarchical features, improving computational efficiency, and handling scale and translation variations. However, based on the signal sampling theorem, existing down-sampling techniques such as strided convolution, Gaussian, and Nearest down-sampling [1, 4, 5, 6] unavoidably reduce the sampling frequency of discrete signals, leading to unexpected frequency aliasing where high frequencies are folded into low frequencies.

To address the aliasing problem, several strategies [7, 8, 9, 10, 11, 12, 13] have been developed. They pre-process the signals applying the low-pass filtering mechanism, which aims to filter out highfrequency information by employing different types of low-pass designs. There are two commonly

IDFT : Inverse Discrete Fourier Transform

SpatialDomain:

Downsampling

Previous Downsampling:

(strides=2)

H

W

CxHxW

C×-

×

2

2

(i) Strides=2 (e.g. Conv, Pooling)

Static weighting

W;=[0.25, 0.25, 0.25, 0.25]

FouriDown(s, wi)

(ii) Ideal low-pass -2x

FourierDomain:

W;=[0.25, 0.00, 0.00, 0.00]

Shuffling

Superposing

(Sec 3.3)

(Sec 3.2)

Wi

When strides=2,

W; E Guassian Distribution

X

X' = wjX

Ours (Sec. 3.3)

Adaptive

Wi

Conv

Relu

Conv

Softmax

DFT : Discrete Fourier Transform

IDFT : Inverse Discrete Fourier Transform

used types including the ideal low-pass filter that truncates high frequencies in the Fourier domain and the Gaussian low-pass filter that gradually attenuates frequency components near the boundary. However, the ideal low-pass filter may introduce ring artifacts due to spectrum leakage, while the Gaussian low-pass filter may result in a significant loss of edge information that is crucial for visual recognition tasks.

The prevailing approaches in the down-sampling family rely on a static weighting strategy, which may lead to unintended biases (See Section 4.4 for details.). As described in Figure 1, strided convolution and strided pooling variants rely on the static template w i = [0 . 25 , 0 . 25 , 0 . 25 , 0 . 25] over the corresponding cornered positions while the ideal low-pass one exploits the w i = [0 . 25 , 0 , 0 , 0] weighting template. (See Appendix B for proofs.) All of them are shared over all the coordinated positions and uninvolved with the feature context. It is widely acknowledged that the static sampling approach, which lacks contextual relevance, is sub-optimal for visual tasks. Therefore, both bridging different down-sampling approaches and achieving an optimal approach are desirable, as shown in Figure 1 where we focus on unifying the down-sampling modeling rules in a learnable and context-adaptive parameterized function in Fourier domain.

In this study, we delve into the working mechanism of the spatial down-sampling family and analyze the biased effects resulting from the static weighting strategy used in existing down-sampling approaches. To solve the bias problem, we propose a novel down-sampling paradigm called FouriDown, which operates in the Fourier domain and adapts the feature sampling based on the image context. Inspired by the signal sampling theorem, we parameterize the non-parameter static weighting down-sampling operator as a learnable and context-adaptive operator in a unified Fourier function. Furthermore, drawing from this insight, we organize the corresponding frequency positions of the 2D plane, ensuring that they are physically closed in a single channel dimension. We then perform pointwise channel shuffling based on an indicator that determines whether a channel's signal frequency bin is prone to aliasing, thereby maintaining the consistency of the weighting parameter learning. FouriDown, as a generic operator, comprises four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform. These components can be readily integrated into existing image restoration networks, allowing for a plug-and-play approach. To verify its efficacy, we conduct extensive experiments across multiple computer vision tasks, including image de-blurring and low-light image enhancement. The results demonstrate that FouriDown consistently outperforms the baselines, showcasing its capability of performance improvement.

In conclusion, this work propose a novel and unified framework for the research of down-sampling, which have the following contributions.

1) We provide the first exploration of the aliasing problem in deep neural networks, analyzing it from a spectrum perspective.

2) To achieve dynamic frequency aliasing, we introduce a unified approach to down-sampling strategies within the Fourier function. Additionally, we propose a learnable and context-adaptive down-sampling operator based on the Nyquist signal sampling theorem.

3) Our proposed down-sampling approach serves as a plug-and-play operator, consistently enhancing the performance of image restoration tasks, such as low-light enhancement and image deblurring.

2 Related Work

2.1 Traditional Down-Sampling

Downsampling is an important and common operator in computer vision [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25], which benefits from enlarging the receptive field and reducing computational costs. So many models incorporate downsampling to allow the primary reconstruction components conducting at a lower resolution. Moreover, with the emergence of increasingly compute-intensive large models, downsampling becomes especially crucial, particularly for high-resolution input images.

Previous downsampling methods often utilized local spatial neighborhood computations (e.g., Bilinear, Bicubic and MaxPooling), which show decent performances across various tasks. However, these computations are relatively fixed, making it challenging to maintain consistent performance across different tasks. To address this, some methods made specific designs to make the downsampling more efficient in specific tasks. For instance, some works [12, 11, 10, 7]introduce the Gaussian blur kernel before the downsampling convolution to combat aliasing for better shift-invariance in classification tasks. Grabinski et al. [26, 27] equip the ideal low-pass filter or the hamming filter into downsampling to enhance model robustness and avoid overfitting.

2.2 Dynamic Down-Sampling

Due to the development of data-driven deep learning, in addition to traditional down-sampling, some other works [28, 29, 30, 31, 32, 33] introduce dynamic downsampling to adaptively adjust for different tasks, thereby achieving better generalizability. For instance, Pixel-Shuffle [28] enables dynamic spatial neighborhood computation through the interaction between feature channels and spaces, restoring finer details more effectively. Kim et al. [29] proposes a task-aware image downsampling to support upsampling for more efficient restoration.

In addition to dynamic neighborhood computation, dynamic strides have also gained widespread attention in recent years. For instance, Riad et al. [30] posits that the commonly adopted integer stride of 2 for downsampling might not be optimal. Consequently, they introduce learnable strides to explore a better trade-off between computation costs and performances. However, the stride is still spatially uniformly distributed, which might not be the best fit for images with uneven texture density distributions. To address this issue, dynamic non-uniform sampling garners significant attention [31, 32, 33]. For example, Thavamani et al. [31] proposed a saliency-guided non-uniform sampling method aimed at reducing computation while retaining task-relevant image information.

In conclusion, most of recent researches focus on dynamic neighborhood computation or dynamic stride for down-sampling, where the paradigm can be represented as Down ( s ) , where s denotes the stride. However, in this work, we observe that the methods based on this downsampling paradigm employ static frequency aliasing, which may potentially hinder further development towards effective downsampling. However, learning dynamic frequency aliasing upon the existing paradigm poses challenges. To address this issue, we revisit downsampling from a spectral perspective and propose a novel paradigm for it, denoted as FouriDown ( s, w ) . This paradigm, while retaining the stride parameter, introduces a new parameter, w , which represents the weight of frequency aliasing during downsampling and is related to strides. Further, based on this framework, we present an elegant and effective approach to achieve downsampling with dynamic frequency aliasing, demonstrating notable performance improvements across multiple tasks and network architectures.

(a) 1-Dimension: F(u) and F'(u)

(b) 2-Dimension: F(u,v) and F'(u,v)

(Better zoom in)

u=

u=

2

x=n

Downsampling

u=

x

xZ=x=n

2

Before

u

=

=a

1

U

75

q1(u,v+

q3(u1+

2

v

;Po(uo);

/pi(uo

qo(u,v)

192（ui+

v

=

4

no

xojun

uo+

u

u

→

Downsampling

After

1

...

v

=

4

Q(ui.v1)

uoo

u

u

3 Method

Definitions. f ( x, y ) ∈ R H × W × C is the 2-D discrete spatial signal and the sampling rates in x and y axis are Ω x and Ω y , respectively. F ( u, v ) ∈ R H × W × C is the Fourier transform of f ( x, y ) , where the maximum frequencies in u and v axis are respectively denoted as u max and v max . Moreover, f ′ ( x, y ) ∈ R H 2 × W 2 × C is 2-strided down-sampled f(x,y) and its Fourier transform F ′ ( u, v ) .

Theorem-1. Shuffling and Superposing. The spatial down-sampling typically results in a shrinkage of the tolerance for the maximum frequency of the signal. Specifically, high frequencies will fold back into new low frequencies and superpose onto the original low frequencies. To illustrate with 1dimensional signal, the high and low frequency superposition in the down-sampling can be formulated as

where S is a superposing operator. Note that the high frequency is F ( u + Ω x 2 ) considering positive directions, while the low frequency is F ( u ) considering positive directions instead.

Theorem-2. Static Averaging Superposing. For an image, the spatial down-sampling operator with 2 strides can be equivalent to dividing the Fourier spectrum into 2 × 2 equal parts and superposing them averagely by 1 4 factor

where F ( i,j ) ( u, v ) is a sub-matrix of F ( u, v ) by equally dividing F ( u, v ) into 2 × 2 partitions and i, j ∈ { 0 , 1 } . Given that Down 2 is 2-strided down-sampling operator and IDFT is inverse discrete Fourier transform, we have

The proof of the above theorem can be found in the Appendix, and examples of 1-D and 2-D signals can also be referred in Figure 2(a) and (b).

Cluster superposing groups

BandAlignment

SpectralShuffle

(Theorem1 and Theorem 2)

Split into

16 patches

2

CxHxW

u

$x

2

$x

R(u,v)=

+n:n

2

,v:v+

2

4×C××

HW

Algorithm 1 Pseudo-code of FouriDown.

3.1 Architecture Design

In this work, we argue that the the static superposing strategy like the stride-based down-sampling in Theorem-2 might lead to biased effects. Motivated by adaptively learning ability of CNNs, we aim to parameterize the non-parameter static superposing step as a learnable and context-adaptive operator in the Fourier domain.

Definition-2 (Shuffle-Invariance) Given an operator z ( . ) that is shuffle-invariant and o 1 , o 2 , o 3 , o 4 as different components, the shuffle-invariant is defined as z ( o 1 , o 2 , o 3 , o 4 ) = z ( shuffle ( o 1 , o 2 , o 3 , o 4 )) , where shuffle ( . ) is shuffling the order of input components arbitrarily.

Note that the average operator in Theorem-2 is shuffle-invariant. For example, Aver ( a, b, c, d ) = Aver ( b, a, c, d )) . However, different from averaging, the convolution operator, which is sensitive to the input order, does not have this property.

To alleviate this problem, we design a spectral-shuffle strategy that first performs shuffling according to Theorem-1 and then aligns across different frequency bands, as shown in Figure 3. Specifically, we initially spilt the original spectrum F ( u, v ) into 16 patches equally. Then, according to Theorem-1, we classify these patches into 4 group, where each group is pixel-wise matched frequency bin for superposing. However, the energy distribution in each group is different. Considering the shufflevariance of convolution operators, we reorder the intra-group sequence for inter-group alignment. The alignment is motivated by wavelet theory, where intra-group frequencies are reordered according to low-frequency and high-frequency in horizontal direction, high-frequency in vertical direction, and high-frequency in diagonal direction. Then, the aligned groups are sorted orderly on channels for pixel-wise matching in the channel dimension. Finally, we perform adaptively weighted superposition on channels by learned weights for the down-sampling results. The main implementation is depicted in Algorithm 1. Code will be public.

4 Experiments and Discussion

To validate the efficacy of our proposed FouriDown, we execute comprehensive experiments across several computer vision tasks and conduct exhaustive ablation studies to evaluate its resilience.

4.1 Experimental Settings

Image enhancement. For image enhancement, we assess our FouriDown model using the LOL [34] and Huawei [35] benchmarks. The LOL dataset contains 500 image pairs (485 for training, 15 for testing), and the Huawei dataset contains 2480 paired images (2200 for training, 280 for testing). We compare our results with two established baselines, SID [36] and DRBN [37].

Image deblurring. For image deblurring, we utilize DeepDeblur [38] and MPRNet [39] on the DVD dataset [40], which includes 2103 training and 1111 test pairs. We further validate our model's generalizability using the HIDE dataset [41].

Image denoising. In the context of image de-noising, our training involves the SIDD dataset [42]. Subsequent performance assessments are carried out on the remaining validation samples from the SIDD dataset and on the DND benchmark dataset [43]. For comparative analysis, we choose baselines such as MIRNet [44], and Restormer [45].

Image dehazing. For image dehazing, we employ RESIDE dataset for evaluations. We also use two different network designs MSBDN [46] and GridNet [47] with our proposed operator for validation.

4.2 Implementation Details

Regarding the above competitive baselines, we perform the comparison over the following configurations by replacing the down-sampling operator, such as strided convolution and strided pooling), with the proposed FouriDown operator. Additionally, we also perform comparisons with previous anti-aliasing down-sampling methods, including Gaussian filter [7] and 'ideal" Low-Pass Filter (LPF) [13], which conduct the static modulation on the spectrum.

Original : The original down-sampling in the baseline;

Gaussian : Following [7], equipping the Gaussian filter in all channels before the original down-sampling for anti-aliasing;

LPF : Following [13], equipping the "ideal" Low-Pass Filter in all channels before the original down-sampling operator for anti-aliasing;

Ours : Replacing our proposed FouriDown with the original down-sampling operator;

(a)Low-light image

(b) SID

(c) SID with FouriDown

(d) )Groud truth

(a)Low-light image

(b) DRBN

(c) DRBN with FouriDown

(d) Groud truth

4.3 Comparison and Analysis

Quantitative Comparison. To demonstrate the effectiveness of our proposed FouriDown, we conduct extensive experiments as shown in Tables 1-4. The best results are in bold. Above and below the baseline are highlighted in red and blue, respectively. From these tables, although previous anti-aliasing methods may be useful for some image restoration tasks, their static weights limit their universality in other tasks. For instance, while the LPF approach performs well in dehazing, it fails to deliver effective in deblurring and low-light enhancement. In contrast, our method is proved to be effective across the majority of image restoration tasks. Specifically, we achieved an improvement of 1.82dB in low-light enhancement and 0.42dB in dehazing on LOL and Reside dataset respectively.

Further, we compare the computing costs with other methods shown in Table 5. We include results from traditional down-sampling techniques like bicubic, bilinear, pixel-unshuffle, 2x2 learnable CNN (with stride=2), max-pooling, average-pooling, LPF, Gaussian and Ours. Noting that the 'Original' down-sampling of the method is pointed by asterisk ('*'). This will allow a clearer contrast and showcase the advantages of our method not only against anti-aliasing approaches but also against these conventional down-sampling methods.

Qualitative Comparison. Due to space constraints, we only present a qualitative comparison on the low-light enhancement task. As illustrated in Figure 4 and Figure 5, our FouriDown reduces original artifacts presented in the SID due to the more flexible frequency interactions. Then, we compare the visualizations of the feature maps and their corresponding spectra between FouriDown and other down-sampling methods (see Figure 6 and Figure 7). It can be observed that the model equipped with FouriDown generates much stronger responses to degradation-aware regions, i.e. global lowillumination in the low-light enhancement task. In contrast, the model with other down-sampling method responds weakly to these regions. The results demonstrates the effectiveness of FouriDown in capturing degradation-aware information by adaptive frequency superposition in down-sampling. For the Gaussian method, its response to degradation is relatively large (second only to FouriDown), thus achieving performance that is also second only to FouriDown. Similarly, as the LFP method has the poorest performance, its feature response of the low-light areas is also the lowest. The performance of other methods is roughly similar, so their feature responses are also quite similar, indicating a similar capability to capture image degradation areas. Additionally, from the spectral comparison in Figure 7, it can be observed that the Gaussian method loses a lot of high-frequency information compared to FouriDown. This leads to challenges in recovering textures in dark areas. Hence, although the Gaussian method exhibits good responses, FouriDown achieves better performances compared to it. More qualitative comparisons can be found in the following supplementary material.

10

10

口

(a) Input

(b)AverPooling

(c)Bicubic

(d)Bilinear

(e)Gaussian

10

10

0

(f)LPF

(g) MaxPooling

(h)PixelShuffle

(i) StrideConv

G) Ours

(a) Input

(b)AverPooling

(c)Bicubic

(d) Bilinear

0

(f)LPF

(g)MaxPooling

(h)PixelShuffle

(i) StrideConv

(G) Ours

4.4 Discussions

Bias Effects by Static Superposing. As shown in Figure 8, we compare different down-sampling methods with static superposing manner, and find they have various bias effects.

Frequency Aliasing Visualization. To delve deeper into the high-low frequency interactions in down-sampling, we examine the spectrum of images down-sampled by factors of 4x, 2x, and 1x. Following Theorem 1, some regions of spectrums are overlaid on the same frequency band, with smaller scales overlaying larger ones, as shown in Figure 9. This alignment of same bandwidth reveals a rectangular contour at the intersections, where high-frequencies not obeying the Nyquist theory fold into low frequencies during down-sampling, as pointed by the yellow arrow. This suggests that it is significant for down-sampling to modulate frequencies, otherwise it might degrade the original signal undesirably.

Other Discussions. Because of space constraints, for more discussions, including extensions to Theorem-2 and revisiting of previous anti-aliasing methods in the proposed FouriDown framework, could be referred to the supplementary material.

5 Limitations

In this work, we explore spatial down-sampling from a frequency-domain perspective and optimize the static weighting of previous down-sampling with a stride of 2 in the frequency domain. Our modeling of down-sampling is based on using uniformly distributed impulse sequences as the sampling function, hence exploring the characteristics of the sampling function in the frequency domain. However, for non-uniform down-sampling, where the sampling rate varies according to the content, our method might become limited. We hope to overcome this limitation in the future work by exploring the frequency domain response of non-uniform sampling functions.

"Natural"

“Aliasingeffect"

“Ringing effect"

"High-frequency loss"

HARDWARE

HARDWARE

HARD

VARE

(a) Input

(b)Nearest-2x

(c)Ideal low-pass -2x

(d)Guassian-2x

6 Conclusion

In our study, we revisit the spatial down-sampling techniques and anti-aliasing strategies from a Fourier domain perspective, recognizing their reliance on static high and low frequency superposing. As a result, we propose a novel approach (FouriDown) to learn a learnable frequency-context interplay among high and low frequencies during down-sampling. Moreover, this work is the first exploration of dynamic frequency interaction in down-sampling. The FouriDown is designed based on the signal sampling theory, so it is convenient to replace most of current down-sampling and anti-aliasing techniques. Extensive experiments demonstrate the performance improvements across a variety of vision tasks.

Ultimately, we believe that down-sampling is a crucial research direction in the future. It allows for network design at a lower resolution, significantly reducing the computational overhead and enabling light-weighting models.

Broader Impact

This work further exploits the potential of down-sampling in the Fourier domain and offers a new perspective that frequency band shuffling and superposing for future research of down-sampling. Down-sampling techniques can potentially make the future model development more efficient and effective, beneficial for various machine learning and AI applications. Nonetheless, the efficacy of our method could be a source of concern if not properly utilized, especially in terms of the safety of real-world applications. To alleviate such concerns, we plan to rigorously investigate the robustness and effectiveness of our approach across a more diverse spectrum of real-world scenarios.

Acknowledgments

This work was supported by the JKW Research Funds under Grant 20-163-14-LZ-001-004-01, and the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.

S=1

S=2

S=4.

Better zoom in.

References

Yongbing Zhang, Debin Zhao, Jian Zhang, Ruiqin Xiong, and Wen Gao. Interpolation-dependent image downsampling. IEEE Transactions on Image Processing , 20(11):3291-3296, 2011.

Ding-Xuan Zhou. Theory of deep convolutional neural networks: Downsampling. Neural Networks , 124:319-327, 2020.

Christian Bailer, Tewodros Habtegebrial, Didier Stricker, et al. Fast feature extraction with cnns with pooling layers. arXiv preprint arXiv:1805.03096 , 2018.

Leif E Peterson. K-nearest neighbor. Scholarpedia , 4(2):1883, 2009.

Michael Lindenbaum, Shaul Markovitch, and Dmitry Rusakov. Selective sampling for nearest neighbor classifiers. Machine learning , 54:125-152, 2004.

Edward A Patrick and Frederic P Fischer III. A generalized k-nearest neighbor rule. Information and Control , 16(2):128-152, 1970.

Richard Zhang. Making convolutional networks shift-invariant again. ArXiv , abs/1904.11486, 2019.

Antônio H. Ribeiro and Thomas Schon. How convolutional neural networks deal with aliasing. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing , pages 2755-2759, 2021.

Cristina Nader Vasconcelos, H. Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin. An effective anti-aliasing approach for residual networks. ArXiv , abs/2011.10675, 2020.

Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. International Journal of Computer Vision , 131:67-81, 2020.

Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11400-11410, 2022.

Shengju Qian, Hao-Chiang Shao, Yi Zhu, Mu Li, and Jiaya Jia. Blending anti-aliasing into vision transformer. In Neural Information Processing Systems , volume 34, pages 5416-5429, 2021.

Hagay Michaeli, Tomer Michaeli, and Daniel Soudry. Alias-free convnets: Fractional shift invariance via polynomial activations. ArXiv , abs/2303.08085, 2023.

Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. n Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 936-944, 2016.

Naishan Zheng, Man Zhou, Yanmeng Dong, Xiangyu Rui, Jie Huang, Chongyi Li, and Feng Zhao. Empowering low-light image enhancer through customized learnable priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1255912569, October 2023.

Xiang Chen, Jinshan Pan, Jiyang Lu, Zhentao Fan, and Hao Li. Hybrid cnn-transformer feature fusion for single image deraining. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 378-386, 2023.

Qi Zhu, Naishan Zheng, Jie Huang, Man Zhou, Jinghao Zhang, and Feng Zhao. Learning spatio-temporal sharpness map for video deblurring. IEEE Transactions on Circuits and Systems for Video Technology , 2023.

Qi Zhu, Man Zhou, Naishan Zheng, Chongyi Li, Jie Huang, and Feng Zhao. Exploring temporal frequency spectrum in deep video deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12428-12437, 2023.

Qi Zhu, Zeyu Xiao, Jie Huang, and Fengmei Zhao. Dast-net: Depth-aware spatio-temporal network for video deblurring. Proceedings of the IEEE/CVF International Conference on Multimedia and Expo , pages 1-6, 2022.

Naishan Zheng, Jie Huang, Qi Zhu, Man Zhou, Feng Zhao, and Zheng-Jun Zha. Enhancement by your aesthetic: An intelligible unsupervised personalized enhancer for low-light images. In Proceedings of the 30th ACM International Conference on Multimedia , pages 6521-6529, 2022.

Naishan Zheng, Jie Huang, Man Zhou, Zizheng Yang, Qi Zhu, and Feng Zhao. Learning semantic degradation-aware guidance for recognition-driven unsupervised low-light image enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 3678-3686, 2023.

Jie Huang, Feng Zhao, Man Zhou, Jie Xiao, Naishan Zheng, Kaiwen Zheng, and Zhiwei Xiong. Learning sample relationship for exposure correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9904-9913, June 2023.

Naishan Zheng, Jie Huang, Feng Zhao, Xueyang Fu, and Feng Wu. Unsupervised underexposed image enhancement via self-illuminated and perceptual guidance. IEEE Transactions on Multimedia , 2022.

Huikang Yu, Jie Huang, Yajing Liu, Qibang Zhu, Man Zhou, and Fengmei Zhao. Source-free domain adaptation for real-world image dehazing. Proceedings of the 30th ACM International Conference on Multimedia , 2022.

Wei Yu, Qi Zhu, Naishan Zheng, Jie Huang, Man Zhou, and Fengmei Zhao. Learning nonuniform-sampling for ultra-high-definition image enhancement. Proceedings of the 31st ACM International Conference on Multimedia , 2023.

Julia Grabinski, Steffen Jung, Janis Keuper, and Margret Keuper. Frequencylowcut pooling plug & play against catastrophic overfitting. ArXiv , abs/2204.00491, 2022.

Julia Grabinski, Janis Keuper, and Margret Keuper. Fix your downsampling asap! be natively more robust via aliasing and spectral artifact free pooling. ArXiv , abs/2307.09804, 2023.

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1874-1883, 2016.

Heewon Kim, Myungsub Choi, Bee Lim, and Kyoung Mu Lee. Task-aware image downscaling. In European Conference on Computer Vision , 2018.

Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour. Learning strides in convolutional neural networks. ArXiv , abs/2202.01653, 2022.

Chittesh Thavamani, Mengtian Li, Francesco Ferroni, and Deva Ramanan. Learning to zoom and unzoom. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5086-5095, 2023.

Dmitrii Marin, Zijian He, Péter Vajda, Priyam Chatterjee, Sam S. Tsai, Fei Yang, and Yuri Boykov. Efficient segmentation: Learning downsampling near semantic boundaries. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2131-2141, 2019.

Babak Ehteshami Bejnordi, Amirhossein Habibian, Fatih Murat Porikli, and Amir Ghodrati. Salisa: Saliency-based input sampling for efficient video object detection. ArXiv , abs/2204.02397, 2022.

Wenhan Yang Jiaying Liu Chen Wei, Wenjing Wang. Deep retinex decomposition for low-light enhancement. In Proceedings of the British Machine Vision Conference , 2018.

Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu Zou, Fang Lin, and Songchen Han. R2rnet: Low-light image enhancement via real-low to real-normal network. arXiv preprint arXiv:2106.14501 , 2021.

Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3291-3300, 2018.

Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3063-3072, 2020.

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3883-3891, 2017.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, MingHsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14821-14831, 2021.

Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. Deep video deblurring for hand-held cameras. Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition , pages 237-246, 2017.

Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In IEEE International Conference on Computer Vision , 2019.

Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1692-1700, 2018.

Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1586-1595, 2017.

Syed Waqas Zamir, Aditya Arora, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for fast image restoration and enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45:1934-1948, 2020.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5728-5739, 2022.

Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2157-2167, 2020.

Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-based multiscale network for image dehazing. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7314-7323, 2019.

Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6849-6857, 2019.

Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics , 36:1 - 12, 2017.

Appendix A: Proofs of Theorem-1 and Theorem-2

Proof of Theorem-1: Shuffling and Superposing

To model the relationship between f ′ ( x ) and f ( x ) , we stand on their derived continuous signal g ( x ) by a specific sampling function. Note that the sampling functions s ∆ T ( x ) is defined as the sum of infinitely impulses units separated by ∆ T intervals:

Based on the Eq. (4), f ( x ) and its down-sampling f ′ ( x ) can be represented as

According to the Fourier transform and convolution theorem, the aforementioned spatial sampling can be expressed in the Fourier domain as:

where G ( u ) and S ( u, ∆ T ) are the Fourier transform of g ( x ) and s ( x, ∆ T ) . From Eq. (6), it can be observed that the spatial sampling introduces the periodicity to the spectrum and the period is 1 ∆ T .

Note that the sampling rates of f ( x ) and f ′ ( x ) are Ω x and Ω ′ x , the relationship between them can be written as

Before down-sampling, to focus on the following down-sampling operation, we assume that f ( x ) adheres to the Nyquist sampling theorem, which implies that u max > 1 Ω x .

After down-sampling, according to Nyquist sampling theorem, a whole sub-frequency band is limited in (0 , Ω x 2 ) . Moreover, the resulted band is the superposition of two original bands, which denoted as

where ˆ u , u are the high frequency above the sampling rate and the low frequency below the sampling rate, respectively, and S is superposition operator.

(1) In the positive sub-band, where u ∈ (0 , Ω x 4 ) , ˆ u and ˜ u should satisfy

According to the aliasing theorem, the high frequency ˆ u is folded back to the low frequency:

where k = 1 , 3 , 5 . . . and ˜ u is folded results by ˆ u .

According to Eq. (9) and Eq. (10), we have

Then, according to Eq. (8) and Eq. (11), we attain

According to Eq. (6), F ( u ) is symmetric with respect to u = Ω x 2 ,

Further, F ( u + Ω x 2 ) = 0 when u ∈ (0 , Ω x 2 -u max ) . Upon Eq. (13), we can uniform Eq. (12) as

The visualization of the aforementioned proof process is depicted in Figure 2(a).

(2) In the negative sub-band, where u ∈ ( Ω x 4 , Ω x 2 ) , different from (1), ˆ u and ˜ u should satisfy

Similarly, we can proof F ( u ) in the negative sub-band as well.

Combined with Eq. (14) and Eq. (16), we obtain

Proof of Theorem-2: Static Superposing.

According to Eq. (7) and Eq. (6), we can deduce that the amplitude of F ′ is half that of F .

Therefore, we can write F ′ ( u ) in the x axis as,

Upon the dual principle, we can prove F ′ ( v ) in the whole sub-band

where u ∈ (0 , Ω x 2 ) , v ∈ (0 , Ω y 2 ) .

Appendix B: Revisiting previous down-sampling manners in the FouriDown

In the main body, we introduce the FouriDown framework which can simultaneously adjust the stride of down-sampling and the characteristics of frequency interaction. Moreover, in the manuscript, we prove that the operator with a stride of 2 can be simplified to a fixed frequency weighting, i.e. averaging. However, this kind of weighting brings aliasing, hence the need for anti-aliasing methods such as ideal low-pass filters and Gaussian filters, which put focused weighting on frequencies. In this section, we revisit these anti-aliasing methods within the FouriDown framework and discover that their weighting methods remain fixed. In other words, within our framework, it's possible to realize the aforementioned anti-aliasing methods by simply changing specific parameters as shown in Figure 10 and 11.

Appendix C: Stride Extension of Theorem-1 and Theorem-2

In the manuscript, we propose FouriDown which primarily focuses on analyzing frequency interactions in the case of stride=2. In this section, we extend the Theorem-2 to arbitrary integer strides s , resulting in the following theorem. The proof for these theorems follow similar logic to that detailed in the main body, and is therefore omitted for brevity.

Shuffling

Superposing

Q

Split into

16 patches

Wo,W1,

W2,W3

"Aliasing effect"

0.25,0.25,0.25,0.25

(a) Nearest-2x

DFT

shuffle

Wo,W1,

W2,W3

0.25 0.00, 0.00, 0.00

"Ringing effect"

(b) Ideal low-pass -2x

Wo.W1,

S

W2,W3

SP

Weighted superposing

DFT

Discrete Fourier Transform (DFT)

"High-frequency loss"

IDFT

Inverse Discrete Fourier Transform (IDFT)

(c) Guassian-2x

3

4

n=0

(1) Nearest downsampling (no interpolation)

"Ringing”

Process

as (1)

(2) Ideal low-pass filter

0

"High-frequencyLoss"

Process

Discrete Fourier

as (1)

Transform (DFT）

InverseDiscreteFourier

Transform (IDFT)

(3)Guassianfilter

Stride Extension of Theorem-1. In the condition of s strides, to illustrate with 1-dimensional signal, the high and low frequency superposition in the down-sampling can be formulated as

where S is a superposing operator and k i = 1 , ..., s -1 .

Stride Extension of Theorem-2. For an image, the spatial down-sampling operator with s strides can be equivalent to dividing the Fourier spectrum into s × s equal parts and superposing them averagely by 1 s 2 factor

Given that Down s is s-strided down-sampling operator and IDFT is inverse discrete Fourier transform, we have

Downsampling （Strides = 2）

Downsampling ( Strides = s )

#x

kSx

u

=

2

x=n

u=

S

u=

S

x=n

75

q（u,v+

1q3（u1+

$x

q（u,0+

Ki+1y

q（u+

k,lx

+

ki+isly

(l)

二

2

q（u.1+

q（u1+

kiny

+In）b

k,hly

Qy

S

2x

qo(un, v1)

q2（ui+

2

V1)

qo（u.D1）1

Ig（u+

P)

V1)

S

u

u

ki = 1,..,S - 1

Background

easytowatch.

V

y

2

S

4

Q(uLD)

u

u

x

u

4

n

2

u=

S

Based on above extension, the down-sampling of 3 strides could be presented as in Figure 12.

Appendix D: More Qualitative comparison.

Due to the limited space, we only report the visual results of the low-light enhancement task in main manuscript. We report more visual results in the supplementary materials. As shown, integrating the FouriDown with the original baseline [36, 37, 48, 49, 39, 45, 44, 46, 47] achieves more visually pleasing results as shown in Figure 13 14 15 16 17.

(a) Low-light image

(b) SID

(c) SID with FouriDown

(d) Groud truth

(a) Hazy image

(b) GridNet

(a) Input image

(b) HDRNet

(c)HDRNet with FouriDown

(a) Input image

(b) UPE

(c) UPE with FouriDown

(d) Groud truth

(a) Noisy image

(b) Restormer

(d) Groud truth

(a) Noisy image

(b) MIRNet

(c) MIRNet with FouriDown

(d) Groud truth

(b)MPRNet

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel down-sampling operator in the Fourier domain, specifically tailored to mitigate the common problem of frequency aliasing encountered during down-sampling. The operator's efficacy has been assessed across diverse computer vision problems, consistently yielding improved performance in these tasks.\n\nSTRENGTHS:\n1. The authors address the long-standing issue of frequency aliasing in signal processing, with a specific focus on image down-sampling. Their meticulously crafted design offers a compelling and well-founded solution to mitigate frequency aliasing.\n\n2. The authors substantiate their claims in the paper with comprehensive and rigorous theoretical proofs, providing a strong basis for their proposed approach.\n\n3. The proposed FouriDown operator serves as a plug-and-play solution, seamlessly integrating into existing networks. Furthermore, it consistently enhances the performance of the underlying models across various tasks.\n\n4. The writing in the paper is clear and the explanations are thorough, ensuring a comprehensive understanding of the presented concepts.\n\nWEAKNESSES:\n1. While the quantitative results are satisfactory, it would be beneficial to include additional efficiency analysis, such as providing the parameters of the FouriDown operator, in the form of a table showcasing relevant metrics. \n\n2. Although the FouriDown operator is currently limited to replacing the stride 2 down-sampling operator, it would be valuable to explore whether the authors have developed other variants capable of handling more general scenarios. \n\n3. It would be advantageous to present more visual results while condensing reducing the length of the theorem proofs. \n\n4. Figure 5 captures attention due to its intriguing nature. It would greatly benefit the readers if the authors could provide more detailed explanations accompanying the figure to enhance understanding.\n\nQUESTIONS:\nPlease see the weaknesses\n\nLIMITATIONS:\nThe authors mentioned the limitations in section 5 and addressed the potential broader impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this research, the authors re-examine the spatial down-sampling mechanism and analyze the biased impacts that arise from the static weighting strategy used in previous methods. To overcome this limitation, this work proposes a new down-sampling paradigm called FouriDown, which incorporates frequency-context adaptive superposing. This component can be seamlessly integrated into existing image restoration networks and significantly improve performance.\n\nSTRENGTHS:\nThis work provides an in-depth exploration of the down-sampling operator, offering a novel perspective that has the potential to significantly advance future research related to operators. Furthermore, the motivation behind this work is clear and well-founded, as evidenced by the compelling depiction in Figure 2 along with the accompanying theoretical proof. Importantly, this research showcases considerable potential across diverse visual tasks.\n\nWEAKNESSES:\n1. To better emphasize the main content of the paper, it is suggested to move the proof of formulas to the supplementary material. Furthermore, the experimental section of the main text could benefit from incorporating the information from Section 1 in the supplementary material, which illustrates the superior extensibility of this framework over other down-sampling methods.\n2. Although the supplementary material includes numerous excellent visual results to showcase the superiority of the proposed method, it is recommended to supplement the updated version with visualizations of the down-sampled feature maps.\n\nQUESTIONS:\n1. The authors present a novel down-sampling method based on Shannon's sampling theorem. As feature maps commonly comprise multiple channels, it would be interesting to investigate the interaction patterns that emerge among these channels.\n2. Based on the diagram provided, the FouriDown method seems to involve information transfer onto the channel followed by channel contraction, similar to the approach used in pixel-shuffle down-sampling. It is crucial to conduct an analysis that specifically explores and demonstrates the advantages of the FouriDown method over pixel-shuffle down-sampling. This comparative analysis will provide valuable insights into the superiority of the proposed method.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a downsampling method in Fourier domain (FouriDown) for several low-level vision tasks and an image classification task. FouriDown is supported by theoretical background proven by the authors. While this work improves multiple vision tasks when compared to previous anti-aliasing downsampling methods, it lacks to truly demonstrate its superiority over existing other downsampling approaches due to not well-organized experiments. Moreover, exploration of other architecture is insufficient. So, despite the robust and well-presented theorems the contribution of this paper is limited.\n\nSTRENGTHS:\nThe explanation and background of this paper are theoretically robust. Why the shuffling and superposition operations of FouriDown are constructed in that manner is clear. The learnable downsampling weight, which is unlike previous anti-aliasing methods, is technically sound.\n\nMany low-level vision tasks have been improved by FouriDown. The tables showing experimental results are easy to understand (however, some issues of experimental composition exist as mentioned in weakness). In addition to low-level tasks, the results that an image classification task is also improved makes FouriDown more robust.\n\nWEAKNESSES:\n1. Most importantly, this work focusing only on downsampling method has limited contribution for being published at this conference. If this paper has studied other architectures (e.g., CNN’s kernel or Transformer’s self-attention) and secondarily introduced the effectiveness of FouriDown, the novel downsampling method could have delivered more importance for deep learning field. This does not mean that FouriDown is not effective. Instead, more urgent architecture designs could be considered and provided in this paper. Moreover, a study regarding how to effectively and efficiently address the hierarchical downsampled multi-scale feature maps, where downsampling itself loses informative details, is suggested to be conducted.\n2. While replacement of static superposition to learnable one is interesting, the composition of this learnable weight is not sophisticated enough to be considered as a novel idea. It just follows standard 1x1 convolution, activation function, and softmax. It should be carefully considered.\n3. Tabs.1,2,3,4,5,6 should include some factors with respect to efficiency: inference time, the number of parameters, etc.\n4. The “Original” downsampling methods in Tabs.1,2,3,4,5,6 are not presented. The authors, therefore, should present other downsampling methods that are not anti-aliasing approaches. For example, pixel-unshuffle, 2x2 learnable CNN (with stride=2), max-pooling, or other interpolation can be considered.\n5. The summation operation along subsample dimension (indicated as “4”) in the Algorithm 1 can be better demonstrated by another ablation study. As alternatives to summation, there exists average operation or weighted summation for the “4”.\n\nQUESTIONS:\nHow about ImageNet 1K classification or object detection? Did you conduct other experiments about high-level tasks? Because the performance gains are more notable in the case of CIFAR classification than low-level tasks, focusing mainly on high-level vision recognition tasks seems better.\n\nLIMITATIONS:\nSee weakness.\n\n\nI think the additional tables, figures, and description made this paper more comprehensive to be published at NeurIPS 2023. Therefore, I would like to recommend this paper to be accepted, and will change my rating from borderline reject to weak accept.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
- This paper explores a new method for downsampling, by factoring it into shuffling and superposing. Firstly, the authors provide an understanding of the aliasing in deep neural networks from a spectrum view. 

 - FouriDown, a unified downsampling approach with the learnable and context-adaptive down-sampling operator, is proposed. This method is made of four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform.

 - Extensive experiments on image de-blurring and low-light image enhancement are conducted, which consistently show that FouriDown can provide significant performance improvements.

STRENGTHS:
- Extensive qualitative and quantitative evaluation results on scale-sensitive tasks are conducted, which show the advantage of the proposed downsampling method over previous ones.

 - This work first provides an exploration of the aliasing issue in neural networks from a spectrum perspective, which will facilitate a deeper understanding of the aliasing issue.

WEAKNESSES:
- Additional details, such as the computation costs and the respective results on images and features, should be provided. (See Question)

QUESTIONS:
- What are the additional computation costs of FouriDown, compared to other downsampling methods?

 - Is there any difference between applying FouriDown to the image and feature level? It's interesting to see the comparisons with other downsampling methods in-between, as images and features in DNN have differences in frequency distribution.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a downsampling method for neural networks in the frequency domain.\nThe proposed downsampling consists of discrete Fourier transform, shuffling, superposing, and inverse discrete Fourier transform.\nThe shuffling shuffles the patches in Fourier domain and the superposing adds the patches with learned and adaptive weights.\nExperimental results present that the proposed method outperforms Gaussian filtering and the ideal low-pass filtering in frequency domain for neural networks on various computer vision tasks.\n\nSTRENGTHS:\nDownsampling in frequency domain is an interesting topic. \nThe proposed method and the experimental results are reasonable.\nPerformance improvements on enhancement tasks are impressive.\n\nWEAKNESSES:\nExperiments lack comparisons and analyzes to convince the effectiveness of the proposed method.\n\nThe proposed method uses additional convolution layers than compared methods, so the computational overheads (FLOPs and latency) should be compared.\nIt is worth noting that DFT and IDFT also require additional overheads compared to the original networks.\n\nComparisons to numerous image downscaling methods (bilinear, bicubic, etc) are missing.\nIn particular, pixel-downshuffling [A] with convolution performs a similar operation to the proposed method without Fourier transform.\n\nFeature map visualization in frequency domain is missing.\nFeature maps of convolution layers usually contain high frequency edges, thus it is curious how the proposed method works in the feature space.\n\nAblation study of additional convolution layers is missing.\nSimple operations instead of the convolution layers such as max or mean will be an interesting ablation study.\n\n[A] Channel Attention Is All You Need for Video Frame Interpolation, AAAI 2020\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nPlease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}