Deep Fractional Fourier Transform
Authors: Hu Yu
Jie Huang
Lingzhi Li
Man Zhou
Feng Zhao
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Fractional Fourier Transform
image restoration

Abstract: Existing deep learning-based computer vision methods usually operate in the spatial and frequency domains, which are two orthogonal \textbf{individual} perspectives for image processing.
In this paper, we introduce a new spatial-frequency analysis tool, Fractional Fourier Transform (FRFT), to provide comprehensive \textbf{unified} spatial-frequency perspectives.
The FRFT is a unified continuous spatial-frequency transform that simultaneously reflects an image's spatial and frequency representations, making it optimal for processing non-stationary image signals.
We explore the properties of the FRFT for image processing and present a fast implementation of the 2D FRFT, which facilitates its widespread use.
Based on these explorations, we introduce a simple yet effective operator, Multi-order FRactional Fourier Convolution (MFRFC), which exhibits the remarkable merits of processing images from more perspectives in the spatial-frequency plane. Our proposed MFRFC is a general and basic operator that can be easily integrated into various tasks for performance improvement.
We experimentally evaluate the MFRFC on various computer vision tasks, including object detection, image classification, guided super-resolution, denoising, dehazing, deraining, and low-light enhancement. Our proposed MFRFC consistently outperforms baseline methods by significant margins across all tasks.

Deep Fractional Fourier Transform
Hu Yu Jie Huang" Lingzhi Li Man Zhou'
Feng Zhao 3,1,
'University of Science and Technology China Allibaaa Group
(yuhu520, hj0117, (anman)@mall usw.tstccss. edu, en fzhao9560ustc, edu,
Abstract
Existing deep learning-based computer vision methods usually operate in the spa- tial and frequency domains, which are two orthogonal individual perspectives for image processing. In this paper, we introduce new ceatial-frequenee analysis tool, Fractional Fourier Transform (FRFT), to provide comprehensive unified spatial-frequency perspectives. The FRFT is unified continuous spatial-frequene transform that simultaneously reflects an image's 's spatial and frequency representa- tions, making optimal for processing non-stationary image signals. We explore the properties of the FRFT for image processing and present a fast implementation the 2D FRFT, which facilitates its widespread use. Based on these explorations, we introduce simple yet effective operator, Multi-order FRactiooal Fourier Convo- lution (MFRFC), which exhibits the remarkable merits of processing images from more perspectives in the spatial-frequency plane. Our proposed MFRFC is general and basic operator that can be easily integrated into various tasks for performance improvement. We experimentally evaluate the MFRFC on various computer vision tasks, including object detection, image classification, guided super-resolution, denoising, dehazing, deraining, and low-light enhancement. Our proposed MFRFC consistently outperforms baseline methods by significant margins across all tasks. Our code is released publicly at httpts:: :/tp::///tt thub. com/yuhuUSTC/FRF
Introduction
Vanilla convolution in the spatial domain has been the dominant approach in Convolutional Neural Networks (CNNs) due to its advantages of fewer parameters with parameter sharing and translation invariance. However, CNNs suffer from limited receptive field, especially in early layers. To address this issue, several methods have attempted to convert signals from the spatial domain into the frequency domain using the Fourier Transform (FT), which offers global receptive field according the spectral convolution theorem [1] and is physically meaningful. However, FT has limitations due to its reliance on stationary signals and lack of frrquenlf properties. As a global transformation. it captures the overall spectrum of an image but cannot express the local spatial characteristics of the image. This makes FT optimal for stationary signals but not for ota-stationary ones, such as images. Additionally, the spatial and frequency domains are only two special and orthogonal perspectives of signals. Then, we wonder if there is an optimal tool for processing non-stationary image signals and providing an intermediate and unified perspective of an image.
To address the aforementioned question, we introduce a unified pattialfrrequnce Jnallsiis tool called the Fractional Fourier Transform (FRFT), which offers several advantages. First, the basis function of the FRFT is the linear frequency modulation function, making well-suited for processing non-stationary image signals. As shown in Figure a),, non-stationary image signals are not separable
'Equal Contribution. |Corresponding Author.
Work partially performed during internship at Alibaba DAMO Academy.
Also, a time-frequee frequency analysis tool for time-varying signals such as video, speech, and radar signals.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

f (p=1)
wndesired part
(p=1)
destred part
Up
Up
Frequency axis
Fractional order axis
= ppt/2
s (p=0)
(p=0)
Spatial axis
(a) FRFT is inherently optimal for non- non-stationary im- age signals and is capable of separating them effec- tively along the fractional order axis..
(b) FRFT is generalized version of Fourier transform, providing unified and intermediate perspective for image processing.
Figure I:
Merits of FRFT for image processing. The relationship between FRFT and spatial-frequeng plane rotation is detailed in subsection
when projected onto the spatial frequency axis, but they are separable on the fractional order axis, enabling the FRFT to extract the desired features from an image. Secondly, as illustrated in Figure [b), the FRFT generalizes the traditional Fourier Transform (FT) to the entire spatial-frequency domain with an additional parameter known as the order which is related to signal rotation angle in the frriiullfreuueccy plane. In ppatial-frequencyy analysis, signal is represented in plane where the spatial and frequency domains are orthogonal axes, as depicted in Figure [ For a signal the frequency axis, meaning 90p rotation in the ppeiial-rreuuncc plane. The FRFT, on the other hand, can be interpreted as the pth-order counterclockwise rotation (rotation angle
fz)) along the spatial axis, its traditional FT F(w) corresponds
pz pr/2) in the pratial-frrquencyy plane (2], where the spatial domain denotes order 0 and the frequency domain corresponds to order 1. The axis ", in Figure 1 represents rotated axis, also known the fractional order axis. Therefore, while the spatial and frequency domains are only two special orthogonal orders, the FRFT of fractional order p can provide any intermediate representation between the spatial and frequency domains. This property allows for a unified spatial-frequent characterization and provides multiple perspectives for image signal processing.
Despite its promising merits, the FRFT faces a challenge in terms of fast implementation. Unlike the Fast Fourier Transform (FFT) for the FT, there is currently no encapsulated package available for the fast implementation of the FRFT. To address this, we develop a fast implementation of the 2D FRFT. Building on these advancements, we have introduced simple yet effective operator, the Multi-order Fractional Fourier Convolution (MFRFC), which processes images from multiple perspectives the frequal-f plane. The MFRFC consists of three different order paths: spatial perspective, a frequency perspective, and a unified spatial-frequent order operations are applied to disjoint subsets of feature channels, and the updated feature maps aggregated to produce the output. MFRFC inherently preserves the advantages of both spatial and frequency operations. More importantly, the additional unified spatial-frequency representation reflect the spatial-frea frequency joint characteristics of signals. Our proposed MFRFC is a general and basic operator that can be easily integrated into existing methods. We have experimentally evaluated our operators on various tasks, including object detection, image classification, guided super-resolution. denoising, dehazing, deraining, and low-light enhancement. The results consistently demonstrate significant performance improvements across all the evaluated tasks.
representation perspective. These various
2
Related Work
Spectral neural networks. recent years, there has been a surge of interest in spectral neural network research 3
4
6
9
Previously used to exclusively speed up convolutions, the spectral

---

domain now also serves as a critical building element for deep networks. For example, the authors in proposed fast Fourier convolution, which has the main hallmarks of non-local receptive fields and cross-scale fusion within the convolutional unit. Rippel et al. [ developed the spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain. frequency-doman dynamic pruning scheme was designed in [6] to exploit the spatial correlations for
network compression and acceleration. Some methods [10 I I2 [3 14 [5] also employ Fourier analysis into image restoration tasks. Our method contributes to the above- mentioned research by generalizing existing methodologies and constructing a unified frequuecy operating unit that processes various perspectives simultaneously.
analysis tool. Spatial-frequency frequency analysis is a powerful method that provides joint distribution information in both the spatial and frequency domains, offering clear description of the relationship between signal frequency and spatial variation. There are several common spatial- frequency analysis tools, including the short-time Fourier transform (STFT) [66 17 and wavelet transform (WT) [18]. The STFT applies a local spatial window to the signal and performs the Fourier transform within this window, assuming that the signal within the time window is stationary. However, it faces the dilemma of frequency and spatial resolution balance. The WT tackles this limitation by adopting an adaptive frequency window. However, the WT faces its limitations, including the optimal choice of wavelet basis function, higher computational complexity, and signal distortion. In this paper, we introduce another -requen-fr analysis tool, FRFT. More importantly, we achieve the fast implementation of 2D FRFT and apply it to different image processing tasks.
Fractional Fourier Transform. The FRFT is a powerful spatial-frequency analysis tool used for signal processing due to its unique capability of capturing the non-stationary characteristics of signals 19.. For example, Ozaktas et al. (20] demonstrated that the Tigner-Ville Distribution of the pth order Fractional Fourier Transform is obtained from the pT/2 angle rotation of the original signal's 's WVD. The FRFT has also found applications in multimedia. including digital watermarking, image compression, and image eneryption systems [2] 22, 23). In communication, the FRFT has been implemented in wireless communications and transmission schemes [24 25). Additionally, Zhang et al. 26 investigated the use of FRFT as framework for biomedical signal detection. In this work, we explore the property of FRFT for deep learning. Based on our exploration, we propose a novel convolutional operator that takes advantage of the unique properties of FRFT.
3 Preliminary of FRFT for deep learning
The Fractional Fourier Transform (FRFT) is a powerful and versatile analysis tool that has been relatively under-explored in the context of deep learning. In this section, we aim to reveal the unique properties of FRFT and explain why it is a valuable addition to the deep learning toolkit.
Definition We first introduce the mathematical definition of FRFT. The 1-D p-th order FRFT of a signal z(u)), where uo represents the initial domain, i.e., time, spatial, frequency, or even an intermediate domain, is defined as follows:
fx (nw)} = X (up)= / Kp (Ma, Up) x (Mo) dua-
Specifically, the kernel Kp (uo, up) is given by the following expression:
An Angl(ngonn oot cot/eoot/2/20/ aaaeer/2-uppp escn 2 cat //22,
# nm
Kp up) -
8(1 w), 8(1 u).
2nu
(2)
(2n 1)m
where = cot and a Pr. Fq. can be called either p-th order Fractional Fourier Transform of signal (ug) or a Fractional Fourier Transform of signal (wa) at angle a. For FRFT, the basis function is linear frequency modulation (LFM) signal, as shown in the kernel function, instead the sine function basis in Fourier transform. Besides, when a = x/2, 2, the formula of FRFT degrades to the formula of FT, which is proved with details in the supplementary material.
Fast diserete implementation of FRFT FT has a well-established fast digital computation algo- rithm that can compute it in N log N time (27]. However, despite the availability of this algorithm,
3

---

there is currently encapsulated package for the fast implementation of FRFT. Furthermore, most existing implementation methods cannot match the computational speed of Fourier transform in practical applications. To address this issue, we have adopted matrix multiplication implementation 28 that was originally developed for natural language processing and adapted for the field of computer vision. Through this adaptation, we have developed an improved fast version of FRFT that as fast as Fourier transform practical settings. This implementation offers a valuable tool for researchers and practitioners seeking to leverage the benefits of FRFT in their work.
FRFT, Wigner distribution, and spatial-frequency plane rotation signal (xx is defined as follows:
The Wigner distribution of
W (x 1) = (x (z= s2avd' de'
The function Wr(2, 2) provides a distribution of signal energy over time and frequency, and it serves as key link between FRFT and spatial-frequency plane rotation. Specifically, the Wigner distribution of the p-th order FRFT of a signal fx) is equivalent to a counterclockwise rotation of Wj (zx, v) by an angle of p2/2 degrees 27
Non-stationary signal non-stationary signal is one whose statistical properties vary with time space, including the mean, variance, and frequency. In contrast to stationary signals, which are often artificially created, natural signals are almost always non-stationary. Natural images are also non-stationary signals since their statistical properties change significantly over space, depending the content of the image. For example, the mean and variance of an image may vary depending on factors such lighting conditions, color palette, image composition. Additionally, images may contain non-stationary features such as edges, textures, and patterns that change over space. These non-stationary features play crucial role many computer vision tasks. Thus, FRFT is well-suited for the processing of the non-stationary image signal.
(a) p=0
(b) p-0.2
(c) p-0.4
(d) p-0.6
(e) P-0.8
(f) p-
Figure 2: The amplitude and phase spectrums of the FRFT of different orders. The upper row is amplitude spectrums, and the bottom is phase spectrums. Note that the fractional order's amplitude spectrum contains the signals' spatial-frequency joint characteristics.
The amplitude and phase spectrums of image in FRFT As shown in Fig. 2 the first row is the amplitude spectrum of FRFT in different orders, and the second is the phase spectrum. For the amplitude spectrum, when the order is small, the outline and detail information of the image can be clearly observed. With the increase of the order, the image gradually becomes blurred, and the energy becomes more and more concentrated. It shows that the spatial domain information in the amplitude spectrum decreases with the increase of the transformation order. contrast, the frequency domain information increases with the rise of the transformation order.
The comparison between spatial image, FT and FRFT Table Table presents comparison between spatial image, Fourier Transform (FT), and Fractional Fourier Transform (FRFT). Traditional FT powerful tool for processing stationary signals, with the sine function as the basis function. In contrast, FRFT is a spatial -ratial-frequency analysis tool that is particularly effective for processing non-stationary signals, with linear frequency modulation signal as the basis function. Spatial and FT are two orthogonal individual perspectives, with the spatial image corresponding to the horizontal s-axis in the spatial- freuunllfreuencyy plane shown in Fig. D and FT corresponding to the vertical f-axis in the same plane. On the other hand, FRFT can represent unified patial-frequenee perspective in the

---

same plane, providing more comprehensive and integrated view of the signal. Additionally, FRFT introduces the concept of fractional transformation order, which allows for the analysis of the signal's spatial and frequency domain information. In contrast, the FT of a signal only contains frequency domain information, limiting its scope of analysis.
Table I: Comprehensive comparison between original spatial image, Fourier transform, and FRFT.
Difference
Spatial
Fourier transform
FRFT
Applicable signal Basis function Rotation angle
Stationary Sine 90" individual Frequency
Time varying non-stationary Linear frequency modulation a (any) unified
0º individual Spatial
Generality
Information
Spatial & Frequency
4 FRFT operator design
Based on the property of FRFT in image processing and our fast implementation of FRFT, we devise simple yet effective operator, the Multi-order Fractional Fourier Convolution (MFRFC).
4.1 MFRFC
Cannch
Convix1
90"
Convixt
992
Channel Split
FRFT
Process
IFRFT
Channel Concatenate
Architecture design of the Multi-order Fractional Fourier Convolution. MFRFC is com- prised of three different order paths: spatial (p=0) path, frequency (p=1) path, and unified spatial-freee frntual- (p=0.5) path. IFRFT denotes Inverse FRactional Fourier Transform.
Figure 3:
The architecture of our proposed Multi-order Fractional Fourier Convolution (MFRFC) is illustrated in Fig. 3 MFRFC comprises five critical operations: a channel split that equally divides the input features into three parts along the channel dimension, 2D discrete Fractional Fourier Transform that converts the input features to the fractional domain with three different orders, separate processing of these three fractional domains, 2D inverse Fractional Fourier Transform that converts the fractional domain features back to the spatial domain, and a channel merge that concatenates the three features along the channel dimension. These five operations work in concert to create powerful and flexible convolutional layer that can leverage the unique properties of FRFT to analyze non-stationary signals and improve the performance of deep learning models.
The key component of MFRFC is the three different order paths: a spatial (p=0) path that conducts ordinary convolutions, a spectral (p=l) path that operates in the spectral domain, and fractional order (p=0.5) path that operates in the unified spatial-frequest operation, we add a full channel convolution to fuse the information from different perspectives.
domain. After the channel concatenation
5

---

4.2 Merits of MFRFC
The design of MFRFC offers several advantages over vanilla convolution and Fourier transform. Firstly, it preserves all the key merits of vanilla convolution and Fourier transform, such parameter sharing and global receptive field. For instance, spectral theory demonstrates the existence of operator duality between convolution in the spatial domain and element-wiss multiplication in the spectral domain, as shown the following equation:
F(f(x1)+ f(x2)) F(11) F(22),
(4)
where and denote convolution and element-wise product, respectively, This enables FT with image-size receptive field.
Moreover, MFRFC provides the added advantage of analyzing features from multiple perspectives. In particular, the introduction of the fractional domain complements the missing perspective that is optimal for non-stationary signal analysis. The fractional domain is inherently well- sull-s for separating and extracting non-stationary features, offering a powerful tool for analyzing image signals with varying statistical properties over space.
4.3 Fractional order selection
Regarding the path number, we select three paths: traditional spatial and frequency paths, as well additional fractional path. This approach allows us leverage the two orthogonal axes of the spatial-frequency plane, as well as the intermediate fractional axis, to provide more comprehensive and representative signal projection perspectives.
In terms of fractional order selection, our experimental results demonstrate that different orders can improve performance D For simplicity, we choose fixed order p=0.5 based on empirical observations. While this choice is simple, it still provides noticeable performance improvement on various tasks, demonstrating the effectiveness of incorporating FRFT, which is key focus of our work. In addition, we also provide an extended version of MFRFC, called MFRFC++, in the supplementary material, which enables the network to learn the optimal order adaptively.
4.4 Complexity analysis
Table 2 compares two major complexity metries of the Multi-order Fractional Fourier Convolution (MFRFC) and vanilla convolution. Thanks to the fast implementation of FRFT, the complexity of FRFT and inverse FRFT is omitted since they are parameter-free and their time complexities are negligible compared to other computation costs. As shown, the computational cost of FRFT is comparable to that of vanilla convolution. Therefore, in specific usage, we can simply insert the MFRFC once into different networks to improve performance with negligible parameter increase.
Table 2: Parameter counts and FLOPs for vanilla convolution, separate components in MFRFC, and entire MFRFC, respectively. C1 and C2 the numbers of input and output channels, respectively. We equally split the input feature channel for every path, but for order=0.5 and order=1 paths, the concatenation of the real and imaginary parts doubles the channel number. So the final channel number for these two paths is of the original. H and W collectively define spatial resolution. K is the convolutional kernel size. For elarity, here, stride and padding
considered.
Params
FLOPss
vanilla
CiCaK²
HW
Path1 (order=0)
k
kG,CK3 HW C,C2K3 HW
Path2 (order=0.5)
10,02
Path3 (order=1)
CiCz
Path fusion
C,C.K²
H.C,KHHW
MFRFC
(1 + $ 9
6
(1+ 1 9K HH
TTe relationship between order and performance is provided the supplementary material.

---

Experiments
To demonstrate the effectiveness of our proposed convolutional operator MFRFC, we conduct exten- sive experiments over multiple computer vision tasks, including object detection, image classification, guided super-resolution, denoising, dehazing, deraining, and low-light enhancement.
5.1 Evaluation Protocols
Object detection. Following (29 the PASCAL VOC 2007 and 2012 training sets (30] are adopted for training, and the PASCAL VOC 2007 testing set is used for evaluation. We employ the FPN-based Faster RCNN with ResNet50 backbone as baseline.
Image classification. We verify our method on the image classification task with the CIFAR-10 dataset [3T] For the baseline, we employ the classical classification method: ResNet [32]
Image denoising. Following [33], to evaluate our method on the image de-noising task, we employ the widely-used SIDD dataset as training benchmark. Further, the corresponding performance evaluation is conducted on the remaining validation samples from the SIDD dataset 34 For the
baseline, we employ the representative denoising method: DrCNN
35
Image enhancement. We verify our method on the image enhancement task with the commonly used dataset, LOL (36). Further, we adopt the two classical baselines, SID 37 and DRBN (38).
Image de-hazing. Following [39]. to evaluate our method on the image dehazing task, we employ ITS and SOTS indoor [40] as our training and testing datasets. For the baseline, we adopt the representative dehazing methods: GridNet [41] and MSBDN 42)
Image de-raining, For validation, we used the widely-used standard benchmark dataset Rain200H. as described in 43 As baselines, we use the representative de-raining methods: PReNet 43
we adopt pan-sharpening, the representative task of guided image super-resolution for evaluating the effectiveness of the MFRFC. The World
Guided image super-resolution. Following 44
45),
II, WorldView III, and GaoFen2 in [4 45) are used for evaluation. We employ two different network designs for validation: PanNet [46, and FusionNet (47). Several widely used image quality assessment (IQA) metrics are employed to evaluate the performance, including the peak signal-to- noise ratio (PSNR), the structural similarity index (SSIM), the spectral angle mapper (SAM) (48). and the relative dimensionless global error in synthesis (ERGAS) 44
5.2 Implementation details
Baseline variant. Besides our MFRFC operator, we also validate baseline variant called Spatial- Frequency Convolution (SFC). Specifically, SFC has two order paths, spatial path (order=0) and a frequency path (order=0). Our MFRFC has additional fractional path than SFC and is supposed to perform better than SFC.
Three configurations. We re-implement the baseline methods following the settings in the corre- sponding paper. this paper, we denote baseline models as "Original". baseline models equipped with SFC as "SFC", and baseline models provided with MFRFC as "MFRFC". We present an example usage of SFC and MFRFC in the supplementary material.
5.3 Analysis and visualization
Quantitative Comparison. We compare model performance over different configurations, as described in implementation details. The quantitative results are reported in Table 3 and It is evident that, by integrating with our proposed MFRFC operator, all "MFRFC obtain performance gain over the "Original" and "SFC" baselines on all the datasets in the evaluated task, which fully validates the effectiveness of our method.
5 7 g
Qualitative Comparison. Due to the page limits we only show the visual results over the representative tasks of guided image super-resolution, image deraining, and image dehazing in Fig. 4 5 and respectively, to better illustrate the effectiveness of MFRFC. It can be casily figured out
Mooe visual comparison results reported in the supplementary materials.
7

---

that "MFRFC" generates the highest-fidede results that also look perceptually close to the reference ground-truths. For example, zooming in the red box region in Figure better recover the texture details. Similar conclusions also apply to Fig. 6
4 the "MFRFC" model can
Table 3: Quantitative comparison over object de-
Table 4: Quantitative comparison on image classi- fication.
tection.
Model
Methods
AP50
mAP
Model
Methods
Acc
Original
77.30
77.26
Original
95.23
Faster RCNN
SFC
77.90
77.89
ResNet
SFC
95.34
MFRFC
78.00
77.96
MFRFC
95.43
Table 5: Quantitative comparison on image denois-
Table 6: Quantitative comparison on image derain- ing.
ing.
Model
Methods
PSNR
SSIM
Model
Methods
PSNR
SSIM
Original
37.1992
0.8904
Original
29.041
0.891
DrCNN
SFC
37.3454
0.8866
PReNet
SFC
29.498
0.901
MFRFC
37.6084
0.8915
MFRFC
29.735
0.906
Table 7: Comparison of quantitative results over guided image super-resolution.
WorldView-II SSIM
GaoFen2
Model
Methods
PSNR
SAM
ERGAS
PSNRT
SSIM
SAM.
EGAS
Original
40.817
0.9626
0.0257
1.0557
43.0659
0.9685
0.0178
0.8577
PanNet
SFC
41.2118
0.9663
0.0236
0.9901
44.4965
0.9801
0.0161
0.7761
MFRFC
41.3428
0.9667
0.0233
0.9898
44.6245
0.9804
0.0146
0.6551
Original
41.0708
0.9651
0.0248
0.9978
44.8852
0.9810
0.0140
0.6661
FusionNet
SFC
41.3941
0.9674
0.0232
0.9898
45.1071
0.9810
0.0138
0.6413
MFRFC
41.5829
0.9676
0.0229
0.9766
45,2534
0.9876
0.0137
0.6377
Table 8: Quantitative comparison over image de-
Table 9: Quantitative comparison low-light im- age enhancement.
hazing.
Model
Methods
PSNR
SSIM
Model
Methods
PSNR
SSIM
Original
32.56
0.9847
Original
20.1062
0.7895
GridNet
SFC
32.97
0.9856
SID
SFC
21.9222
0.7988
MFRFC
33.39
0.9887
MFRFC
22.3666
0.7998
Original
32.19
0.9813
Original
19.7931
0.8361
MSBDN
SFC
32.26
0.9825
DRBN
SFC
21.8372
0.8356
MFRFC
32.45
0.9838
MFRFC
21.9916
0.8426
(a) Input
(b) Original
(c) SFC
(d) MERFC
(e) GT
Figure 4: The visual comparison of PANNET over WorldView-II. The bottom is map.

---

(a) Input rany
(b) Otiginal
(c) SFC
(c) MFRIC
(c) GT
Figure 5: The visual comparison of PReNet over the Rain200H.
Input hazy
Original
SRC
MFRFC
GT
Figure The visual comparison of GridNet over SOTS.
Limitations and Discussions
We evaluate the effectiveness of the proposed method several computer vision tasks, and will conduct more extensive experiments on other computer vision tasks (e.g.. image segmentation and video processing). Furthermore, on the specific task and may be critical for the further understanding and application of FRFT in specific task. To emphasize, the critical contribution of this work is that we clear the way to the popularization of FRFT, and make the first attempt to integrate FRFT into a basic operator.
recognize that the optimal fractional order may vary depending
7 Conclusion
In this paper, we introduce a new spatial-frequency analysis tool, FRFT, to generalize existing FT and provide comprehensive perspectives for vision tasks. The FRFT unified paaiial-rrquuecy transform, simultancously reflecting an image's 's spatial and frequency representations. Notably, we explore the properties of FRFT for image processing and achieve the fast implementation of 2D FRFT. Besides, we devise the convolutional operator MFRFC, which has the impressive merits of processing an image from more perspectives the spatial-frequency plane. The MFRFC is general convolutional operator and thus can be easily integrated into existing methods to improve their performance further. Extensive experiments have demonstrated the effectiveness of our method.
Broader Impact
The past decades have witnessed the development computer vision driven by convolution neural networks. However, vanilla spatial convolution is only one common perspective of signal processing. Some spatial-frequency combination methods have already exhibited potential of two-perspective signal processing. As general version of FT, FRFT is quite promising for image processing but is less explored largely due to the missing fast discrete implementation algorithm and unclear properties in image processing. In this paper, have cleared all these obstacles to the popularization of

---

FRFT and made the first attempt to integrate FRFT into novel convolutional operator, which has demonstrated its effectiveness in various computer vision tasks. Besides, we hold the belief that there is much more space for the further exploration of FRFT in deep learning. Additionally, our work may inspire the rethinking of neural networks from more diverse perspectives.
Acknowledgements
This work was supported by the JKW Research Funds under Grant 111-166-1---2-01-0004---- and the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built MCC Lab of Information Science and Technology Institution, USTC.
10

---

References
[1] Yitzhak Katznelson, An introduction to harmonic analysis. Cambridge University Press, 2004.
[2] Haldun M Ozaktas, M Alper Kutay, and David Mendlovic. Introduction to the fractional Fourier transform and its applications. In Advances in Imaging and Electron Physics, volume 106, pages 239-291. Elsevier, 1999.
[3] Lu Chi, Borui Jiang, and Yadong Mu, Fast Fourier convolution. Advances in Neural Information Processing Systems, 13:4479-4443 2020.
[4] Oren Rippel, Jasper Snoek, and Ryan P Adams. Spectral representations for convolutional neural networks. Advances in Neural Information Processing Systems, 28, 2015.
[5] Zhisheng Zhong, Tiancheng Shen, Yibo Yang, Zhouchen Lin, and Chao Zhang. Joint sub- bands learning with clique structures for wavelet domain super-resolution. Advances in Neural Information Processing Systems, 31, 2018.
[6] Zhenhua Liu, Jizheng Xu, Xiulian Peng, and Ruiqin Xiong. Frequency-domaie dynamic pruning for convolutional neural networks. Advances in Neural Information Processing Systems, 31, 2018.
[7] Michael Mathieu, Mikael Henaff, and Yann LeCun. Fast Training of Convolutional Networks through FFTs. arXiv preprint arXiv:1312.585)/ 2013.
[8] Harry Pratt, Bryan Williams, Frans Coenen, and Yalin Zheng. FCNN: Fourier Convolutional Neural Networks. In Joint European Conference on Machine Learning and Knowledge Discov- ery in Databases, pages 788-798. Springer, 2017.
[9] Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengho Ren. Learning in the frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1740-1749, 2020.
[10] Hu Yu, Naishan Zheng, Man Zhou, Jie Huang, Zeyu Xiao, and Feng Zhao. Frequency and spatial dual guidance for image dehazing. In European Conference on Computer Vision, pages 181-198. Springer, 2022.
[11] Man Zhou, Jie Huang, Keyu Yan, Hu Yu, Xueyang Fu, Aiping Liu, Xian Wei, and Feng Zhao Spatial-frequency domain information integration for pan-sharpening. In European Conference on Computer Vision, pages 274-291. Springer, 2022.
[12] Man Zhou, Hu Yu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy, Deyu Meng, and Chongyi Li. Deep fourier up-sampling. arXiv preprint 2r0iv:22/0.0517,, 2022.
[13] Man Zhou, Jie Huang, Chongyi Li, Hu Yu, Keyu Yan, Naishan Zheng, and Feng Zhao. Adap- tively learning low-high frequency information integration for pan-sharpening. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3375-3384, 2022.
[14] Zizheng Yang, Jie Huang, Jiahao Chang, Man Zhou, Hu Yu. Jinghao Zhang, and Feng Zhao Visual recognition-drive -rrivn image restoration for multiple degradation with intrinsic semantics recovery. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14059-14070, 2023.
[15) Yu, Jie Huang, Yajing Liu, Qi Zhu, Man Zhou, and Feng Zhao Source-free domain adaptation for real-world image dehazing. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6655-6654, 2022.
[16] Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transuctions acousties, speech, and signal processing, 32(2):236-243, 1984.
[17] Lutfiye Durak and Orhan Arikan. Short-time fourier transform: two fundamental properties and an optimal implementation. IEEE Transactions on Signal Processing, 11(5):1231-1242, 2003.
[18] Marie Farge Wavelet transforms and their applications to turbulence. Annual review of fluid mechanics, 24(1):395-45 4588, 1992.

---

[19] Victor Namias. The fractional order Fourier transform and its application to quantum mechanics. IMA Journal of Applied Mathematics, 25(3):241 1980.
[20] Haldun M Ozaktas, Billur Barshan, David Mendlovic, and Levent Onural. Convolution, filtering, and multiplexing in fractional Fourier domains and their relation to chirp and wavelet transforms. JOSA A, 11(2):547-555 1994.
[21] Musrrat Ali, Chang Wook Ahn, and Millie Pant. An efficient lossless robust watermark- ing scheme by integrating redistributed invariant wavelet and fractional Fourier transforms. Multimedia Tools and Applications, 1751-11175 2018.
[22] R Naveen Kumar, BN Jagadale, and JS Bhat. A lossless image compression algorithm using wavelets and fractional Fourier transform. SN Applied Sciences, 1(3):1-8, 2019.
[23] Di Zhang, Xiaofeng Liao, Bo Yang, and Yushu Zhang. A fast and efficient approach to color- image encryption based compressive sensing and fractional Fourier transform. Multimedia Tools and Applications, 77(2):2191 2018.
[24] Yong Li, Zhiqun Song, and Xuejun Sha. The melth-weighted type fractional Fourier transform scheme and its application over wireless communications. EURASIP Journal Wireless Communications and Networking, 2018(1):1-10, 2018.
[25] Lei Ni, Xinyu Da, Hang Hu, Yuan Liang, and Ruiyang Xu. Phy-aided secure communication via weighted fractional Fourier transform. Wireless Communications and Mobile Computing, 2018, 2018.
[26] Yalin Zhang, Qinyu Zhang, and Shaohua Wu. Biomedical signal detection based on frac- tional Fourier transform. In 2008 International Conference Information Technology and Applications in Biomedicine, pages 349-352. IEEE, 2008.
[27] Haldun M Ozaktas, Orhan Arikan, M Alper Kutay, and Gozde Bozdagt. Digital computation of the fractional fourier transform. IEEE Transactions on signal processing, 4499):2141-2155, 1996.
[28] Furkan Sahinuę and Aykut Koe. Fractional fourier transform meets transformer encoder. IEEE Signal Processing Letters, 29:2258-2266 2022.
[29] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6): 133--119,, 2017.
[30] M. Everingham, L. V. Gool, Cki Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-335 2010.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[33] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings the IEEF/CVE conference on computer vision and pattern recognition, pages 14821-14831, 2021.
[34] AbdeIrahman Abdelhamed, Stephen Lin, and Michael Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1692-1700, 2018.
[35] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions on image processing. 26(7):3145) 1103:31222 42-3155, 2017.
[36] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv: 1808. 0456004560 2018.
12

---

[37] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3291- 33000, 2018.
[38] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to percep- tual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3063-3072 2020.
[39] Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang. Yuan Xie, and Lizhuang Ma. Contrastive learning for compact single image dehazing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 10551-10560, 2021.
[40] Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng. Wenjun Zeng, and Zhangyang Wang. Benchmarking ingle-image dehazing and beyond. IEEE Transactions on Image Processing. :8(1):492-500 505, 2019.
[41] Xiaohong Liu, Yongrui Ma. Zhihao Shi, and Jun Chen. Griddehazenet: Attention-basel multi- scale network for image dehazing. Proceedings of the IEEE/CVF international conference on computer vision, pages 7314-7323, 2019.
[42] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2154-2164, 2020.
[43] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng, Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[44] Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. Effective pan- sharpening with transformer and invertible neural network. IEEE Transactions on Geoscience and Remote Sensing, 60:1 1-5,, 2022.
[45] Keyu Yan, Man Zhou, Liu Liu, Chengjun Xie, and Danfeng Hong. When pansharpening meets graph convolution network and knowledge distillation. IEEE Transactions Geoscience and Remote Sensing, 60:1-15, 2022.
[46] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. Pannet: A deep network architecture for pan-sharpening. In IEEE International Conference on Computer Vision, pages 5449-5457, 2017.
[47] Liang-Jian Deng, Gemine Vivone, Cheng Jin, and Jocelyn Chanussot. Detail injection deep convolutional neural networks for pansharpening. IEEE Transactions on Geoscience and Remole Sensing, 59(8):6995-7010, 2020.
injection-based based
[48] Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. In JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop, 1992.
[49] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce. Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data fusion contest. IEEE Transactions on Geoscience and Remote Sensing, 4511033301--3011, 2007.
13


Reviews:
[{"review_id": "pTazPYYIC7", "reviewer": "Reviewer_Cjk7", "summary": "1. This technique delves into novel fundamental operators for deep learning, the Fractional Fourier Transform (FRFT), exploring a new perspective of signal processing between two orthogonal domains (spatial and frequency domains). \n\n2. This technique have implemented a fast and differentiable Multi-order Fractional Fourier Transform Convolution, an elegant combination of deep learning and traditional image processing. \n\n3. Unignorably, as an theoretical extension of FFC, it is expected to make positive contributions to the research community.", "questions": "In Figure 2, compared to the spatial and frequency domains, the fractional amplitude spectra exhibit a scaling effect. Why is this phenomenon present?", "limitations": "Yes. The limitations and potential negative societal impact of this paper have been addressed in this paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This work provides a solid foundation for delving into this unexplored territory for deep learning. A unified analysis tool in the spatial-frequency domain, known as the Fractional Fourier Transform (FRFT), has been introduced, accompanied by sufficient theoretical analysis with following points: \n\n+ The spatial and frequency domains have been extensively explored, while the intermediate chaotic region between the two has been underestimated. \n\n+ The author has developed a fast implementation of the 2D FRFT, enabling comprehensive image processing from multiple perspectives in the spatial-frequency plane.\n\n+ Sufficient experimentation. The operators were experimentally evaluated on a range of vision tasks and results demonstrate its substantial performance improvements.\n\n+ Overall, the study is solid, authors also provide executable code implementation. It is a valuable supplement to deep learning-based vision toolkits.", "weaknesses": "There are also few concerns:\n- The three-branch design in MFRFT indeed reasonable. However, it is worth considering whether there is a justifiable explanation for utilizing 1x1 convolutions to process signals in fractional domains. \n- Additionally, has the author taken into account the possibility of handling this chaotic information through the MLP layers instead of a unified convolutional operator?\n\n-The introduced FRFT is mainly applied to CNN-based architectures. The transformer has also demonstrated strong capacities and performance. Thus, the  authors may also need to discuss the possibility of introducing FRFT in transformer architecture. The basic operator for transform may be different from that for CNN.\n- How FRFT in low-level vision tasks and high-level vision tasks. In FFC, it only conduct experiments in high-level vision tasks. Why authors choose low-level vision tasks for experiments conduction?\n- Several minor issues that need to be modified.\n(1) Inconsistent abbreviation for the citation of figures. Figure in line 248 and Fig. in line 249.\n(2) Figure 1 is easy to understand, but the detail of this figure may need further improving for beauty.\n(3). The caption for Table 2 is abnormally long, compared to other Figures and Tables.\n(4) The evaluation for guided image super-resolution task adopts two more metrics than all other evaluated low-level tasks, which may be redundant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1: 1x1 convolutions in fractional domains.**   \nFRFT is a generalized and extended version of Fourier transform. For Fourier transform, spectral theory demonstrates the existence of operator duality between convolution in the spatial domain and element-wise multiplication in the spectral domain, and thus 1x1 convolution is a default setting in spectral domain [1] [2]. Correspondingly, we also employ 1x1 convolution for fractional fourier domain.    \n[1] Fast Fourier convolution. NeurIPS, 2020.    \n[2] Deep Fourier Up-sampling. NeurIPS, 2022.  \n\n**2: FRFT in CNN, MLP and Transformer architectures.**   \n(1) Our work is based on FFC [1] with an theoretical extension. Following this baseline work, we design a convolutional operator for CNN-based architectures. Most of previous networks are CNN-based, endowing our method with promising application prospect.  \n(2) Fourier transform has been explored in both MLP [2] and Transformer [3] architectures. Our work is an improved version of Fourier transform, and can also been explored in these architectures as a future work.  For example, GFNet [3] replaces the self-attention sub-layer with the frequency filter layer via Fourier transform. Such operation can be directly replaced with FRFT. Besides, FRFT can be applied to more architectures and tasks since our work unlocks FRFT in deep learning.  \n[1] Fast Fourier convolution. NeurIPS, 2020.  \n[2] Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.  \n[3] Global filter networks for image classification. NeurIPS, 2021.\n\n\n**3: FRFT in both low-level and high-level vision tasks.**    \n(1) FRFT is a basic tool and our designed MFRFC is a general operator. Thus we conduct comprehensive experiments on both low-level and high-level vision tasks to validate the effectiveness of our method.   \n(2) Besides the consideration of comprehensive verification on different tasks, we find support that FRFT is closely related to low-level vision tasks [1]. This also motivates us to verify our method on low-level vision tasks.  \n[1] Optimal image restoration with the fractional Fourier transform. JOSA A, 1998.\n\n\n**4: Scaling effect in the fractional amplitude spectra.**   \nThe scaling effect in the fractional amplitude spectra is the inherent characteristic of FRFT. It is to do with the projection of fractional domain signal on the spatial domain, as shown in Figure 1 in the main manuscript. As the fractional order varies from 0 to 1, the energy projection on the spatial domain gets less, demonstrating as the scaling effect. This phenomenon is also explained as the energy distribution property of FRFT in the supplementary material."}, {"review_id": "UHnFuhFiS9", "reviewer": "Reviewer_TTF6", "summary": "This paper proposed a fractional Fourier transform-based module, which can simultaneously exploit the information from spatial and frequency perspectives. With a fast implementation of FRFT, multi-order MFRFC module can be easily incoporated to existing convolutional networks for different tasks. ", "questions": "see weakness", "limitations": "see weakness", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A unified spatial-frequency analysis module based on fractional Fourier transofrm is proposed\n2. The proposed MFRFC module is applied to several tasks including denoising, deraining, classification, dehazing, detection etc, where the proposed module brings performance gains. ", "weaknesses": "1. The main issue is the methods for different tasks are not new, e.g., DnCNN has been much inferior to recent denoisers in terms of quantitative metrics. So it should be evaluated whether the proposed module still works for recent denoisers with better performance. I guess the perfmance gains would be not much significant. Currently, I am at the borderline leaning to accept this work. But if my guess is correct, the contribution of this work would be not so significant, and may turn down my rating. \n\n2. It is not clear how the MFRFC module is applied in existing methods for different tasks, e.g., for DnCNN, all the convolutional layers are replaced?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1: Baseline methods for different tasks.**   \n(1) Our method is not SOTA-oriented. Instead, the key of this paper is that we unlock the FRFT in deep learning paradigm, solving the biggest challenge for the popularization of FRFT: vague characteristics and missing fast implementation. Fourier transform is pretty useful and vastly employed in deep learning era. As a generalized and improved version of Fourier transform, FRFT surely has great application prospect and exploration value.    \n(2) We validate the effectiveness of FRFT on various tasks with classical and commonly acknowledged baselines. We select the classical methods as baselines for three reasons: 1. We follow the setting of previous basic operator-based methods [1][2], which also employ classical methods as baselines. 2. The selected baselines are the representative and commonly acknowledged works in the related tasks. In addition, as a general operators, implementing the proposed operator in the standard benchmarks is more fair. 3. The performance of classical methods are highly likely to be reproducible.  \n(3) To further solve your misgiving, we also apply our operator to recent SOTA baselines. We choose GPPNN [3] and LACNET [4] for guided image super-resolution task on WorldView-II dataset, NBNet [5] and Restormer [6] for image denoising task on SIDD dataset, URetinex [7] and SNR [8] for low-light image enhancement task on LOL dataset. As can be seen from the following three tables, our method still largely elevates the performance of these SOTA baselines.  We will add these experiments and conduct more experiments on remaining tasks in the final version. Besides SOTA baselines, our method may be critical to small networks in the practial deployment,  which shows the irreplaceable and important role of this basic tool and operator.\n\n| Model | Methods |PSNR | SSIM |\n| :-----| ----: | ----: | :----: |\n| GPPNN  | Original | 41.16  | 0.968  |\n|        | SFC      | 40.14  | 0.967  |\n|        | MFRFC    | 41.47  | 0.972  |\n| LACNET | Original | 41.45  | 0.972  |\n|        | SFC      | 41.52  | 0.973  |\n|        | MFRFC    | 41.74  | 0.975  |\n\n| Model | Methods |PSNR | SSIM |\n| :-----| ----: | ----: | :----: |\n| NBNet     | Original | 39.75  | 0.959  |\n|           | SFC      | 39.86  | 0.959  |\n|           | MFRFC    | 39.97  | 0.960  |\n| Restormer | Original | 40.02  | 0.960  |\n|           | SFC      | 40.05  | 0.960  |\n|           | MFRFC    | 40.19  | 0.961  |\n\n\n| Model | Methods | PSNR | SSIM |\n| :-----| ----: | ----: | :----: |\n| URetinex | Original | 21.33  | 0.835  |\n|          | SFC      | 21.82  | 0.837  |\n|          | MFRFC    | 22.35  | 0.839  |\n| SNR      | Original | 24.61  | 0.842  |\n|          | SFC      | 24.84  | 0.846  |\n|          | MFRFC    | 24.98  | 0.848  |\n\n\n[1] Fast Fourier Convolution. NeurIPS, 2020.  \n[2] Deep Fourier Up-sampling. NeurIPS, 2022. \n[3] Deep gradient projection networks for pan-sharpening. CVPR, 2021.     \n[4] LAGConv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening. AAAI, 2022.   \n[5] NBNet: Noise basis learning for image denoising with subspace projection. CVPR, 2021.   \n[6] Restormer: Efficient transformer for high-resolution image restoration. CVPR, 2022.   \n[7] URetinex-Net: Retinex-based Deep Unfolding Network for Low-light Image Enhancement. CVPR, 2022.    \n[8] SNR-Aware Low-light Image Enhancement. CVPR, 2022.   \n\n\n\n**2: Implementation of MFRFC.**    \nWe apply the MFRFC in the middle layer of the network. We find that this is enough to incorporate fractional domain information into the network and can get significant performance improvement with the computational burden introduced by our method negligible."}, {"review_id": "u2mEAIQ7IX", "reviewer": "Reviewer_c5M1", "summary": "The paper discusses Fractional Fourier Transform (FRFT) in the context of deep learning based computer vision methods. FRFT is a unified continuous spatial-frequency transform which reflects spatial and frequency representations of images. Based on FRFT, the paper proposes a new convolutional operator (MFRFC) which is more suitable for image signal processing. With MFRFC, the networks achieve consistent performance improvement across several computer vision tasks.", "questions": "The introduced FRFT is a spatial-frequency analysis tool of signals. But, there also exists other spatial-frequency analysis tools including wavelet transform and Gabor transform, among which wavelet transform has been widely employed in the deep learning methods. What’s the difference between FRFT and these spatial-frequency analysis tools?", "limitations": "The authors have discussed the limitations from different aspects. While they do not include sufficient discussions about the computational cost, which are suggested to be included. Since the authors claim that they solved the fast implementation of FRFT, it is encouraged to clearly compare the speed of the original discrete FRFT, the author’s fast implementation version, and the fast implementation of discrete Fourier transform (FFT). ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ The paper proposes a good idea to firstly introduce the well-known FRFT in recent deep learning-based methods and achieves substantial performance improvement on various tasks and networks.\n+ The proposed method is simple yet effective. The fast implementation of 2D discrete FRFT is vital to the research community. The experimental results show empirical improvement in visual quality in image restoration tasks.\n+ The writing is clear and lucid. The experimental settings are clearly described.\n", "weaknesses": "- The concept and properties of FRFT is broad. This paper introduces certain properties of FRFT, but does not comprehensively investigate the FRFT. The proposed operator MFRFC is very effective, while may also be treated as one solution of all the possibilities.\n- The paper lacks comparisons with other spectral-related methods that also try to incorporate spectral information in the deep learning pipeline (for example, the FFC method mentioned in the related work).\n- MFRFC has three different order paths: a spatial (p=0) path, a spectral (p=1) path, and a fractional order (p=0.5) path. The authors should also evaluate how the order of fractional order path effects performance.\n- Visual comparison is incomplete regarding the conducted experiments. The visual results for image denoising and low-light enhancement are missing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1: Properties of FRFT.**   \nIt is true that the properties of FRFT is broad. Here, we introduce the main properties inherent in FRFT. For more properties associated with specific tasks, we believe that with our pioneer work, it will be easier to explore them in the near future. Similar case applies to Fourier transform. Fourier transform itself has only several main properties. However, with the fast implementation FFT, researcher can easily explore the properties of Fourier transform with specific tasks [1][2]. With our work, similar combination can also be made be with FRFT.    \n[1] Learning frequency-aware dynamic network for efficient super-resolution. ICCV, 2021.  \n[2] Frequency and spatial dual guidance for image dehazing. ECCV, 2022.   \n\n\n**2: Comparisons with other spectral-related methods.**    \nIn fact, we have compared with other spectral-related methods. Previous similar spectral-related methods mainly works in a two-branch manner with spatial and frequency branches. They employ Fourier transform for the frequency branch. In our method, the SFC method works in a similar way with spatial and frequency branches. SFC is designed with the same manner as our three-branch MFRFC operator, with the only difference in fractional branch. Such design is enough to demonstrate the superiority of FRFT over Fourier transform in a fair setting.  \n\n\n**3: Relationship between order and performance.**   \nWe explore the relationship between order and performance in the supplementary material, including different orders and adaptive order. We draw two conclusions from our empirical results. First, different fractional orders in the MFRFC operator all can significantly elevate the performance over the original baseline, with slight difference between different fractional orders. Secondly, the adaptive order version achieves nearly optimal performance among different fractional orders. Besides, our selection order=0.5 in the main manuscript also gets relatively optimal performance among different fractional orders.\n\n\n**4: Difference with other spatial-frequency analysis tools.**   \n(1) Gabor and Wavelet transform are also time-frequency analysis tools. The key difference between FRFT and these two spatial-frequency analysis tools is that Gabor and Wavelet transform are indeed special cases of short time Fourier transform. The window function of Gabor is Gaussian function and the window function of Wavelet transform is adaptive. While, FRFT is a generalized version of Fourier transform.   \n(2) Gabor has the inherent limitation of short time Fourier transform, such as fixed window function and poor time-frequency resolution balance. Besides, Gabor has more hyper-parameters which makes it hard to optimize. Wavelet transform also suffers from the difficulty of selecting the proper wavelet basis function.\n\n\n**5: Speed of the fast implementation version.**   \nWe compare the parameters and Flops of MFRFC and vanilla convolution in Table 2 in the main manuscript. As for the speed, the baseline equipped with previous discrete FRFT is about 20 times slower than the original baseline in average. While, the baseline equipped with our fast implementation is about 5\\% slower than the original baseline in average. Our method can substantially elevate the performance of baseline method with negligible additional computational burden including parameters, Flops and running speed. Beisdes, we also provide the code for FRFT in the supplement."}, {"review_id": "PBtsHmveYu", "reviewer": "Reviewer_6jPA", "summary": "+ The paper provides an implementation framework using deep learning for fractional Fourier transform.", "questions": "- Why there is not fast algo for FRFT?\n\n- Can you tell about the accuracy of Deep FRFT on simulated signals compared to others?\n\nAnswered.", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Good deep algo for FRFT.", "weaknesses": "- Baselines for different applications are a bit dated.\n\n- The paper could have focused on just one application and dealt deeper.\n\n- The writing is not that good, needs to be precise and with enough motivation.\n\nAddressed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1: Baseline methods for different tasks.**   \n(1) Our method is not SOTA-oriented. Instead, the key of this paper is that we unlock the FRFT in deep learning paradigm, solving the biggest challenge for the popularization of FRFT: vague characteristics and missing fast implementation. Fourier transform is pretty useful and vastly employed in deep learning era. As a generalized and improved version of Fourier transform, FRFT surely has great application prospect and exploration value.   \n(2) We validate the effectiveness of FRFT on various tasks with classical and commonly acknowledged baselines. We select the classical methods as baselines for three reasons: 1. We follow the setting of previous basic operator-based methods [1][2], which also employ classical methods as baselines. 2. The selected baselines are the representative and commonly acknowledged works in the related tasks. In addition, as a general operators, implementing the proposed operator in the standard benchmarks is more fair. 3. The performance of classical methods are highly likely to be reproducible.  \n(3) To further solve your misgiving, we also apply our operator to recent SOTA baselines. We choose GPPNN [3] and LACNET [4] for guided image super-resolution task on WorldView-II dataset, NBNet [5] and Restormer [6] for image denoising task on SIDD dataset, URetinex [7] and SNR [8] for low-light image enhancement task on LOL dataset. As can be seen from the following three tables, our method still largely elevates the performance of these SOTA baselines. We will add these experiments and conduct more experiments on remaining tasks in the final version. Besides SOTA baselines, our method may be critical to small networks in the practial deployment,  which shows the irreplaceable and important role of this basic tool and operator.\n\n| Model | Methods |PSNR | SSIM |\n| :-----| ----: | ----: | :----: |\n| GPPNN  | Original | 41.16  | 0.968  |\n|        | SFC      | 40.14  | 0.967  |\n|        | MFRFC    | 41.47  | 0.972  |\n| LACNET | Original | 41.45  | 0.972  |\n|        | SFC      | 41.52  | 0.973  |\n|        | MFRFC    | 41.74  | 0.975  |\n\n| Model | Methods |PSNR | SSIM |\n| :-----| ----: | ----: | :----: |\n| NBNet     | Original | 39.75  | 0.959  |\n|           | SFC      | 39.86  | 0.959  |\n|           | MFRFC    | 39.97  | 0.960  |\n| Restormer | Original | 40.02  | 0.960  |\n|           | SFC      | 40.05  | 0.960  |\n|           | MFRFC    | 40.19  | 0.961  |\n\n\n| Model | Methods | PSNR | SSIM |\n| :-----| ----: | ----: | :----: |\n| URetinex | Original | 21.33  | 0.835  |\n|          | SFC      | 21.82  | 0.837  |\n|          | MFRFC    | 22.35  | 0.839  |\n| SNR      | Original | 24.61  | 0.842  |\n|          | SFC      | 24.84  | 0.846  |\n|          | MFRFC    | 24.98  | 0.848  |\n\n\n[1] Fast Fourier Convolution. NeurIPS, 2020.  \n[2] Deep Fourier Up-sampling. NeurIPS, 2022. \n[3] Deep gradient projection networks for pan-sharpening. CVPR, 2021.     \n[4] LAGConv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening. AAAI, 2022.   \n[5] NBNet: Noise basis learning for image denoising with subspace projection. CVPR, 2021.   \n[6] Restormer: Efficient transformer for high-resolution image restoration. CVPR, 2022.   \n[7] URetinex-Net: Retinex-based Deep Unfolding Network for Low-light Image Enhancement. CVPR, 2022.    \n[8] SNR-Aware Low-light Image Enhancement. CVPR, 2022. \n\n\n**2: The paper could have focused on just one application.**  \n(1) Our paper first and comprehensively introduces the characteristics and properties of FRFT in the deep learning paradigm and designs a simple and general operator. Since FRFT is a basic tool and our operator is general, we naturally need to verify the effectiveness and generality of our method on various tasks and classical baselines. \n(2) Our paper unlocks FRFT in deep learning. With this pioneer work, the closer and deeper combination with FRFT and specific tasks is expected to be easier and has great application prospect.\n\n\n**3: The writing quality.**   \nDo you have specific suggestions for the writing. Reviewer c5M1 highly praises the writing \"The writing is clear and lucid. The experimental settings are clearly described.\" Reviewer HR4i confirms our writing: \"This paper is clearly written and easy to follow.\"\n\n\n**4: Why there is not fast algo for FRFT?**  \n(1) The fast discrete implementation of FRFT is difficult is theory [1][2][3]. Intuitively, the formulation of FRFT shown in equation 1 of the main manuscript is much complicated than that of Fourier transform.   \n(2) There exists no fast implementation in practice. Previous discrete implementation of FRFT is much slower than FFT in practical use. There also exists no official package for FRFT.  \n[1] The fractional order Fourier transform and its application to quantum mechanics. IMA Journal of Applied Mathematics, 1980.  \n[2] Digital computation of the fractional Fourier transform. IEEE Transactions on signal processing, 1996.  \n[3] Two dimensional discrete fractional Fourier transform. Signal Processing, 1998.  \n\n**5: FRFT on simulated signals.**  \nThere may be a misunderstanding to our work. Our method validates the effectiveness of FRFT on various tasks. The processed signals are 2-D image signals, instead of simulated signals. What do you mean by simulated signals.\n\n"}, {"review_id": "gpiw0sFwLt", "reviewer": "Reviewer_HR4i", "summary": "This paper introduces Fractional Fourier Transform to provide comprehensive unified spatial-frequency perspectives for deep learning, and further introduces a basic operator, Multi-order Fractional Fourier Convolution. Besides, this paper experimentally evaluates the effectiveness of MFRFC on various computer vision tasks, including object detection, image classification, guided super-resolution, denoising, dehazing, deraining, and low-light enhancement.", "questions": "I only have several concerns here:\n1. Does there exists a constant optimal fractional order for each task. Besides, the authors exploring the optimal fractional order via learnable parameters. Is there exists other traditional method that determines the optimal fractional?\n2. The non-stationary image signals can be better decoupled with FRFT as shown in Figure 1. Can you show some specific examples.\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "There are several strengths here:\n1. This paper is of high novelty and originality. Fourier transform is very important and is employed by many deep learning-based methods. But Fractional Fourier transform, a generalized version of Fourier transform, is less explored in deep learning era. This paper introduces FRFT into deep learning, analyses the property of FRFT, and designs a basic convolutional operator.\n2. This paper achieves the fast implementation of FRFT, which is very important for the future development of FRFT in the deep learning pipeline.\n3.  This paper validates the effectiveness of the MFRFC operator on various computer vision tasks, including high-level tasks (object detection and image classification) and low-level tasks (guided super-resolution, denoising, dehazing, deraining, and low-light enhancement).\n4. This paper is clearly written and easy to follow.\n", "weaknesses": "Three are several weakness here:\n1. The FRFT itself has a long history and is explored in many research areas. The discussion in the related work part is limited and not comprehensive. For example, the recent work [1] also employs FRFT in image super-resolution. \n2. It is necessary to explain the design of MFRFC operator and why such design is optimal. The MFRFC operator employs three paths. But is it possible to apply more paths in the operator. How is the performance and parameters comparison between different number of paths.\n3. The MFRFC operator integrates multiple paths/domains, which is superior to single path. But, how is the performance of the operator with only fractional branch compared to other single path. Can single fractional path achieves comparable performance to the full MFRFC operator?\n\n[1] Adaptive Image Super-Resolution Algorithm Based on Fractional Fourier Transform.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1: The research history of FRFT in many research areas.**   \nThanks for the suggestion, we will make a comprehensive discussion in the research history of FRFT in many other research areas in the related work. The reference work you mentioned is quite simple and is not deep learning based, so we did not include it in the submission. We will also add it in the related work.\n \n\n**2: The design choice of MFRFC operator.**  \nWe devise the MFRFC operator from two main perspectives.   \n(1) The number of branches. Our work takes FFC [1] , a spatial and frequency two-branch operator, as baseline. To verify the effectiveness of FRFT, we add one more fractional order branch. The two-branch baseline is designed with the same manner as our three-branch MFRFC operator, with the only difference in fractional branch. Such design is enough to demonstrate the superiority of FRFT over Fourier transform in a fair setting.   \n(2) The order of the fractional domain. The order selection of the fractional domain is more important. To this end, we experiment with different fractional order in the supplementary material. Besides, we also conduct experiments with adaptive order. The adaptive order version achieves nearly optimal performance among different fractional orders. Besides, our selection order=0.5 in the main manuscript also gets relatively good performance among different fractional orders.  \n[1] Fast Fourier Convolution. NeurIPS, 2020.\n\n\n**3: The performance of the operator with only fractional branch.**  \nIt is a good question and we explore the performance of the operator with only fractional branch in the supplementary material. We choose two representative tasks, object detection with Faster RCNN as backbone and guided image super-resolution with PanNet as backbone. The experimental results show that single branch performs slightly worse than the integrated operator SFC and MFRFC. For the comparison between different single branches, spectral branch (order=0.5 and order=1) performs slightly better than the spatial branch (which is also the baseline).\n\n**4: About the optimal fractional order.**  \n(1) We hold the belief that the optimal fractional order of an image is to do with the task. In other words, it depends on what we want to disentangle from the image. This means that the optimal fractional order is related with the data and task. Thus, for tasks in deep learning paradigm, a fixed optimal fractional order may not exist, since the data is different and uncertain.   \n(2) Certain previous method [1] employs entropy maximization to determine the optimal fractional order in hyperspectral anomaly detection task. But this method is coupled with this specific task and data format and this method is not deep learning based. In contrast, setting adaptive order with learnable parameters is a more general way in deep learning paradigm.    \n[1] Hyperspectral anomaly detection by fractional Fourier entropy. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.\n\n\n**5: Specific examples.**   \nIt is a good question since specific examples are intuitive for the better understanding of FRFT. We show the decoupling capacity of FRFT for non-stationary signal with a\nprevious work [1]. This work explores decoupling degradation from the degraded image with FRFT. FRFT can well decouple degradation and the clean image, filter the degradation, and return the clean image in this case. This well supports our work with specific examples.   \n[1] Optimal image restoration with the fractional Fourier transform. JOSA A, 1998.\n"}]

Author rebuttal:

Meta-review: All reviewers recommended to accept this paper, with 3 "strong accepts". 
Connecting the bridge between deep learning and Fractional Fourier Transform was very well received by the reviewers, and the work can serve as a foundation for future improvement in deep learning. 
The AC agrees with the reviewers, and recommend to accept this paper.
Decision: Accept (spotlight)