SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baseline variant. Besides our MFRFC operator, we also validate a baseline variant called Spatial-Frequency Convolution (SFC). Specifically, SFC has two order paths, a spatial path (order=0) and a frequency path (order=0). Our MFRFC has an additional fractional path than SFC and is supposed to perform better than SFC.
Three configurations. We re-implement the baseline methods following the settings in the corresponding paper. In this paper, we denote baseline models as "Original", baseline models equipped with SFC as "SFC", and baseline models provided with MFRFC as "MFRFC". We present an example usage of SFC and MFRFC in the supplementary material.
5.3 Analysis and visualization
Quantitative Comparison. We compare model performance over different configurations, as described in implementation details. The quantitative results are reported in Table 3, 4, 5, 6, 7, 8, and 9. It is evident that, by integrating with our proposed MFRFC operator, all "MFRFC" obtain performance gain over the "Original" and "SFC" baselines on all the datasets in the evaluated task, which fully validates the effectiveness of our method.
Qualitative Comparison. Due to the page limits  $^{3}$ , we only show the visual results over the representative tasks of guided image super-resolution, image deraining, and image dehazing in Fig. 4, 5, and 6, respectively, to better illustrate the effectiveness of MFRFC. It can be easily figured out
3 More visual comparison results are reported in the supplementary materials.
that "MFRFC" generates the highest-fidelity results that also look perceptually close to the reference ground-truths. For example, zooming in the red box region in Figure 4, the "MFRFC" model can better recover the texture details. Similar conclusions also apply to Fig. 6.
Table 3: Quantitative comparison over object detection.
Model	Methods	AP50	mAP
Faster RCNN	Original	77.30	77.26
SFC	77.90	77.89
MFRFC	78.00	77.96
Table 5: Quantitative comparison on image denoising.
Model	Methods	PSNR	SSIM
DnCNN	Original	37.1992	0.8904
SFC	37.3454	0.8866
MFRFC	37.6084	0.8915
Table 4: Quantitative comparison on image classification.
Model	Methods	Acc
ResNet	Original	95.23
SFC	95.34
MFRFC	95.43
Table 6: Quantitative comparison on image deraining.
Model	Methods	PSNR	SSIM
PReLU	Original	29.041	0.891
SFC	29.498	0.901
MFRFC	29.735	0.906
Table 7: Comparison of quantitative results over guided image super-resolution.
Model	Methods	WorldView-II	GaoFen2
PSNR↑	SSIM↑	SAM↓	ERGAS↓	PSNR↑	SSIM↑	SAM↓	EGAS↓
PanNet	Original	40.817	0.9626	0.0257	1.0557	43.0659	0.9685	0.0178	0.8577
SFC	41.2118	0.9663	0.0236	0.9901	44.4965	0.9801	0.0161	0.7761
MFRFC	41.3428	0.9667	0.0233	0.9898	44.6245	0.9804	0.0146	0.6551
FusionNet	Original	41.0708	0.9651	0.0248	0.9978	44.8852	0.9810	0.0140	0.6661
SFC	41.3941	0.9674	0.0232	0.9898	45.1071	0.9810	0.0138	0.6413
MFRFC	41.5829	0.9676	0.0229	0.9766	45.2534	0.9876	0.0137	0.6377
Table 8: Quantitative comparison over image dehazing.
Model	Methods	PSNR	SSIM
GridNet	Original	32.56	0.9847
SFC	32.97	0.9856
MFRFC	33.39	0.9887
MSBDN	Original	32.19	0.9813
SFC	32.26	0.9825
MFRFC	32.45	0.9838
Table 9: Quantitative comparison on low-light image enhancement.
Model	Methods	PSNR	SSIM
SID	Original	20.1062	0.7895
SFC	21.9222	0.7988
MFRFC	22.3666	0.7998
DRBN	Original	19.7931	0.8361
SFC	21.8372	0.8356
MFRFC	21.9916	0.8426
(b) Original
(c) SFC
Figure 4: The visual comparison of PANNET over WorldView-II. The bottom row is error map.
(d) MFRFC
(e) GT
(a) Input_rainy
(c) SFC
Figure 5: The visual comparison of PReNet over the Rain200H.
(a) Input_hazy
(e) GT
Figure 6: The visual comparison of GridNet over SOTS.
6 Limitations and Discussions
We evaluate the effectiveness of the proposed method over several computer vision tasks, and we will conduct more extensive experiments on other computer vision tasks (e.g., image segmentation and video processing). Furthermore, we recognize that the optimal fractional order may vary depending on the specific task and may be critical for the further understanding and application of FRFT in specific task. To emphasize, the critical contribution of this work is that we clear the way to the popularization of FRFT, and make the first attempt to integrate FRFT into a basic operator.
In this paper, we introduce a new spatial-frequency analysis tool, FRFT, to generalize existing FT and provide comprehensive perspectives for vision tasks. The FRFT is a unified spatial-frequency transform, simultaneously reflecting an image's spatial and frequency representations. Notably, we explore the properties of FRFT for image processing and achieve the fast implementation of 2D FRFT. Besides, we devise the convolutional operator MFRFC, which has the impressive merits of processing an image from more perspectives in the spatial-frequency plane. The MFRFC is a general convolutional operator and thus can be easily integrated into existing methods to improve their performance further. Extensive experiments have demonstrated the effectiveness of our method.
The past decades have witnessed the development of computer vision driven by convolution neural networks. However, vanilla spatial convolution is only one common perspective of signal processing. Some spatial-frequency combination methods have already exhibited the potential of two-perspective signal processing. As a general version of FT, FRFT is quite promising for image processing but is less explored largely due to the missing fast discrete implementation algorithm and unclear properties in image processing. In this paper, we have cleared all these obstacles to the popularization of
FRFT and made the first attempt to integrate FRFT into a novel convolutional operator, which has demonstrated its effectiveness in various computer vision tasks. Besides, we hold the belief that there is much more space for the further exploration of FRFT in deep learning. Additionally, our work may inspire the rethinking of neural networks from more diverse perspectives.
This work was supported by the JKW Research Funds under Grant 20-163-14-LZ-001-004-01, and the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
[37] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3291-3300, 2018.
[38] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3063-3072, 2020.
[39] Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. Contrastive learning for compact single image dehazing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10551-10560, 2021.
[40] Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng, Wenjun Zeng, and Zhangyang Wang. Benchmarking single-image dehazing and beyond. IEEE Transactions on Image Processing, 28(1):492-505, 2019.
[41] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-based multiscale network for image dehazing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7314-7323, 2019.
[42] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2154-2164, 2020.
[43] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[44] Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. Effective pan-sharpening with transformer and invertible neural network. IEEE Transactions on Geoscience and Remote Sensing, 60:1-15, 2022.
[45] Keyu Yan, Man Zhou, Liu Liu, Chengjun Xie, and Danfeng Hong. When pansharpening meets graph convolution network and knowledge distillation. IEEE Transactions on Geoscience and Remote Sensing, 60:1-15, 2022.
[46] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. Pannet: A deep network architecture for pan-sharpening. In IEEE International Conference on Computer Vision, pages 5449-5457, 2017.
[47] Liang-Jian Deng, Gemine Vivone, Cheng Jin, and Jocelyn Chanussot. Detail injection-based deep convolutional neural networks for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 59(8):6995-7010, 2020.
[48] Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. In JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop, 1992.
[49] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce. Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data fusion contest. IEEE Transactions on Geoscience and Remote Sensing, 45(10):3012-3021, 2007.