SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qi Zhu  $^{1}$ ; Man Zhou  $^{2,3}$ ; Jie Huang  $^{1}$ , Naishan Zheng  $^{1}$ , Hongzhi Gao  $^{1}$ , Chongyi Li  $^{4}$ , Yuan Xv  $^{3}$ , Feng Zhao  $^{1\dagger}$
<sup>1</sup>University of Science and Technology of China, <sup>2</sup>S-Lab, <sup>3</sup>Nanyang Technological University, <sup>4</sup>Nankai University {zqcrafts,hj0117,nszheng, hongzhigao}@mail.ustc.edu.cn, {man.zhou, xu.yuan}@ntu.edu.sg, lichongyi25@gmail.com, fzhao956@ustc.edu.cn
Spatial down-sampling techniques, such as stripped convolution, Gaussian, and Nearest down-sampling, are essential in deep neural networks. In this paper, we revisit the working mechanism of the spatial down-sampling family and analyze the biased effects caused by the static weighting strategy employed in previous approaches. To overcome the bias limitation, we propose a novel down-sampling paradigm in the Fourier domain, abbreviated as FouriDown, which unifies existing down-sampling techniques. Drawing inspiration from the signal sampling theorem, we parameterize the non-parameter static weighting down-sampling operator as a learnable and context-adaptive operator within a unified Fourier function. Specifically, we organize the corresponding frequency positions of the 2D plane in a physically-closed manner within a single channel dimension. We then perform point-wise channel shuffling based on an indicator that determines whether a channel's signal frequency bin is susceptible to aliasing, ensuring the consistency of the weighting parameter learning. FouriDown, as a general operator, comprises four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform. These components can be easily integrated into existing image restoration networks. To demonstrate the efficacy of FouriDown, we conduct extensive experiments on image de-blurring and low-light image enhancement. The results consistently show that FouriDown can provide significant performance improvements. The code is publicly available to facilitate further exploration and application of FouriDown at https://github.com/zqcrafts/FouriDown.
Down-sampling technique [1, 2, 3] plays a vital role in deep neural networks because of its benefits in enlarging the receptive field, extracting hierarchical features, improving computational efficiency, and handling scale and translation variations. However, based on the signal sampling theorem, existing down-sampling techniques such as stripped convolution, Gaussian, and Nearest down-sampling [1, 4, 5, 6] unavoidably reduce the sampling frequency of discrete signals, leading to unexpected frequency aliasing where high frequencies are folded into low frequencies.
To address the aliasing problem, several strategies [7, 8, 9, 10, 11, 12, 13] have been developed. They pre-process the signals applying the low-pass filtering mechanism, which aims to filter out high-frequency information by employing different types of low-pass designs. There are two commonly
*Both authors contributed equally to this research.
Figure 1: Comparison on the flowcharts of different down-sampling techniques in  $2 \times$  scale. The previous spatial down-sampling community stands on the static weighting templates and is not relevant with the image context. By contrast, inspired by the signal sampling theorem, we parameterize the static weighting down-sampling operator as a learnable and context-adaptive operator in a unified Fourier function.
used types including the ideal low-pass filter that truncates high frequencies in the Fourier domain and the Gaussian low-pass filter that gradually attenuates frequency components near the boundary. However, the ideal low-pass filter may introduce ring artifacts due to spectrum leakage, while the Gaussian low-pass filter may result in a significant loss of edge information that is crucial for visual recognition tasks.
The prevailing approaches in the down-sampling family rely on a static weighting strategy, which may lead to unintended biases (See Section 4.4 for details.). As described in Figure 1, strided convolution and strided pooling variants rely on the static template  $w_{i} = [0.25, 0.25, 0.25, 0.25]$  over the corresponding cornered positions while the ideal low-pass one exploits the  $w_{i} = [0.25, 0, 0, 0]$  weighting template. (See Appendix B for proofs.) All of them are shared over all the coordinated positions and uninvolved with the feature context. It is widely acknowledged that the static sampling approach, which lacks contextual relevance, is sub-optimal for visual tasks. Therefore, both bridging different down-sampling approaches and achieving an optimal approach are desirable, as shown in Figure 1 where we focus on unifying the down-sampling modeling rules in a learnable and context-adaptive parameterized function in Fourier domain.
In this study, we delve into the working mechanism of the spatial down-sampling family and analyze the biased effects resulting from the static weighting strategy used in existing down-sampling approaches. To solve the bias problem, we propose a novel down-sampling paradigm called FouriDown, which operates in the Fourier domain and adapts the feature sampling based on the image context. Inspired by the signal sampling theorem, we parameterize the non-parameter static weighting down-sampling operator as a learnable and context-adaptive operator in a unified Fourier function. Furthermore, drawing from this insight, we organize the corresponding frequency positions of the 2D plane, ensuring that they are physically closed in a single channel dimension. We then perform pointwise channel shuffling based on an indicator that determines whether a channel's signal frequency bin is prone to aliasing, thereby maintaining the consistency of the weighting parameter learning. FouriDown, as a generic operator, comprises four key components: 2D discrete Fourier transform, context shuffling rules, Fourier weighting-adaptively superposing rules, and 2D inverse Fourier transform. These components can be readily integrated into existing image restoration networks, allowing for a plug-and-play approach. To verify its efficacy, we conduct extensive experiments across multiple computer vision tasks, including image de-blurring and low-light image enhancement. The results demonstrate that FouriDown consistently outperforms the baselines, showcasing its capability of performance improvement.
In conclusion, this work proposes a novel and unified framework for the research of down-sampling, which have the following contributions.
1) We provide the first exploration of the aliasing problem in deep neural networks, analyzing it from a spectrum perspective.
2) To achieve dynamic frequency aliasing, we introduce a unified approach to down-sampling strategies within the Fourier function. Additionally, we propose a learnable and context-adaptive down-sampling operator based on the Nyquist signal sampling theorem.
3) Our proposed down-sampling approach serves as a plug-and-play operator, consistently enhancing the performance of image restoration tasks, such as low-light enhancement and image deblurring.
2.1 Traditional Down-Sampling
Downsampling is an important and common operator in computer vision [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25], which benefits from enlarging the receptive field and reducing computational costs. So many models incorporate downsampling to allow the primary reconstruction components conducting at a lower resolution. Moreover, with the emergence of increasingly compute-intensive large models, downsampling becomes especially crucial, particularly for high-resolution input images.
Previous downsampling methods often utilized local spatial neighborhood computations (e.g., Bilinear, Bicubic and MaxPooling), which show decent performances across various tasks. However, these computations are relatively fixed, making it challenging to maintain consistent performance across different tasks. To address this, some methods made specific designs to make the downsampling more efficient in specific tasks. For instance, some works [12, 11, 10, 7]introduce the Gaussian blur kernel before the downsampling convolution to combat aliasing for better shift-invariance in classification tasks. Grabinski et al. [26, 27] equip the ideal low-pass filter or the hamming filter into downsampling to enhance model robustness and avoid overfitting.
2.2 Dynamic Down-Sampling
Due to the development of data-driven deep learning, in addition to traditional down-sampling, some other works [28, 29, 30, 31, 32, 33] introduce dynamic downsampling to adaptively adjust for different tasks, thereby achieving better generalizability. For instance, Pixel-Shuffle [28] enables dynamic spatial neighborhood computation through the interaction between feature channels and spaces, restoring finer details more effectively. Kim et al. [29] proposes a task-aware image downsampling to support upsampling for more efficient restoration.
In addition to dynamic neighborhood computation, dynamic strides have also gained widespread attention in recent years. For instance, Riad et al. [30] posits that the commonly adopted integer stride of 2 for downsampling might not be optimal. Consequently, they introduce learnable strides to explore a better trade-off between computation costs and performances. However, the stride is still spatially uniformly distributed, which might not be the best fit for images with uneven texture density distributions. To address this issue, dynamic non-uniform sampling garners significant attention [31, 32, 33]. For example, Thavamani et al. [31] proposed a saliency-guided non-uniform sampling method aimed at reducing computation while retaining task-relevant image information.
In conclusion, most of recent researches focus on dynamic neighborhood computation or dynamic stride for down-sampling, where the paradigm can be represented as  $\text{Down}(s)$ , where  $s$  denotes the stride. However, in this work, we observe that the methods based on this downsampling paradigm employ static frequency aliasing, which may potentially hinder further development towards effective downsampling. However, learning dynamic frequency aliasing upon the existing paradigm poses challenges. To address this issue, we revisit downsampling from a spectral perspective and propose a novel paradigm for it, denoted as  $\text{FouriDown}(s, w)$ . This paradigm, while retaining the stride parameter, introduces a new parameter,  $w$ , which represents the weight of frequency aliasing during downsampling and is related to strides. Further, based on this framework, we present an elegant and effective approach to achieve downsampling with dynamic frequency aliasing, demonstrating notable performance improvements across multiple tasks and network architectures.
Figure 2: The visualization of the shuffling and superposing theory in the 1-D and 2-D signals.
**Definitions.**  $f(x, y) \in \mathbb{R}^{\mathrm{H} \times \mathrm{W} \times \mathrm{C}}$  is the 2-D discrete spatial signal and the sampling rates in  $x$  and  $y$  axis are  $\Omega_x$  and  $\Omega_y$ , respectively.  $F(u, v) \in \mathbb{R}^{\mathrm{H} \times \mathrm{W} \times \mathrm{C}}$  is the Fourier transform of  $f(x, y)$ , where the maximum frequencies in  $u$  and  $v$  axis are respectively denoted as  $u_{max}$  and  $v_{max}$ . Moreover,  $f'(x, y) \in \mathbb{R}^{\frac{\mathrm{H}}{2} \times \frac{\mathrm{W}}{2} \times \mathrm{C}}$  is 2-strided down-sampled  $f(x, y)$  and its Fourier transform  $F'(u, v)$ .
Theorem-1. Shuffling and Superposing. The spatial down-sampling typically results in a shrinkage of the tolerance for the maximum frequency of the signal. Specifically, high frequencies will fold back into new low frequencies and superpose onto the original low frequencies. To illustrate with 1-dimensional signal, the high and low frequency superposition in the down-sampling can be formulated as
$$
F ^ {\prime} (u) = \mathbb {S} \left(F (u), F \left(u + \frac {\Omega_ {x}}{2}\right)\right) \quad w h e n \quad u \in \left(0, \frac {\Omega_ {x}}{2}\right), \tag {1}
$$
where  $\mathbb{S}$  is a superposing operator. Note that the high frequency is  $F(u + \frac{\Omega_x}{2})$  considering positive directions, while the low frequency is  $F(u)$  considering positive directions instead.
Theorem-2. Static Averaging Superposing. For an image, the spatial down-sampling operator with 2 strides can be equivalent to dividing the Fourier spectrum into  $2 \times 2$  equal parts and superposing them averagely by  $\frac{1}{4}$  factor
$$
F (u, v) = \left[ - \frac {F _ {(0 , 0)} (u , v)}{F _ {(1 , 0)} (u , v)} + \frac {F _ {(0 , 1)} (u , v)}{F _ {(1 , 1)} (u , v)} - \right], \tag {2}
$$
where  $F_{(i,j)}(u,v)$  is a sub-matrix of  $F(u,v)$  by equally dividing  $F(u,v)$  into  $2 \times 2$  partitions and  $i, j \in \{0,1\}$ . Given that  $\mathrm{Down}_2$  is 2-strided down-sampling operator and IDFT is inverse discrete Fourier transform, we have
$$
\operatorname {D o w n} _ {2} (f (x, y)) = \operatorname {I D F T} \left(\frac {1}{4} \sum_ {i = 0} ^ {1} \sum_ {j = 0} ^ {1} F _ {(i, j)} (u, v)\right). \tag {3}
$$
The proof of the above theorem can be found in the Appendix, and examples of 1-D and 2-D signals can also be referred in Figure 2(a) and (b).
Figure 3: Overview of the proposed spectral shuffle.
Algorithm 1 Pseudo-code of FouriDown.
Data: Input:  $x\in \mathbb{R}^{N\times C\times H\times W}$  . Set of real numbers:  $\mathbb{R}$  . Set of complex numbers:  $\mathbb{C}$    
Result:  $y\in \mathbb{R}^{N\times C\times \frac{H}{2}\times \frac{W}{2}}$ $x_{fft}\gets FFT(x)\in \mathbb{C}^{N\times C\times H\times W}$  // Fast Fourier transform   
 $x_{shuffle}\gets Shuffle(x_{fft})\in \mathbb{C}^{N\times 4\times C\times \frac{H}{2}\times \frac{W}{2}}$ $x_{weight}\gets Softmax(Convs_{1x1}(x_{shuffle},groups = C))\in \mathbb{C}^{N\times 4\times C\times \frac{H}{2}\times \frac{W}{2}}$ $x_{superposed}\gets Sum(x_{weight}*x_{shuffle},dim = 1)\in \mathbb{C}^{N\times C\times \frac{H}{2}\times \frac{W}{2}}$ $y\gets iFFT(x_{superposed})//$  Inverse Fourier transform
3.1 Architecture Design
In this work, we argue that the static superposing strategy like the stride-based down-sampling in Theorem-2 might lead to biased effects. Motivated by adaptively learning ability of CNNs, we aim to parameterize the non-parameter static superposing step as a learnable and context-adaptive operator in the Fourier domain.
Definition-2 (Shuffle-Invariance) Given an operator  $z(\cdot)$  that is shuffle-invariant and  $o_1, o_2, o_3, o_4$  as different components, the shuffle-invariant is defined as  $z(o_1, o_2, o_3, o_4) = z(shuffle(o_1, o_2, o_3, o_4))$ , where  $shuffle(\cdot)$  is shuffling the order of input components arbitrarily.
Note that the average operator in Theorem-2 is shuffle-invariant. For example,  $Aver(a,b,c,d) = Aver(b,a,c,d)$ . However, different from averaging, the convolution operator, which is sensitive to the input order, does not have this property.
To alleviate this problem, we design a spectral-shuffle strategy that first performs shuffling according to Theorem-1 and then aligns across different frequency bands, as shown in Figure 3. Specifically, we initially spilt the original spectrum  $F(u,v)$  into 16 patches equally. Then, according to Theorem-1, we classify these patches into 4 groups, where each group is pixel-wise matched frequency bin for superposing. However, the energy distribution in each group is different. Considering the shuffling variance of convolution operators, we reorder the intra-group sequence for inter-group alignment. The alignment is motivated by wavelet theory, where intra-group frequencies are reordered according to low-frequency and high-frequency in horizontal direction, high-frequency in vertical direction, and high-frequency in diagonal direction. Then, the aligned groups are sorted orderly on channels for pixel-wise matching in the channel dimension. Finally, we perform adaptively weighted superposition on channels by learned weights for the down-sampling results. The main implementation is depicted in Algorithm 1. Code will be public.
4 Experiments and Discussion
To validate the efficacy of our proposed FouriDown, we execute comprehensive experiments across several computer vision tasks and conduct exhaustive ablation studies to evaluate its resilience.
Image enhancement. For image enhancement, we assess our FourierDown model using the LOL [34] and Huawei [35] benchmarks. The LOL dataset contains 500 image pairs (485 for training, 15 for testing), and the Huawei dataset contains 2480 paired images (2200 for training, 280 for testing). We compare our results with two established baselines, SID [36] and DRBN [37].
Image deblurring. For image deblurring, we utilize DeepDeblur [38] and MPRNet [39] on the DVD dataset [40], which includes 2103 training and 1111 test pairs. We further validate our model's generalizability using the HIDE dataset [41].
Image denoising. In the context of image de-noising, our training involves the SIDD dataset [42]. Subsequent performance assessments are carried out on the remaining validation samples from the SIDD dataset and on the DND benchmark dataset [43]. For comparative analysis, we choose baselines such as MIRNet [44], and Restormer [45].
Image dehazing. For image dehazing, we employ RESIDE dataset for evaluations. We also use two different network designs MSBDN [46] and GridNet [47] with our proposed operator for validation.
Regarding the above competitive baselines, we perform the comparison over the following configurations by replacing the down-sampling operator, such as strided convolution and strided pooling), with the proposed FouriDown operator. Additionally, we also perform comparisons with previous anti-aliasing down-sampling methods, including Gaussian filter [7] and "ideal" Low-Pass Filter (LPF) [13], which conduct the static modulation on the spectrum.
1) Original: The original down-sampling in the baseline;
2) Gaussian: Following [7], equipping the Gaussian filter in all channels before the original down-sampling for anti-aliasing;
3) LPF: Following [13], equipping the "ideal" Low-Pass Filter in all channels before the original down-sampling operator for anti-aliasing;
4) Ours: Replacing our proposed FouriDown with the original down-sampling operator;
Table 1: Image enhancement comparison.
Method	Config	LOL	Huawei
PSNR	SSIM	PSNR	SSIM
DRBN	Original	19.92	0.7712	20.21	0.6742
Gaussian	20.21	0.8146	20.66	0.6955
LPF	18.91	0.7441	20.34	0.6812
Ours	21.64	0.8513	21.46	0.7213
SID	Original	21.46	0.8584	20.38	0.6931
Gaussian	21.78	0.8612	20.52	0.6926
LPF	20.74	0.8124	20.54	0.6841
Ours	23.28	0.8708	20.90	0.7002
Table 3: Image de-noising comparison.
Method	Config	SIDD	DND
PSNR	SSIM	PSNR	SSIM
Restormer	Original	39.41	0.9171	39.67	0.9173
Gaussian	39.43	0.9169	39.69	0.9177
LPF	39.35	0.9162	39.64	0.9167
Ours	39.47	0.9174	39.73	0.9180
MIRNet	Original	39.52	0.9182	39.41	0.9146
Gaussian	39.55	0.9184	39.45	0.9148
LPF	39.49	0.9179	39.35	0.9141
Ours	39.64	0.9186	39.56	0.9251
Table 2: Image deblurring comparison.
Method	Config	DVD	HIDE
PSNR	SSIM	PSNR	SSIM
DeepDeblur	Original	29.32	0.8817	29.60	0.8849
Gaussian	29.36	0.8823	29.62	0.8892
LPF	29.19	0.8751	29.51	0.8851
Ours	29.44	0.8856	29.70	0.8904
MPRNet	Original	30.12	0.8958	30.04	0.8945
Gaussian	30.23	0.8922	30.06	0.8966
LPF	30.00	0.8918	29.95	0.8937
Ours	30.31	0.8996	30.25	0.9102
Table 4: Image dehazing comparison.
Method	Config	Indoor	Outdoor
PSNR	SSIM	PSNR	SSIM
MSBDN	Original	29.77	0.9591	28.88	0.9581
Gaussian	30.09	0.9607	28.91	0.9583
LPF	29.92	0.9598	29.03	0.9591
Ours	30.19	0.9612	29.21	0.9604
GridNet	Original	30.16	0.9616	29.54	0.9605
Gaussian	30.21	0.9617	29.62	0.9622
LPF	30.18	0.9611	29.58	0.9615
Ours	30.42	0.9654	29.71	0.9641
(a) Low-light image
(b) SID
(c) SID with FouriDown
(d) Groud truth
Figure 4: Visual comparison of SID on the LOL dataset. FouriDown enhances the global color perception ability of the original model, thereby improving the model's performance without adding extra parameters or computational overhead.
(a) Low-light image
Figure 5: Visual comparison of DRBN on the LOL dataset. The more flexible frequency interaction mechanism in FouriDown reduces artifacts compared to the original methods.
(b) DRBN
(c) DRBN with FouriDown
4.3 Comparison and Analysis
Quantitative Comparison. To demonstrate the effectiveness of our proposed FouriDown, we conduct extensive experiments as shown in Tables 1-4. The best results are in bold. Above and below the baseline are highlighted in red and blue, respectively. From these tables, although previous anti-aliasing methods may be useful for some image restoration tasks, their static weights limit their universality in other tasks. For instance, while the LPF approach performs well in dehazing, it fails to deliver effective in deblurring and low-light enhancement. In contrast, our method is proved to be effective across the majority of image restoration tasks. Specifically, we achieved an improvement of 1.82dB in low-light enhancement and 0.42dB in dehazing on LOL and Reside dataset respectively.
Further, we compare the computing costs with other methods shown in Table 5. We include results from traditional down-sampling techniques like bicubic, bilinear, pixel-unshuffle,  $2 \times 2$  learnable CNN (with stride=2), max-pooling, average-pooling, LPF, Gaussian and Ours. Noting that the "Original" down-sampling of the method is pointed by asterisk (\*). This will allow a clearer contrast and showcase the advantages of our method not only against anti-aliasing approaches but also against these conventional down-sampling methods.
Qualitative Comparison. Due to space constraints, we only present a qualitative comparison on the low-light enhancement task. As illustrated in Figure 4 and Figure 5, our FouriDown reduces original artifacts presented in the SID due to the more flexible frequency interactions. Then, we compare the visualizations of the feature maps and their corresponding spectra between FouriDown and other down-sampling methods (see Figure 6 and Figure 7). It can be observed that the model equipped with FouriDown generates much stronger responses to degradation-aware regions, i.e. global low-illumination in the low-light enhancement task. In contrast, the model with other down-sampling method responds weakly to these regions. The results demonstrate the effectiveness of FouriDown in capturing degradation-aware information by adaptive frequency superposition in down-sampling. For the Gaussian method, its response to degradation is relatively large (second only to FouriDown), thus achieving performance that is also second only to FouriDown. Similarly, as the LFP method has the poorest performance, its feature response of the low-light areas is also the lowest. The performance of other methods is roughly similar, so their feature responses are also quite similar, indicating a similar capability to capture image degradation areas. Additionally, from the spectral comparison in Figure 7, it can be observed that the Gaussian method loses a lot of high-frequency information compared to FouriDown. This leads to challenges in recovering textures in dark areas. Hence, although the Gaussian method exhibits good responses, FouriDown achieves better performances compared to it. More qualitative comparisons can be found in the following supplementary material.
(b) AverPooling
(c) Bicubic
(d) Bilinear
(e) Gaussian
(j) Ours
(f) LPF
(g) MaxPooling
(h) PixelShuffle
(i)StrideConv
Figure 6: Feature comparison between our FouriDown and other down-sampling methods in low-light enhancement task. Due to the unique global modeling mechanism in the frequency domain, the features extracted by our method significantly achieve a larger response than others.
(a) Input
(f) LPF
Figure 7: Spectrum comparison of the feature maps in Figure 6. The spectrum following FouriDown obtains the outstanding smooth response in both high and low frequencies.
4.4 Discussions
Bias Effects by Static Superposing. As shown in Figure 8, we compare different down-sampling methods with static superposing manner, and find they have various bias effects.