SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Frequency Aliasing Visualization. To delve deeper into the high-low frequency interactions in down-sampling, we examine the spectrum of images down-sampled by factors of  $4\mathrm{x}$ ,  $2\mathrm{x}$ , and  $1\mathrm{x}$ . Following Theorem 1, some regions of spectrums are overlaid on the same frequency band, with smaller scales overlaying larger ones, as shown in Figure 9. This alignment of same bandwidth reveals a rectangular contour at the intersections, where high-frequencies not obeying the Nyquist theory fold into low frequencies during down-sampling, as pointed by the yellow arrow. This suggests that it is significant for down-sampling to modulate frequencies, otherwise it might degrade the original signal undesirably.
Other Discussions. Because of space constraints, for more discussions, including extensions to Theorem-2 and revisiting of previous anti-aliasing methods in the proposed FouriDown framework, could be referred to the supplementary material.
In this work, we explore spatial down-sampling from a frequency-domain perspective and optimize the static weighting of previous down-sampling with a stride of 2 in the frequency domain. Our modeling of down-sampling is based on using uniformly distributed impulse sequences as the sampling function, hence exploring the characteristics of the sampling function in the frequency domain. However, for non-uniform down-sampling, where the sampling rate varies according to the content, our method might become limited. We hope to overcome this limitation in the future work by exploring the frequency domain response of non-uniform sampling functions.
(b) Nearest-2×
(c) Ideal low-pass  $-2\times$
(d) Gaussian-2X
Config	LOL	FLOPs(G)	Para(M)
PSNR	SSIM
Bicubic	21.35	0.8497	13.764	7.84
Bilinear	21.26	0.8464	13.764	7.84
Pixle-shuffle	21.41	0.8552	13.954	8.11
Stride Conv	21.36	0.8534	13.954	8.11
Max pooling *	21.46	0.8584	13.753	7.84
Average pooling	21.34	0.8481	13.754	7.84
Gaussian	21.79	0.8612	16.102	8.54
LPF	20.74	0.8124	16.102	8.54
Ours	23.28	0.8708	13.827	7.87
Table 5: The effectiveness and efficiency comparison between our FouriDown and other down-sampling methods in low-light enhancement task.
Figure 8: The comparison of results by different down-sampling manner in  $2x$  scale. In (b), which is the nearest down-sampling of (a), some unnatural stripes in the edge of windows and walls, called aliasing effects. To relieve the occurrence of aliasing, (c) and (d) employ the "ideal low-pass filter" and the Gaussian filter before down-sampling respectively. However, these manners usually lead to ringing effects or heavy high-frequency loss inevitably.
Figure 9: The spectrums of different-scale images placed under the same frequency coordinate system.
In our study, we revisit the spatial down-sampling techniques and anti-aliasing strategies from a Fourier domain perspective, recognizing their reliance on static high and low frequency superposing. As a result, we propose a novel approach (FouriDown) to learn a learnable frequency-context interplay among high and low frequencies during down-sampling. Moreover, this work is the first exploration of dynamic frequency interaction in down-sampling. The FouriDown is designed based on the signal sampling theory, so it is convenient to replace most of current down-sampling and anti-aliasing techniques. Extensive experiments demonstrate the performance improvements across a variety of vision tasks.
Ultimately, we believe that down-sampling is a crucial research direction in the future. It allows for network design at a lower resolution, significantly reducing the computational overhead and enabling light-weighting models.
This work further exploits the potential of down-sampling in the Fourier domain and offers a new perspective that frequency band shuffling and superposing for future research of down-sampling. Down-sampling techniques can potentially make the future model development more efficient and effective, beneficial for various machine learning and AI applications. Nonetheless, the efficacy of our method could be a source of concern if not properly utilized, especially in terms of the safety of real-world applications. To alleviate such concerns, we plan to rigorously investigate the robustness and effectiveness of our approach across a more diverse spectrum of real-world scenarios.
Appendix A: Proofs of Theorem-1 and Theorem-2
Proof of Theorem-1: Shuffling and Superposing
To model the relationship between  $f'(x)$  and  $f(x)$ , we stand on their derived continuous signal  $g(x)$  by a specific sampling function. Note that the sampling functions  $s_{\Delta T}(x)$  is defined as the sum of infinitely impulses units separated by  $\Delta T$  intervals:
$$
s (x, \Delta T) = \sum_ {n = - \infty} ^ {\infty} \delta (x - n \Delta T). \tag {4}
$$
Based on the Eq. (4),  $f(x)$  and its down-sampling  $f'(x)$  can be represented as
$$
f (x) = g (x) s (x, \Delta T), \quad f ^ {\prime} (x) = g (x) s (x, 2 \Delta T). \tag {5}
$$
According to the Fourier transform and convolution theorem, the aforementioned spatial sampling can be expressed in the Fourier domain as:
$$
\begin{array}{l} F (u) = G (u) \star S (u, \Delta T) = \int_ {- \infty} ^ {\infty} G (\tau) S (u - \tau , \Delta T) d \tau \\ = \frac {1}{\Delta T} \sum_ {n} ^ {\infty} \int_ {- \infty} ^ {\infty} G (\tau) \delta \left(u - \tau - \frac {n}{\Delta T}\right) d \tau = \frac {1}{\Delta T} \sum_ {n} ^ {\infty} G \left(u - \frac {n}{\Delta T}\right), \tag {6} \\ \end{array}
$$
where  $G(u)$  and  $S(u, \Delta T)$  are the Fourier transform of  $g(x)$  and  $s(x, \Delta T)$ . From Eq. (6), it can be observed that the spatial sampling introduces the periodicity to the spectrum and the period is  $\frac{1}{\Delta T}$ .
Note that the sampling rates of  $f(x)$  and  $f'(x)$  are  $\Omega_x$  and  $\Omega_x'$ , the relationship between them can be written as
$$
\Omega_ {x} = \frac {1}{\Delta T}, \quad \Omega_ {x} ^ {\prime} = \frac {1}{2 \Delta T} = \frac {1}{2} \Omega_ {x}. \tag {7}
$$
Before down-sampling, to focus on the following down-sampling operation, we assume that  $f(x)$  adheres to the Nyquist sampling theorem, which implies that  $u_{max} > \frac{1}{\Omega_x}$ .
After down-sampling, according to Nyquist sampling theorem, a whole sub-frequency band is limited in  $(0,\frac{\Omega_{\pi}}{2})$ . Moreover, the resulted band is the superposition of two original bands, which denoted as
$$
F ^ {\prime} (u) = \mathbb {S} (F (u), F (\hat {u})), \tag {8}
$$
where  $\hat{u}$ ,  $u$  are the high frequency above the sampling rate and the low frequency below the sampling rate, respectively, and  $\mathbb{S}$  is superposition operator.
(1) In the positive sub-band, where  $u \in (0, \frac{\Omega_x}{4})$ ,  $\hat{u}$  and  $\tilde{u}$  should satisfy
$$
u \in (0, \frac {\Omega_ {x}}{4}) \quad a n d \quad \hat {u} \in (\frac {\Omega_ {x}}{4}, u _ {\max }) \tag {9}
$$
According to the aliasing theorem, the high frequency  $\hat{u}$  is folded back to the low frequency:
$$
\tilde {u} = \left| \hat {u} - \frac {(k + 1) \Omega_ {x} ^ {\prime}}{2} \right|, \quad \frac {k \Omega_ {x} ^ {\prime}}{2} \leq \hat {u} \leq \frac {(k + 2) \Omega_ {x} ^ {\prime}}{2} \tag {10}
$$
where  $k = 1,3,5\ldots$  and  $\tilde{u}$  is folded results by  $\hat{u}$ .
According to Eq. (9) and Eq. (10), we have
$$
\tilde {u} = \frac {\Omega_ {x}}{2} - \hat {u}, \quad a n d \quad \tilde {u} \in \left(\frac {\Omega_ {x}}{2} - u _ {\max }, \frac {\Omega_ {x}}{4}\right). \tag {11}
$$
Then, according to Eq. (8) and Eq. (11), we attain
$$
F ^ {\prime} (u) = \left\{ \begin{array}{l l} F (u) & \text {i f} u \in \left(0, \frac {\Omega_ {x}}{2} - u _ {\max }\right), \\ \mathbb {S} \left(F (u), F \left(\frac {\Omega_ {x}}{2} - u\right)\right) & \text {i f} u \in \left(\frac {\Omega_ {x}}{2} - u _ {\max }, \frac {\Omega_ {x}}{4}\right). \end{array} \right. \tag {12}
$$
According to Eq. (6),  $F(u)$  is symmetric with respect to  $u = \frac{\Omega_x}{2}$ ,
$$
F \left(\frac {\Omega_ {x}}{2} - u\right) = F \left(u + \frac {\Omega_ {x}}{2}\right). \tag {13}
$$
Further,  $F\left(u + \frac{\Omega_x}{2}\right) = 0$  when  $u \in (0, \frac{\Omega_x}{2} - u_{max})$ . Upon Eq. (13), we can uniform Eq. (12) as
$$
F ^ {\prime} (u) = \mathbb {S} (F (u), F (u + \frac {\Omega_ {x}}{2})) \quad \text {w h e n} \quad u \in (0, \frac {\Omega_ {x}}{4}) \tag {14}
$$
The visualization of the aforementioned proof process is depicted in Figure 2(a).
(2) In the negative sub-band, where  $u \in \left(\frac{\Omega_x}{4}, \frac{\Omega_x}{2}\right)$ , different from (1),  $\hat{u}$  and  $\tilde{u}$  should satisfy
$$
u \in \left(\frac {\Omega_ {x}}{4}, \frac {\Omega_ {x}}{2}\right) \quad a n d \quad \hat {u} \in \left(\frac {\Omega_ {x}}{2} - u _ {\max }, \frac {\Omega_ {x}}{4}\right). \tag {15}
$$
Similarly, we can proof  $F(u)$  in the negative sub-band as well.
$$
F ^ {\prime} (u) = \mathbb {S} (F (u), F \left(u + \frac {\Omega_ {x}}{2}\right)) \quad w h e n \quad u \in \left(\frac {\Omega_ {x}}{4}, \frac {\Omega_ {x}}{2}\right) \tag {16}
$$
Combined with Eq. (14) and Eq. (16), we obtain
$$
F ^ {\prime} (u) = \mathbb {S} (F (u), F \left(u + \frac {\Omega_ {x}}{2}\right)), \quad w h e n \quad u \in \left(0, \frac {\Omega_ {x}}{2}\right). \tag {17}
$$
Proof of Theorem-2: Static Superposing.
According to Eq. (7) and Eq. (6), we can deduce that the amplitude of  $F'$  is half that of  $F$ .
Therefore, we can write  $F^{\prime}(u)$  in the  $x$  axis as,
$$
F ^ {\prime} (u) = \frac {1}{2} F (u) + \frac {1}{2} F \left(u + \frac {\Omega_ {x}}{2}\right), \quad \text {w h e n} \quad u \in \left(0, \frac {\Omega_ {x}}{2}\right). \tag {18}
$$
Upon the dual principle, we can prove  $F'(v)$  in the whole sub-band
$$
F ^ {\prime} (u, v) = F ^ {\prime} \left(F ^ {\prime} (u, y), v\right) = \frac {1}{4} \left(F (u, v) + F \left(u + \frac {\Omega_ {x}}{2}, v\right) + F (u, v + \frac {\Omega_ {y}}{2}) + F \left(u + \frac {\Omega_ {x}}{2}, v + \frac {\Omega_ {y}}{2}\right)\right), \tag {19}
$$
where  $u\in (0,\frac{\Omega_x}{2}),v\in (0,\frac{\Omega_y}{2})$
Appendix B: Revisiting previous down-sampling manners in the FouriDown
In the main body, we introduce the FouriDown framework which can simultaneously adjust the stride of down-sampling and the characteristics of frequency interaction. Moreover, in the manuscript, we prove that the operator with a stride of 2 can be simplified to a fixed frequency weighting, i.e. averaging. However, this kind of weighting brings aliasing, hence the need for anti-aliasing methods such as ideal low-pass filters and Gaussian filters, which put focused weighting on frequencies. In this section, we revisit these anti-aliasing methods within the FouriDown framework and discover that their weighting methods remain fixed. In other words, within our framework, it's possible to realize the aforementioned anti-aliasing methods by simply changing specific parameters as shown in Figure 10 and 11.
Appendix C: Stride Extension of Theorem-1 and Theorem-2
In the manuscript, we propose FouriDown which primarily focuses on analyzing frequency interactions in the case of stride  $= 2$ . In this section, we extend the Theorem-2 to arbitrary integer strides  $s$ , resulting in the following theorem. The proof for these theorems follows similar logic to that detailed in the main body, and is therefore omitted for brevity.
Figure 10: The achievements of different down-sampling manners in the FouriDown by simply changing specific parameters.
Figure 11: Revisiting different down-sampling methods from the perspective of spectrum.
Stride Extension of Theorem-1. In the condition of  $s$  strides, to illustrate with 1-dimensional signal, the high and low frequency superposition in the down-sampling can be formulated as
$$
F ^ {\prime} (u) = \mathbb {S} \left(F (u), F \left(u + \frac {k _ {i} \Omega_ {x}}{s}\right)\right) \quad \text {w h e n} \quad u \in \left(0, \frac {\Omega_ {x}}{2}\right), \tag {20}
$$
where  $\mathbb{S}$  is a superposing operator and  $k_{i} = 1,\dots,s - 1$
Stride Extension of Theorem-2. For an image, the spatial down-sampling operator with  $s$  strides can be equivalent to dividing the Fourier spectrum into  $s \times s$  equal parts and superposing them averagely by  $\frac{1}{s^2}$  factor
Given that  $\mathrm{Down}_{\mathrm{s}}$  is  $s$ -strided down-sampling operator and IDFT is inverse discrete Fourier transform, we have
Downsampling ( Strides  $= 2$  )
Downsampling ( Strides  $= \mathbf{s}$ )
Figure 12: Illustration of theorem extension from  $\text{strides} = 2$  to  $\text{strides} = s$ . Due to visualization space limitation,  $s$  is taken as 3 in the figure. The background region has no meaning, just to show more clearly.
$$
\operatorname {D o w n} _ {\mathrm {s}} (f (x, y)) = \operatorname {I D F T} \left(\frac {1}{s ^ {2}} \sum_ {i = 0} ^ {s - 1} \sum_ {j = 0} ^ {s - 1} F _ {(i, j)} (u, v)\right). \tag {21}
$$
Based on above extension, the down-sampling of 3 strides could be presented as in Figure 12.
Appendix D: More Qualitative comparison.
Due to the limited space, we only report the visual results of the low-light enhancement task in main manuscript. We report more visual results in the supplementary materials. As shown, integrating the FouriDown with the original baseline [36, 37, 48, 49, 39, 45, 44, 46, 47] achieves more visually pleasing results as shown in Figure 13 14 15 16 17.
Figure 13: Visual comparison on the low-light enhancement task.
(c) SID with FouriDown
(d) Groud truth
(a) Hazy image
(b) MSBDN
(c) MSBDN with FouriDown
(a) Hazy image
Figure 14: Visual comparison on the dehazing task.
(b) GridNet
(c) GridNet with FouriDown
(a) Input image
(b) HDRNet
(c) HDRNet with FouriDown
(b) UPE
(c) UPE with FouriDown
Figure 15: Visual comparison on the UHD-enhancement task.
(a) Noisy image
(b) Restormer
(c) Restormer with FouriDown
(a) Noisy image
Figure 16: Visual comparison on the denoising task.
(b) MIRNet
(c) MIRNet with FouriDown
(a) Blurry image
(b) DeepDeblur
(c) DeepDeblur with FouriDown
(b) MPRNet
(c) MPRNet with FouriDown
(d) Groud truth
Figure 17: Visual comparison on the deblurring task.