Deep Fourier Up-Sampling
Authors: man zhou
Hu Yu
Jie Huang
Feng Zhao
Jinwei Gu
Chen Change Loy
Deyu Meng
Chongyi Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Image restoration
image de-raining
image de-hazing
image super-resolution

Abstract: Existing convolutional neural networks widely adopt spatial down-/up-sampling for multi-scale modeling. However, spatial up-sampling operators (e.g., interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, incapably exploring the global dependency. In contrast,  the Fourier domain is in accordance with the nature of global modeling according to the spectral convolution theorem. Unlike the spatial domain that easily performs  up-sampling with the property of local similarity, up-sampling in the Fourier domain is more challenging as it does not follow such a local property. In this study, we propose a theoretically feasible Deep Fourier Up-Sampling (FourierUp) to solve these issues. We revisit the relationships between spatial and Fourier domains and reveal the transform rules on the features of different resolutions in the Fourier domain, which provide key insights for FourierUp's designs. FourierUp as a generic operator consists of three key components: 2D discrete Fourier transform,  Fourier dimension increase rules, and 2D inverse Fourier transform, which can be directly integrated with existing networks. Extensive experiments across multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp. Code will be publicly available.

Deep Fourier Up-Sampling
Man Zhou! Hu ; Huang", Feng Zhao?,
Jinwei Gu Chen Change Loy", Deyu Meng 1.5, Chongyi Li
'efeei Institute of Physical Science, Chinese Academy of Sciences, China University of Science and Technology of China, China
38 Ss-Lab, Nanyang Technological University, Singapore "Xi'an Jiaotong University, China
"Pazhou Laboratory (Huangpu), China
"SenseBrain Technology (SenseTime Research USA), USA
{manman, , yuhu520, h1111)2m00..01111m111111...1.11m ustc. edu. cn, fzhao9560ustc. edu.eisstteeeeccccm...
gnjineeedeesseebainn..et ai, dymeng@mail.xxjuu. edu.
(ccloy,chonggi..11)0mee
http:///iiccooggg.gggi.........a filess
Abstract
Existing convolutional neural networks widely adopt spatial down-/up-sampling for multi-scale modeling. However, spatial up-sampling operators (e.g., interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, in- capably exploring the global dependency. contrast, the Fourier domain obeys the nature of global modeling according to the spectral convolution theorem. Unlike the spatial domain that performs up-sampling with the property of local similarity, up-sampling in the Fourier domain is more challenging as it does not follow such a local property. this study, we propose theoretically sound Deep Fourier Up- Sampling (FourierUp) to solve these issues. We revisit the relationships between spatial and Fourier domains and reveal the transform rules on the features of differ- ent resolutions in the Fourier domain, which provide key insights for FourierUp's 's designs. FourierUp as a generic operator consists three key components: 2D dis- crete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform, which can be directly integrated with existing networks. Extensive experiments across multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super- resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp. Code is available at htttps://mannn11990 gi thub.
1 Introduction
Spatial down-/up-sampling has been widely used in convolutional neural networks for multi-scale modeling. For example, U-Net M variation of encoder-decoder, employs pooling layers to reduce the feature resolution in the encoder and then recovers the resolution using up-sampling operations in the decoder. In addition, the feature pyramid (2-5 and image pyramid 6 driven multi-scale neural networks rely on the down-/up-sampling operation to obtain multi-scale property and improve modeling capability. However, spatial up-sampling operators (e.g.. interpolation. transposed convolution, and un-pooling) heavily depend on local pixel attention, and thus cannot
MMan Zhou and Hu Yu contribute equally. "ooresponning author: Chongyi Li.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Down
Up
explore deep Fourier Spatial domain Down-and Up-sampling
DFT
Down
Up
IFTT
(b) Fourier domuin with the same Down-and Up-sampling as (a)
IDFT
Periodic Padding
Section-3.1
Area Interpolation
Down
DFT
IDFT
Section 3.2
Area mreacropping
Supplementary materials
IDFT
Corner Interpolntion
(c) Fourier domain with our proposed "Fourier Up-sampling"
Figure I: Motivation. (a) and (b) depict that arbitrary up-sampling, e.g., interpolation, in the Fourier domain produces sub-optimal result it does follow the same local similarity property as that in the spatial domain. This motivates us to design a more ingenious "Fourier Up-Sampling" operator, dubbed as FourierUp. has three alternative variants: Periodic Padding, Area Interpolation/Cropping and Corner Interpolation, as illustrated in (c).
explore the global dependency that is indispensable for many computer vision tasks [10
20 According to the spectral convolution theorem, the Fourier domain obeys the nature of global alternative solution for multi-scale modeling. However, unlike the spatial
modeling, providing domain with local similarity property, up-sampling in the Fourier domain more challenging as it does not follow such local property. The observation encourages up-sampling.
Recent studies have explored information interaction in both spatial and Fourier domains. FFC 21 for instance, replaces the conventional convolution with spatial- Fourier interaction, which consists of a spatial (or local) path that performs conventional convolution on portion of input feature channels and spectral (or global) path that operates in the Fourier domain. DFT devises Residual Fast Fourier Transform Block integrate both low- and high-frequency residual information by performing the interaction between a regular spatial residual stream and channel-wist Fourier transform stream. However, the aforementioned methodologies only interact at single resolution scale, and the spatial-For -Foriir interaction potential multiple scales in the Fourier domain has not been investigated. The key to solving this problem lies in how to implement deep Fourier up- sampling for multi-scale Fourier pattern modeling.
Challenges. Owing to the local similarity and cross-scale position invariant properties of the spatial domain, the various spatial up-sampling operations including transposed convolution, un an-pooling, and interpolation techniques are capable of using the pixel neighboring relationship to interpolate unknown pixel values at local regions, increasing the spatial resolution of the features, as shown in Figure aa). In contrast to the spatial domain, the Fourier domain does not share the same scale- invariant property and local texture similarity, and hence cannot implement up-sampling using the same techniques the spatial domain, illustrated in Figure bb..

---

Solutions. In this paper, we wish to investigate the possibility of devising a reliable up-sampling in the Fourier domain in theoretical sound manner. To answer this question, we first revisit the relationship between spatial and Fourier domains, revealing the transform rules on the features of different resolutions in the Fourier domain (see Section [.1 and Section 3.2 On the basis of the above rules, we propose theoretically feasible Deep Fourier Up-Sampling (FourierUp). Specifically, we develop three variants (Periodic Padding, Area Interpolation/Cropping and Corner Interpolation) of FourierUp (see Section 3.3). as illustrated in Figure c)). Each variant consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform. FourierUp is generic operator that can be directly integrated with existing networks. Extensive experiments on multiple computer vision tasks, including object detection, image segmentation, image de- de-raining, image dehazing, and guided image super-resolution. demonstrate the consistent performance gains obtained by introducing our FourierUp. We believe that the proposed FourierUp could refresh the neural network designs where the spatial and Fourier information interaction at only a single resolution scale are mainstream choices.
Contributions. 1) We propose Deep Fourier Up-Sampling. a novel method that enables the integration of the features of different resolutions in the Fourier domain. This is the first thorough effort to explore the Fourier up-sampling for multi-scale modeling. 2) The proposed FourierUp is generic operator that can be directly integrated with the existing networks in a plug-and-pun manner. 3) Equipped with the theoretically sound FourierUp, show that existing networks could achieve consistent performance improvement across multiple computer vision tasks.
2 Related Work
Spatial Up-Sampling. Convolutional neural networks with spatial loww-/up-sampllee have become the de facto structures in many computer vision tasks (23/31]. Typically, U-Net [I builds multi-scale feature maps using the encoder with down-sampling and then utilizes the up-sampling operation to fuse the multi-scale features in the decoder. Additionally, the feature pyramid 2-5 and image pyramid 9 are commonly used to obtain the multi-scale property neural networks 6 9 Among them. spatial up-sampling plays a significant role in multi-scale modeling. However, existing up- sampling operations only work in the spatial domain and current studies rarely explore the potential (e.g., the global modeling capability) of up-sampling in the frequency domain.
Spatial-Fourier Interaction. Recently, several studies attempt to employ Fourier transform in deep Some of these efforts use discrete Fourier transform to transfer the spatial features to the Fourier domain and then use frequency information improve the performance of
models 32 35
21
34 Another line is to use convolution theorem to speed up the models, such as using fast Fourier transform (FFT) K5 21. For example, FFC (2 replaces the convolution with the spatial-Fourie interaction. The work proposed in [36 uses spectral pooling to reduce feature resolution by truncating the frequency domain representation. However, all the techniques only interact with each other at a single spatial resolution and have not explored the interaction potential at multiple resolutions in both spatial and frequency domains as performing the frequency up-sampling is non-trivial. As a tentative exploration, we study the relationship between the spatial domain and Fourier domain and reveal the transform rules over the feature of different resolutions in the Fourier domain. This delivers the underlying insights for the designs of multi-scale Fourier modeling patterns, which has the potential of versatility for different network architectures.
particular tasks 32
3 Deep Fourier Up-Sampling
We first explore the mapping relationship between the spatial and Fourier domains, and then present three Deep Fourier up- samplin variants, including i) periodic padding of magnitude and phase, ii) area up-sampling of magnitude and phase, and iïi) corner interpolation of magnitude and phase, based on the explored transform rules. In terms of the first two variants, we provide two theorems and their proofs as follows while the third is reported in supplementary materials.
Definitions. [xx y) E 1R2MX21 X2N is the 2-times zero-inserted up-sampled version of g(r, v) E R in spatial domain, and F(u, v) E R2Mx2N G(u, v) E MNN H Hu,, v) E 2R2 x2N is the 2-times area-interpolation up-sampled Fourier transform of G(u, v), and h(2, v) E RMxN denotes their inverse Fourier transform.
xN denote their Fourier transforms.
3

---

Theorem-1. F(u, u) = F F + N) = F(u+ M,U+ N) and G(u, = F(u.)
where u = 0, 0, ..........
and e = 1, M F(u, v) exactly the periodic padding of G(u, v) where G(u, 0) is exactly the quarter of (u, with the value being times decay.
Theorem-2. (2u, 2v) (2u + 1,2v) (2u, 2r 1) H (2u 20 1) G(u, v) with 0, . M- and 0, ,.... N 1 and
h(,,8) v) = A v)
g v)
4
h(x + M,9) A(2M I. v)
g U)
(I)
h(T, + N) A(r, 2N y)
g(2,8) y)
4
h(x + M N) A(2M - z, 2N y)
g y)
4
where A(x,V) = + e ArF
+
+
+ ) and x 0,
M - 1,y=0,1, 0,1.
Theorem-3. Suppose the corner interpolated F (u.v) v) of the Fourier map G(u, v) MR holds that the inverse Fourier transform foor (z, v) of P (u
fg (x, v) o x'
2 2 Je 1 4/) (-1)(=-1)
(2)
where x' = 2x and y' 2y, 0, 1, M - and 0, 1...... N 1
3.1 Proof-1 of Theorem-1: Periudic Padding of Magnitude and Phase
Note that f(x, v) E R2M is ua-sampled over g(x, RMX by factor 2. The relationship between g( y) and f(x, y) can be written as
f(r,y) = {
( 5)
I 2m, y = 2n others
(3)
0,
where m = 1.2,.... M - and n = expressed as
N .,2,...,NV
1, the Fourier transform F(u, v) of f(z, y) is
2N
u(,,u) 4MN Σ
Σ f(x, 12ml 12
M-1N- N-1
Σ
-52 m(20) m(2u) f(2x, 2y)e
4MN
==
(4)
M-IN-1
Σ
f(2x, 2y)e 122
4MN
0=0
4MN Σ Σ
g(x,g)e -2x (3
Then, we show the periodicity of V) E X2Mx2N with M and N. means F(u,v) F(u M, Flu, + NN = MM +N) with 0, 2........A N - and
0, 0,1,2 ........- 1.
We take the F(u, F(u + M, for example and recall Eq. as
1
f(2x, 2yje 52 (u+
F(u M.v) = 4MN
M M-IN-1
1
Σ
f(2x, 2y)e
1200 (uz 5 2182
4MN
(5)
M N-1
52nt (22 (xF+
f(2x, 2y)e
4MN
==0 ==0
= F(u, v),

---

where e 22jz = 1 for any itteeer z. Similarly, we can proof the periodicity of F(u, 2) as well.
Based the above proof, the DFT of g(ar, y) can be formulated as:
M- N-1
G(u, v)
1
g(z, -))
(6)
MN
Revising Eq. 4 we can figure out that G(u, v) F(u.o)
3.2 Proof-2 of Theorem-2: Area Interpolation of Magnitude and Phase
The 2D Inverse Discrete Fourier transform (IDFT) of G(u, v) can be written as:
M-1N-1
g(a,
Σ Σ
(7)
MN
We up-sample G(u,, v) with a size of M x N to get (u, v) with size of 2M x 2N. Specifically, the area interpolation shown in Figure 1b) used for interpolation and then the interpolated pixels
are the same as the original pixel in the x 2 local regions. Namely, H (2u, 2v) = H2u + 1, 2v)
H (2u, 2c 1) H2n + 1, 2r + 1) C(u, v) with O, l,....., M
and v 0, 1,.,,.. N
Similar to Eq. (4).
we can infer
2M
2M-122 12N-1
h(a.8)
Σ
Σ
H(u, 22 #
4MN
W=D
M-IN- IN-1
M-1 N-1
(2u, 20 20
Σ
Σ
H2u + 1,20)e 12 337)
AMN
AMN
M-1 N-1
M-IN-1
Σ Σ
H(2u, 2v 1)
ΣΣ
lf2u 1,20 1) (2a Dualay
4MN
4MN
E
Σ
G(a,
++
4MN
(8)
Similarly, we can write g(x, 3) as
M M-1N-- --1 Σ ==0 0=0
g(I, y)
(9)
MN
Recalling Eq. (8) and Eq. 10 can infer
Jee + e
h(x, y)
M
e
g(z, y).
(10)
4
where 0, 1..1,..... M 1,
N 1. We remark + e jex +e N + e ie x)
+e'N
A(x, y).
|A v)1² = (1 + cos
+ cos N
+ cos .( N 1)² + (sin _ M
x
+ sin n N
+ sin .(
x
+ N ))²
M
M
We find that the variable shares the operations the variable y. For brevity, we only take the operation variable r as example
21A(z, 2)
2T
M (1 + coS x
+ cos ग V + cos + N
N )) sin sin x M + y
))
Or
M
M
M
2T
(sin
+ sin
N + sin (( x
y
M N Mccos (( x M) - cos a M
+
+ ))
(11)
M
M
4x
(sin
I [l coST( T( ))- M
M
Equally, we have
a A(, r)
4m
sin (1 cos T
(12)
By
N N
M
We prove that the partial derivative of |A y) on both x and y is negative for x E (0, M 1] and y E [0, N - 1]. Besides, we have
[A(z, v) |A(2M v, v) AA2M 2N y) = |4(2M 2, v)1- That to say, the intensity drops from the side to the center, shown in Figure 4 Specifically, the intensity drops zero at the position of I M y N.
(13)
5

---

def DFU_Pe Pudddadddngg)
def DFU ArealInterrolation
# X: input with shape Di, C, н, w] # and P the amplitude and phase A, FFT(X)
# X: input with shape DI, C, H, W] and are the amplitude and phase A, P FFT(X)
# Fourier up-sampling transform
Fourier up-sampling transform rules A_aip Area-Interpolationn (A)
rules Periodic-Padding
A_pep
P_aip Area-Interpollttonn (P)
P P_pep
Periodic- Periodic-Padding(P (P)
A_aip Conva_1xi (A_aip)
A_pep Convs_ (aav__lx(A_ppep)
P_aip Convs_ixi (P_aip)
P-pep = Convs_ 1x1 (p_pexix1pppppp peep)
Inverse Fourier transform iAFT(__aip, P_aip)
Inverse Fourier transform Y 1FFT(A_pep, P_pep)
#Area Cropping
Return Y # [N, C, 2H, 2W]
Area- Prea-Cropping(1) Resize (Y)
Return Y # [N, C, 2H, 2W]
Figure 2: Pseudo-code of the two variants of the proposed deep Fourier up-sampling, The left is periodic pudding variant while the right is the area interpolation-cxppping variant.
2W
2W
W
W
II
211 II
211
(a) Periodic Padding
(b) Area Interpolation
Figure 3: The illustrations of (a) periodic padding and (b) area interpolation Figure 2 Each small color square represents pixel of the amplitude/phase component in the Fourier domain.
3.3 Architectural Design
Recall the Theorem-1 and Theorem-2, we propose two deep Fourier up-sampling variants: Periodic Padding and Area
Prarpolation-rroppin
Periodic Padding Up-Sampling. The pseudo-code of Periodic Padding Up-Sampling shown in the left of Figure 2 Given image X R R we first adopt the Fourier transform FFT(X) to obtain its amplitude component and phase com- ponent P. We then perform the periodic padding over A and P two times in both the H and W dimensions, as illustrated in Figure
3 (). The padded A_p and P P_pepp then fed into two independent convolution module with x kernel and followed by the inverse Fourier transform iFFT(.) to project the padded ones back to spatial domain.
0.4
0/6
Figure The surface of A(x,y) y).
Up-Sampling. The pseudo-code of Area Ceappollaion-Cropppiii Up-Sampling shown in
Area Interpolation
right of Figure We first conduct the Area interpolation over the phase and amplitude by 2 x area interpolation with the same pixel, as illustrated Figure3(bb) We then employ the inverse Fourier transform to project the interpolated ones back to spatial domain. As described in Section 3.2 the

---

inverse spatial representation will be periodic while the pixel value will be decay. The degree of decay of the pixel increases when the pixel is closer the center. To better maintain the information. we perform the area cropping operation (detailed in Figure 1 in the four corners with the W # 2 2 size and then merge them together as a whole at spatial dimension, finally resize them to the size of 2H x 2W.
Note that albeit being designed on the basis of strict theories, both constructed spectral up-sampling modules contain certain approximations, like learnable 1 x 1 convolution operator instead of strictly 1 1/4 as described in Theorem-1 1 of main manuscript, and an approximation cropping to preserve the map corners instead of accurate A mapping as proved in Theorem-2 of main manuscript and Theorem-3 of supplementary materials. Such strategy makes the proposed modules able to more easily implemented and more flexibly represent real data spectral structures. It is worth noting that this should be the first attempt for constructing easy equitable spectral upsampling modules, and hope it would inspire more effective and rational ones from more spectral perspectives.
Experiments
To demonstrate the efficacy of our proposed deep Fourier up-sampling, we conduct extensive exper- iments on multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution. We provide more experimental results in the supplementary material.
4.1 Experimental Settings
Object Detection. Following [2]. the PASCAL VOC 2007 and 2012 training sets 37 are used training data. The PASCAL VOC 2007 testing set is used for evaluations the ground truth annotations of VOC 2012 testing set are not publicly available. We employ the FPN-based Faster with ResNet50 backbone and YOLO-v3 with Darknet53 [8 baselines.
RCNN 2
Image Segmentation. Following 39 40 Synapse Dataset and CANDI Dataset are used as the testbed of medical image segmentation. We adopt the two representative image segmentation algorithms, U-Net and Att-UNet 41 as the base models.
Image De-raining, Following 42 we choose two widely-used standard benehmark datasets, including Rain200H and Rain200L, for evaluations. we employ two representative de-raining methods, LPNet with up-sampling 11 and PReNet without up-sampling [42 as baselines.
Image Dehazing. Following [10. employ RESIDE[43) dataset [4 for evaluations. We also use two different network designs AODNet [4] without up-sampling operator and MSBDN 10 with
up-sampling operator, for validation.
Guided Image Super-resolution. Following 14 46), we adopt the pan-sharpening. the representative task of guided image super-resolution, for evaluations. The WorldView II, WorldView III, and GaoFen2 in [14 (46) are used for evaluations. We employ two different network designs for validation, including PANNET 47 without up-sampling operator and DCFNET 48 with up-sampling operator.
Several widely-used image quality assessment (IQA) metries are employed to evaluate the per- formance, including the relative dimensionless global error in synthesis (ERGAS) [49]. the peak signal-to-noiss ratio (PSNR), the spectral angle mapper (SAM) [50, DSC, and HD95.
4.2 Implementation Details
Regarding the above competitive baselines, they can divided into two categories: one with spatial up-sampling ((I) Att- UNet [41). DCFNET 48) LPNet (II] MSBDN (10) and another without
spatial up-sampling (PReNet (42), AODNet 45), PANNET (7)). The purpose of the exploration on
the baselines without spatial up-sampling is to show the versatility of our FourierUp for different network structures. Different from directly replacing the spatial sasslmmling with the FourierUp in the baselines with spatial up-sampling, we need to encapsulate the FourierUp for the baselines without spatial up-sampling, in which a down- down-sampling operation is introduced to first reduce the resolution of features. We provide the detailed structures of the encapsulated FourierUp and the baselines with the FourierUp in the Figure 5 supplementary material.
For the baselines with spatial up-sampling, perform the comparison over four configurations:
7

---

Spotialt p
Transspoeed lntepolation,
with up up-sampling
FouricrUip
Padding, cruppitg etc.
2 2"""
Hix W.
Spotcl Down
oearestlincer Jonn-smmpling, etc.
(b) with uy-sampling
H Hxwxc
with up-sampling
up-sampling
Pouricrlip
Spatialtip
Configuration - YourictUp-Arealpppte
(c) Configuration of "Spatial-Up"
Figure 5: The implementation details of FourierUp into the existing baselines with up-sampling.
1) Original: the baseline without any changes;
2) FourierUp-AreaUpp replacing the original model's spatial up-sampling with the union of the Area-Interpolationn variant of our FourierUp and the spatial up-sampling itself;
3) FourierUp-Padding: replacing the original model's spatial up sampling operator with the union of the Periodic-Padding variant our FourierUp and the spatial up-sampling itself;
4) Spatial-Up: replacing the variants of FourierUp in the settings of 2)/3) with the spatial up- up-samplnng. For a fair comparison, we use the same number of trainable parameters as 2)/3).
For the baselines without spatial up-sampinng, perform the comparison over four configurations:
1) Original: the baseline without any changes;
2) FourierUp-Areal replacing the original model's convolution with the encapsulated FourierUp that is equipped with the Area-Interpolation variant;
3) FourierUp-Padding replacing the original model's convolution with the encapsulated FourierUp that is equipped with the Periodic-Padded Panidii variant;
4) Spatial-Up: replacing the the encapsulated FourierUp of the settings of 2)/3) with the spatial up- ua-sanpling. For fair comparison, we use the same number of trainable parameters as 2))
4.3 Comparison and Analysis
Quantitative Comparison. We perform the model performance comparison over different config- urations, as described in implementation details. The quantitative results are presented in Tables ] to5] where the best and second best results are highlighted in bold and Underline. From the results, by integrating with our proposed two FourierUp variants, we can observe performance gain against the baselines across all the datasets in all tested tasks, suggesting the effectiveness of our approach. For example, for the PReNet of Table I "FourierUp-padding" and "FourierUp-Arcal" 0.83dB/0.65dB and 2. 1..dB/1.99 19B PSNR gains than the "Original", 0.52dB/0.3444 and 1. 1.7dB/1.ddB PSNR gains than "SSpatia-Up" on the Rain200H and Rain200L datasets, respectively. Such results validate the effectiveness of our proposed FourierUp. The corresponding visualization consistently supports the analysis in Figure where the FourierUp is capable of better maintaining the details.
obtain
Qualitative Comparison. Due to the limited space, we only report the visual results the de- raining/dehazing task in Figures @egures/aande and that can more clearly show the effectiveness of FourierUp. More results can be found the supplementary materials. As shown, integrating the FourierUp with the original baseline achieves more visually pleasing results. Specifically, zooming-in in the red box

---

region of Figures 6 and 7 the model equipped with the FourierUp is capable of better recovering the texture details while removing the rain/hazy effect.
Table I: Quantitative comparisons of image de-raining.
Rain200H
Rain200L
Model
Configurations
PSNR
SSIM
PSNR
SSIM
Original
22.907
0.775
32.461
0.947
Spatial-Up
22.956
0.777
32.522
0.950
LPNet
FourienUp-Areaup
22.163
0.783
32.681
0.954
FourierUp-Padding
23.295
0.786
32.835
0.956
Original
29.041
0.891
37.802
0.981
Spatial-Up
29.357
0.901
38.271
0.985
PReNet
FourierUp Uprrrc
29.690
0.903
39.776
0,985
CourierUp-Paddieee
29.871
0.908
39.971
0.987
Table 2: Comparison over image dehazing.
Table 3: Comparison over object detection.
Model
configurations
PSNR
SSIM
Model
Methods
AP50
mAP
Original
18.80
0.834
Original
79.13
79.10
Spatial- Up
18.91
0.838
Spatial-Up
79.14
79.10
AODNet
Faster RCNN
FourierUp-Areaty
19.16
0.843
FourierUp-AreaUUp
79.16
79.13
FourierUp-Padding
19.35
0.847
FourierUp-Padding
79.19
79.15
Original
33.79
0.984
Original
81.68
81.63
Spatial-Up
33.90
0.984
Spatial-Up
81.68
81.63
MSBDN
YOLO-V3
FourierUp-Areatu
34.21
0.985
FourierUp-AreaUp
81.70
81,65
FourierUp-Padding
34.35
0.987
FourierUp-Paddding
81.72
81.68
Table 4: Quantitative comparisons of medical image segmentation.
Model
Configurations
synapse
CANDI
DSC
HD95 1
DSC
HD95 J
Original
76.85
39.70
86.50
3,946
Spatial-Up
76,94
38.59
86.59
3.826
U-Net
FourierUp-AreaUp
77.25
36.02
86.63
3.751
FourierUp-Padding
77.37
35.86
86.70
3,327
Original
77.77
36.02
86.29
5.601
Spatial-Up Up
77.85
35.91
86.35
5,588
Att-UNet
FourierUp-Areate
78.11
34.54
86.50
4.851
FourierUpp-Padding
78.34
34.29
86.64
4,833
Table 5: Quantitative comparisons of pan-sharpening.
WorldView-11
GanFen2
Model
Configurations
PSNR
SSIMI
SAM,
ERGAS,
PSNR1
SSIM1
SAMJ
BGASJ
Original
40.817
0.963
0.025
1.055
43.066
0.968
0.018
0.855
Sputial-Up
0.963
0.025
0.973
PANNET
PourierUp-AreaUp
41.167
0.963
1.010
65,964
0.979
0.015
0.653
SourierUp-Paddingg
41.288
0.965
0.024
1.007
46.145
0.982
0.622
Original
40.276
0.968
0.028
1.051
42.986
0.967
0.019
0.858
Spatial-Up Upp
40.319
0.968
0.028
1.046
43.157
0.970
0.017
0.850
DCFNET
FourierUp-Aree
40.484
0.968
0.025
L.15
43.881
0.979
0.014
0.829
FouricrUp Pariddiup-
40,546
0.968
0.025
1.022
44.153
0.981
0.014
0.765
5
Limitations
First, the more comprehensive experiments on broader computer vision tasks (e.g. image de-noising and image de-blurring) have not been explored. Second, the deep Fourier Up-sampling integrated with spatial up-sampling will increase the model parameter numbers. This is negligible at the significant performance gain at fewer parameter increase. Note that, the focus of this work is beyond designing plug-and-ple and-play module to integrate it into existing networks for further performance gain. This work

---

(a) Rainy image
(b) PeeNet
(c) PReNet with FourierUp
(d) Ground truth
Figure 6: Visual comparison of PReNet on the Rain200H.
(x) Hazy image
(b) AODNet
(c) AODNet with FourierUp Ground truth
Figure 7: Visual comparison of AODNet on the SOTS.
also provides powerful scale change choice of the up-sampling operator pool when developing new model from start.
Conclusion
In this paper, we have proposed deep Fourier up-sampling to explore the possibility of the up- sampling in the Fourier domain, which provides the key insight for the multi-scale Fourier pattern modeling. We theoretically demonstrate that our designs of Fourier up up-sampling are feasible. appealing that the proposed FourierUp is generic operator, thus being directly integrated with existing networks. Extensive experiments demonstrate the effectiveness our method. We believe the FourierUp has the potential to advance broader computer vision tasks, e.g., image/video super- resolution and image/video in-painting.
Broader Impact
Our work shows the promising capability of up-sampling in the Fourier domain for computer vision algorithms through two novel designs with theoretical proofs. Using our deep Fourier up-sampling with negligible computational cost will improve the performance neural networks and facilitate the development of AI in real-world applications. However, the efficacy of our method may raise potential concerns when it is improperly used. For example, the safety of the applications of method in real-world applications may not be guaranteed. We will investigate the robustness and effectiveness of method in broader real-world applications.
Acknowledgements
We gratefully acknowledge the support of the Major Key Project of PCL (PCL2021A12), MindSpore, and Ascend Processor
CANN, the JKW Research Funds under Grant 10-163--14L.2--01--000 used for this research. Chongyi Li and Chen Change Loy are supported by the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
10

---

References
[1] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[2] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[3] Seung-Wook Kim, Hyong-Keun Kook, Jee-Young Sun, Mun-Cheon Kang, and Sung-Jea Ko. Parallel feature pyramid network for object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 250, 2018.
[4] Selim Seferbekov, Vladimir Iglovikov, Alexander Buslaev, and Alexey Shvets. Feature pyramid network for multi-class land segmentation. In Proceedings the Vision and Pattern Recognition Workshops, pages 272-275, 2018
the IEEE Conference on Computer
[5] Lei Zhu, Zijun Deng, Xiaowei Hu, Chi-Wing Fu, Xuemiao Xu, Jing Qin, and Pheng-Ann Heng. Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 121-136, 2018.
[6] Yanwei Pang, Tiancai Wang, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Ling Shao. Efficient featurized image pyramid network for single shot detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7336-7344, 2019.
[7] Ziming Liu, Guangyu Gao, Lin Sun, and Li Fang. Ipg-net: Image pyramid guidance network for small object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 1026-1027, 2020.
[8] Edward H Adelson, Charles Anderson, James R Bergen, Peter J Burt, and Joan M Ogden. Pyramid methods in image processing. RCA engineer, 29())333-41, 1984.
[9] Jiapeng Luo, Jiaying Liu, Jun Lin, and Zhongfeng Wang. A lightweight face detector by integrating the convolutional neural network with the image pyramid. Pattern Recognition Letters, 10-1180- 2020.
[10] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2154-2164, 2020.
[11] Xueyang Fu, Borong Liang, Yue Huang, Xinghao Ding, and John Paisley. Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems, 3 1794 2020.
[12] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards til- object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-1144 2017.
[13] Xueyang Fu, Zeyu Xiao, Gang Yang, Aiping Liu, Zhiwei Xiong, et al. Unfolding taylor's approximations for image restoration. Advances in Neural Information Processing Systems, 34:18997-19009, 2021.
[14] Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. Effective pan- sharpening with transformer and invertible neural network. IEEE Transactions on Geoscience and Remole Sensing, 60:1-15, 2022.
[15] Man Zhou, Jie Huang, Xucyang Fu, Feng Zhao, and Danfeng Hong. Effective pan-sharpening by multiscale invertible neural network and heterogencous task distilling. IEEE Transactions on Geoscience and Remote Sensing, 60: 10:1-14, 2022.
[16] Man Zhou, Keyu Yan, Jie Huang, Zihe Yang, Xueyang Fu, and Feng Zhao. Mutual information- driven sharsharen Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1798-1808, June 2022.
11

---

[17] Man Zhou, Fan Wang, Xian Wei, Rujing Wang, and Xue Wang. Pid cnttroleer-issiree model design for single image de-raining. IEEE Transactions on Circuits and Systems II: Express Briefs, (9)4) 2022.
[18] Man Zhou and Rujing Wang. Control theory-inspired model design for single image de-raining. IEEE Transactions on Circuits and Systems II: Express Briefs, 69(2):649-653, 2022.
[19] Man Zhou, Jie Xiao, Yifan Chang, Xueyang Fu, Aiping Liu, Jinshan Pan, and Zheng-Jun Zha. Image de-raining via continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1907-4910 June 2021.
[20] Man Zhou, Keyu Yan, Jinshan Pan, Wenqi Ren, Qi Xie, and Xiangyong Cao. Memory- augmented deep unfolding network for guided image super-resolution, 2022.
[21] Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, 2020.
[22] Xintian Mao, Yiming Liu. Wei Shen, Qingli Li, and Yan Wang. Deep residual fourier transfor- mation for single image deblurring arXiv preprint arXiv: 2021.
[23] Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. Advances in Neural Information Processing Systems, 34, 2021.
[24] Yulun Zhang, Kunpeng Li, Kai Li, Gan Sun, Yu Kong, and Yun Fu, Accurate and fast image denoising via attention guided scaling, IEEE Transactions on Image Processing, 30:6255-6265. 2021.
[25] Yuchen Fan, Jiahui Yu, Yiqun Mci, Yulun Zhang, Yun Fu, Ding Liu, and Thomas Huang. Neural sparse representation for image restoration. Advances in Neural Information Processing Systems, 33:15334-15404 2020.
[26] Wenqi Ren, Jiawei Zhang, Jinshan Pan, Sifei Liu, Jimmy Ren, Junping Du, Xiaochun Cao, and Ming-Hsuan Yang. Deblurring dynamic scenes via spatially varying recurrent neural networks. IEEE transactions on pattern analysis and machine intelligence, 2021.
[27] Wenqi Ren, Jiawei Zhang, Lin Ma, Jinshan Pan, Xiaochun Cao, Wangmeng Zuo, Wei Liu, and Ming-Hsuan Yang. Deep non-blind deconvolution via generalized low-rank approximation. Advances in neural information processing systems, 31, 2018.
[28] He Zhang and Vishal M Patel. Density- aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 695-704, 2018.
[29] He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using a conditional generative adversarial network. IEEE transactions on circuits and systems for video technology. 2019.
[30] Jinshan Pan, Sifei Liu, Deqing Sun, Jiawei Zhang, Yang Liu, Jimmy Ren, Zechao Li, Jinhui Tang, Huchuan Lu, Yu- fu-Wing Tai, et al. Learning dual convolutional neural networks for low- level vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3070-3079, 2018.
[31] Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang. Variational denoising network: Toward blind noise modeling and removal. In Advances in Neural Information Processing Systems, volume 32, 2019,
[32] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation, 2020.
[33] Shaohua Li, Kaiping Xue, Bin Zhu, Chenkai Ding, Xindi Gao, David Wei, and Tao Wan. Falcon: A fourier transform based approach for fast and secure convolutional neural network predictions. In Proceedings of the IEFE//VV Conference on Computer Vision and Pattern Recognition, pages 7705-8714, 2020.
12

---

[34] Jae-Han Lee, Minhyeok Heo, Kyung-Ran Rae Kim, and Chang- Su Kim. Single-image depth estimation based on fourier domain analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 330-339, 2018.
[35] Caiwen Ding, Siyu Liao, Yanzhi Wang, Zhe Li, Ning Liu, Youwei Zhuo, Chao Wang, Xuehai Qian, Yu Bai, Geng Yuan, et al. Cirenn: accelerating and compressing deep neural networks us- ing block-cireulant weight matrices. In Proceedings of the 50th Annual IEEF/ACM International Symposium on Microurchitecttre, pages 395-408, 2017.
[36] Oren Rippel, Jasper Snoek, and Ryan P Adams. Spectral representations for convolutional neural networks. Advances neural information processing systems, 28, 2015.
[37] M. Everingham, L. V. Gool, Cki Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2)3303-338, 2010.
[38] Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable and efficient object detection. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10778-10787, 2008,,0077,000 2020.
[39] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang. Le Lu, Alan Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.043300 2021.
[40] Shuhao Fu, Yongyi Lu, Yan Wang. Yuyin Zhou, Wei Shen, Elliot Fishman, and Alan Yuille. Domain adaptive relational reasoning for 3d multi-organ segmentation. In International Con- ference Medical Image Computing and Computer-Assisted Intervention, pages 656-666. Springer, 2020.
[41] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint 1004vv:884.003333 2018.
[42] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: better and simpler baseline. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[43] Boyi Li. Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng, Wenjun Zeng, and Zhangyang Wang. Benchmarking single-image dehazing and beyond. IEEE Transactions on Image Processing, 28(1):492-50) 505, 2019.
[44] Yanfu Zhang, Li Ding, and Gaurav Sharma. Hazerd: An outdoor scene dataset and benchmark for single image dehazing. In 2017 IEEE International Conference on Image Processing (ICIP), pages 2205-3209, 2017.
[45] Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng. Aod-net: All-in-one dehazing network. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[46] Keyu Yan, Man Zhou, Liu Liu, Chengjun Xie, and Danfeng Hong. When pansharpening meets graph convolution network and knowledge distillation. IEEE Transactions on Geoscience and Remote Sensing, 60:1-15, 2022.
[47] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. Pannet: deep network architecture for pan-sharpening. In IEEF International Conference on Computer Vision, pages 4449, 2017.
[48] Xiao WU, Ting-Zhu Huang, Liang-Jian Deng, and Tian-Jing Zhang. Dynamic cross feature fusion for remote sensing pansharpening. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), pages 14687-14696, October 2021.
[49] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce. Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data fusion contest, IEEE Transactions on Geoscience and Remote Sensing, 45(10)33012-300 30220022111,,, 2007.
13

---

[50] A. F. Goetz J. R. H. Yuhas and J. M. Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. Proc. Summaries Annu. JPL Airborne Geosci. Workshop, pages 147-149, 1992.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes], see Section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes], See Section
3.1 and Section
(b) Did you include complete proof's of all theoretical results? [Yes]. See Section 3.1 and
Section 3.2
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] , the effect of random seed could almost be negligible since we set the same initiation seed during experiments. Reproducibility can be guaranteed.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you't 're using/curating? [Yes], the data creators claim that they allow and encourage the data for scientific research.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We train and test the framework using the publicly accessible datasets.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "Gl95N6_2LfB", "reviewer": "Reviewer_mfGV", "summary": "This paper tries to solve a challenging issue in frequency related neural network designs, i.e., how to model the multi-scale frequency patterns. Unlike the simplex up-sampling operations in the spatial domain, up-sampling frequency information in a neural network is non-trivial because the local property of spatial up-sampling cannot be directly used in the frequency domain. According to the spectral convolution theorem, this work proposes an interesting and theoretically sound solution, called Deep Fourier Up-Sampling (DFU). DFU is the first work to consider the modeling of multi-scale frequency patterns and provides key insights for the frequency related network designs. Besides, the DFU can be implemented by different variants with different theoretical support. The DFU is also flexible in its compatibility with existing neural networks. Extensive experiments on multiple vision tasks demonstrate the effectiveness of the proposed DFU.", "questions": "See the above weaknesses. The authors are suggested to give the detailed responses. ", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Interesting motivation: This work takes an interesting and challenging issue, multi-scale information modeling in the frequency domain, into account. Although there are some frequency-related neural networks, they cannot effectively and accurately model the relationship of multi-scale frequency information due to the Fourier transformation characteristics. However, the multi-scale frequency information is important for vision tasks. This key issue is the focus of this paper.\n2. Theoretical proof: For the proposed Deep Fourier Up-Sampling, this work provides detailed and rigorous proof, making a theoretical sound solution. These proofs provide a new view to understand deep Fourier up-sampling and convince the readers. \n3. Flexibility: The proposed Deep Fourier Up-sampling can be a generic operator, which can be compatibility with existing neural networks and boost their performance. \n4. This paper conducts sufficient experiments to demonstrate the effectiveness of the proposed Deep Fourier Up-sampling. To show the versatility of the proposed method, the experimental comparisons are carried out on diverse vision tasks, including object detection, image segmentation, image de-raining, image de-hazing, and guided image super-resolution. The experiments on multiple vision tasks provide strong supports for the advantages of the proposed multi-scale frequency information modeling.\n\n\nWeaknesses:\n1. Although this paper provides sufficient experimental results, it would be better if the parameter comparisons and more visual results could be provided. The reviewer understands that this paper has conducted extensive experiments, covering low-level and high-level vision takes in the limited space. More results could be provided in the supplementary material. \n2. In Section “Comparison and Analysis”, the underlying reasons why the FourierUp works better are insufficient. More explanations are expected. Furthermore, it is a little confusing that whether each variant of the FourierUp prefers a specified vision task? How to choose the variant when one uses the FourierUp for different vision tasks? Some insightful analysis or suggestion should be provided, which would guide the followers to make full use the capability of the proposed FourierUP. \n3. It would be better if more visual results could be provided in the main paper rather than the supplementary material. It is fine to shrink the sizes of figures for showing more cases. Current format is inconvenient to switch between the main paper and the supplementary material for checking the visual results. \n4. The flowchart of implementation details in supplementary materials are suggested to transfer to main manuscript.   \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DcZS8xHBqd", "reviewer": "Reviewer_tPpX", "summary": "The authors propose an upsampling layer based on the Fourier transform and properties of the frequency domain. The motivation behind the work is quite vague, only resorting to the fact that the Fourier transform is a global operator rather than local.", "questions": "I would like the authors to clarify where the novelty lies in the proposed method and a more precise explanation of why we should expect gains with respect to existing techniques.", "limitations": "The authors addressed the limitation that two very important benchmarks (classification and denoising) are missing. Still, when proposing a very general technique, the body of proof required to demonstrate it is advantageous needs to be substantially larger than what is presented.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is generally poorly written, lacks a clear motivation, and the novelty is unclear. \n\nStrengths:\n- Frequency-based techniques for layer operations are an area of research that deserves more interest\n\nWeaknesses:\n- Poor presentation. Readability is very low, especially due to long derivations and proofs in the text rather than in the appendix. It is hard to grasp what the method does at a glance\n- Novelty unclear. It seems that the work just exploits well known properties of the Fourier transform of upsampled signals. It is also unclear why the scheme should outperform any other technique\n- Experiments show very marginal improvements and lack two very important problems: image classification and denoising", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bz7hsF4igaQ", "reviewer": "Reviewer_SXL9", "summary": "This paper proposes a theoretically feasible Deep Fourier Up-Sampling (FourierUp) to address up-sampling operators in Fourier domain. Extensive experiments across multiple computer vision tasks are conducted.", "questions": "Please see [Weaknesses].", "limitations": "Yes, the authors discussed the limitations in Sec. 5. However, I am also curious about the failure cases that the proposed method cannot handle currently.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Idea is interesting\n- Easy to follow\n- Clear writing\n\nWeaknesses:\n1. Sec. 3 introduces the details and theoretical evidences of the two kinds of FourierUp Modules, including Periodic Padding interpolation and Area Interpolation. However, it's not clear to me why the proposed FourierUp method is better. Even with the illustration in Fig. 1, the results of the two variants are not that good.  Thus, deeper analysis on why and how FourierUp helps improve the performance should be discussed and illustrated.\n\n2. In experiments, more recent SOTA competitors are required to be compared. For example, PReNet is from CVPR'19, Aod-Net and PanNet are from ICCV'17. I think it's ok to adopt some classical methods as the baselines when proposing a novel framework for the existing task. It would be better if the proposed method is compared with the SOTA cutting-edge methods to provide some deeper understanding on the tasks, which may make the proposed method more insightful and convincing.\n\n3. I am curious about what kind of failure cases the proposed method may have. Sec. 5 actually avoids to show such cases.\n\n4. Complexity comparisons against the competitors are required.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-_Xzwdn1fZ", "reviewer": "Reviewer_XGiH", "summary": "This paper proposes a novel up-scaling strategy to upscale the spatial features in an encoder-decoder-based spatial up-down sampling network. Upscaling is performed on the frequency domain to explore the global dependency. The upscaling module has been experimented with several computer vision tasks and experimental results show significant performance improvement in those tasks.", "questions": "Why did this paper not explore the frequency down-sampling strategies and experiments with different variants? Did frequency down-sampling not give satisfactory results? If so, why does only frequency upscaling work?", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper mainly proposes two different types of Fourier upscaling, namely periodic padding and area-interpolation. The main strengths are as follows.\n1. This paper explored the relationship between spatial and frequency domains. The theoretical evidence in support of upscaling module design is explained properly.\n2. The experiments in comparison with existing upscaling and different variants of Fourier upscaling are performed. The experimental outcome shows promising performance.\n3. The paper writing and explanations are good. I actually had this kind of idea but never tried this. It is refreshing to see that this paper has done this. This work will drive the community in a certain direction. \n\nThere is no such major weakness in this paper. I would like to know an explanation behind those results as we can see from Figure 1 that reconstructions in the spatial domain are really bad after periodic padding or area-interpolation. Why does it still work?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept