Transform Once: Efficient Operator Learning in Frequency Domain
Authors: Michael Poli
Stefano Massaroli
Federico Berto
Jinkyoo Park
Tri Dao
Christopher Re
Stefano Ermon
Venue: NeurIPS 2022 Accept (2022)
Keywords: convolutions
long range dependencies
neural operators
high-resolution
frequency
transform
differential equation
dynamics
turbulence
fluid flows
PDE

Abstract: Spectral analysis provides one of the most effective paradigms for information-preserving dimensionality reduction, as simple descriptions of naturally occurring signals are often obtained via few terms of periodic basis functions. In this work, we study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time: frequency-domain models (FDMs). Existing FDMs are based on complex-valued transforms i.e. Fourier Transforms (FT), and layers that perform computation on the spectrum and input data separately. This design introduces considerable computational overhead: for each layer, a forward and inverse FT. Instead, this work introduces a blueprint for frequency domain learning through a single transform: transform once (T1). To enable efficient, direct learning in the frequency domain we derive a variance preserving weight initialization scheme and investigate methods for frequency selection in reduced-order FDMs. Our results noticeably streamline the design process of FDMs, pruning redundant transforms, and leading to speedups of 3x to 10x that increase with data resolution and model size. We perform extensive experiments on learning the solution operator of spatio-temporal dynamics, including incompressible Navier-Stokes, turbulent flows around airfoils and high-resolution video of smoke. T1 models improve on the test performance of FDMs while requiring significantly less computation (5 hours instead of 32 for our large-scale experiment), with over 20% reduction in predictive error across tasks.

Transform Once Efficient Operator Learning in Frequency Domain
Michael Poli" Stanford University DL
Stefano Massaroli" Mila DiffeqML
Federico Berto* KAIST DiffeqML
Jinykoo Park KAIST
Tri Dao Stanford University
Christopher Ré Stanford University
Stefano Ermon Stanford University CZ Biohub
Abstract
Spectral analysis provides one of the most effective paradigms for information- preserving dimensionality reduction, as simple descriptions of naturally occurring signals are often obtained via few terms of periodic basis functions. In this work, we study deep neural networks designed to harness the structure in frequency do- main for efficient learning of long-range correlations in space time: frequency- domain models (FDMs). Existing FDMs are based on complex-valee vaauue transforms i.e. Fourier Transforms (FT), and layers that perform computation trum and input data separately. This design introduces considerable computational overhead: for each layer, forward and inverse FT. Instead, this work introduces a blueprint for frequency domain learning through a single transform: transform once (T1). To enable efficient, direct learning in the frequency domain derive a variance preserving weight initialization scheme and investigate methods for fre- queney selection in reduced-orde -rder FDMs. Our results noticeably streamline the design process of FDMs, pruning redundant transforms, and leading to speedups of 3 10x x that increase with data resolution and model size. We perform exten- sive experiments on learning the solution operator of spatio-temporal dynamics, including incompressible Navier-Stokes, turbulent flows around airfoils and high- resolution video of smoke. T1 models improve on the test performance of FDMs while requiring significantly less computation (5 hours instead of for our large- scale experiment), with over 20% reduction in predictive error across tasks.
the spec-
Introduction
Nature uses only the longest threads weave her patterns, so that each small piece of her fabric reveals the organization of the entire tapestry. (Feynman, 1965)
Naturally occurring signals often sparse when projected periodic basis functions (Strang, 1999), Central to recently-introduced instances of frequency-domain neural operators (Li et al., 2020: Tran et al., 2021), which refer to frequency-domen models (FDMs), is the idea of learning to modify specific frequency components of inputs to obtain desired output in data space. With hierarchical structure that blends learned transformations frequency domain coefficients with regular convolutions. FDMs are able to effectively approximate global, long-range dependen- cies in higher resolution signals without requiring prohibitively deep architectures.
"Equal contribution authors. Contact email: poli@stanford.com edu.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Yet, existing FDMs suffer from several drawbacks:
1. Slow inference: every layer of an FDM performs forward and inverse frequency domain transform, introducing a considerable computational overhead.
2. Expensive parameter scaling: each layer of typical FDMs et,,, 2020) performs long convolution over the inputs by parametrizing it in frequency domain, which scales poorly in the signal resolution.
3. Incompatibility; parameter initialization schemes and layers devised to learn directly in data space can be highly suboptimal when introduced without modifications to FDMs.
Despite attempts to improve performance (Gupta et al., 2021; Tran et al., 2021), scaling FDMs to larger data resolutions and model sizes remains fundamentally challenging'.
In this work, we start by posing the question:
To reap the benefis of learning on frequency domain representations, is if necessary to construct hierarchical deep models that perform forward and inverse frequency transforms each layer?
We provide the answer in Transform Once (T1), a model that builds representations directly in frequeney domain, after single forward transform. Each aspect T1 addresses a specific limitation of existing FDMs:
1. Fast: by performing single forward transform and optimizing directly frequency do- main coefficients of target data, T1 iterations are at least 3x to 10x faster. When scaling to larger models and higher resolutions, the relative speedups increase as the overhead of each transform grows.
2. Favourable scaling:
T1 employs a single real-valued transform, which we observe to stabi- lize training and finetuning of deep networks in frequency domain.
3. Enhanced compatibility: removing redundant transforms streamlines the design space for
T1 architectures compared to existing FDMs, allowing direct introduction of optimized layers developed for other applications e.g. UNets (Ronneberger et al., 2015).
In $ 2.1 we provide a (short) history on frequency domain approaches in deep learning, followed by background
FDMs $ 2.3. In 3, we describe how to train T1 directly in frequency domain and motivate the choice of DCT, in $ 3.1 we discuss how to choose modes of reduced-order FDMs and in & 3.2 we introduce simple Finally, in $ 4 we evaluate T1 on a suite of benchmarks related to learning solution operators for variety of dynamies: incompressible Navier-Stokes, flow around different airfoil geometries, and high-resolution videos of turbulent smoke (Eckert et al., 2019).
prriance-preserving weight initialization scheme for all FDMs.
Across tasks, T1 is 3x to 10x faster and reduces predictive errors by 20% on average. Training T1 models on high resolution videos (600 x 1062) of turbulent dynamics is significantly faster, requiring 5 hours instead of 32 hours (FNOs) for the same number of iterations.
2 Related Work and Background
2.1 Learning and Frequency Domain: A Short History
Links between seqquency-domailn signal processing and neural network architectures have been ex- plored for decades, starting with the original CNN designs (Fukushima and Miyake, 1982). Mathieu al. (2013); Rippel al. (2015) proposed replacing convolutions pixel space with element-wise multiplications in Fourier domain. In the context of learning to solve partial disferential equations (PDEs), Fourier Neural Operators (FNOs) (Li et al., 2020) popularized the state-of-the-an FDM layer structure: forward transform - learned layer - inverse transform. Similar architectures had been previously proposed for generic image classification tasks in (Pratt et al., 2017; Chi et al., 2020). Modifications to the basic FNO recipe provided in (Tran et al., 2021; Guibas et al., 2021; Wen et al., 2022). A frequency domain representation of convolutional weights has also been used
Existing methods to overcome this limitation avoid the frequency domain of inputs, instead introducing an intermediate patch embedding step (Guibas al., 2021; Pathak al., 2022).

---

for model compression (Chen et al., 2016), Fourier features of input domains and periodic activation functions play important roles in deep implicit representations (Sitzmann et al., 2020; Dupont al., 2021; Poli et al., 2022) and general-purpose models (Jaegle et al., 2021).
2.2 Learning to Solve Differential Equations
A variety of deep learning approaches have been developed to solve differential equations: neural operators and physics-informed networks (Long et al., 2018; Raissi et al., 2019; La et al., 2019; Karniadakis et al., 2021), specialized architectures (Wang et al., 2020; Lienen and Günnemann, 2022), hybrid neural-numerical methods (Poli al., 2020; Kochkov al., 2021; Mathiesen al., 2022; Berto et al., 2022), and FDMs (Li et al., 2020; Tran et al., 2021), the focus of this work.
2.3 Frequency-Domale Models
Let Dn, (n-space) to be the set of real-valued discrete signals' of resolution N. Our objective is to develop efficient neural networks to process discrete signals E Dn,
Zo. 21, ,.
E R.
We define a layer of FDMs mapping x to an output signal á E D,, as the structured operator:
X = T(x)
Forward Transform
fo
y
X fX(
Learned Map
(1)
3 T (8)
Inverse Transform
ÿ = 5 + g(x)
Residual
where T is an orthogonal (possibly complex) linear operator. We denote the --transformed n-space with DA (k-space) so that ' 'D, - Dx.- Typically, we assume TI to be a Fourier-type transform' (Oppenheim, 1999, Chapter 8) so that the k-space corresponds to the frequency domain and its elements form the spectrum of the input signal x.
The learned parametric map fo Dr + Dr is the stem of a FDM layer: it maps the k-space into itself and is typically chosen to be rank-deficient in the linear case, e.g. fx( S Sm.) A(0) Cm (m I N). The matrix Sm E Raxm selects m desired elements of X, setting the rest to zero. In the case of frequency domain transforms, this allows (1) to preserve or modify only specific frequencies of the input signal rr
Residual connections residual convolutions g (Li et al., 2020; Wen al., 2022) optionally added to reintroduce frequency components filtered by Sm- FDM mixes global transformations applied to coefficients of the chosen transform to local transformations g i.e. convolutions with finite kernel sizes. To ensure that such models can approximate generic nonlinear functions, nonlinear activations are introduced after each inverse transform.
Fourier Neural Operators Layers of the form (1) appear in recent FDMs such as Fourier Neural Operators (FNOs) (Li et al., 2020) and variants (Tran et al., 2021; Guibas et al., 2021; Wen et al., 2022).
In example, an FNO recovered from (1) by letting T be Discrete Fourier Transform (DFT) è T- o fo o T(x) = W A(0)S Wz where W NNN is the standard N-dimensional DFT matrix and W its conjugate transpose. The Discrete Fourier Transforms (DFTss is natural choice of T as it can be computed in O(N log N) via Fast Fourier Transform (FFT) algorithms (Oppenheim, 1999, Chapter 9.2).
We identify two major limitations of FDMs in the form (I); each layer performs T and T and DFTs are complex-valued, resulting in overheads and a restriction of the design space for fx(X..
With T1, we aim to develop an FDM that does not require more than a single T, while preserving or improving predictive accuracy. Ideally, the transform in T1 should be (1) real-valued, to avoid restrictions in the design space of the architecture and thus retain compatibility with existing
} For clarity exposition, models and algorithms proposed in the paper are introduced without loss of generality for one-dimensional scalar signals (i.c. D, = Rd).
e.g. discrete Fourier transform (DFT), discrete cosine transform (DCT), etc.
3

---

pretrained models, (2) universal, to allow the representation of target signals, and (3) approximately sparse or structured, allow dimensionality reduction.
W
W
W
T
X A(0)S.n
X
X
ST
Sm
fe
Commutative diagrams for FDM layers (I) and linear FNOs (frequency domain part).
3
Transform Once: The T1 Recipe
With T1, introduce major modifications to the way FDMs designed and optimized. In par- ticular, T1 is defined, inferred and trained directly in the frequency domain with only a single direct transform required process data. Hence follows the name: transform once (T1).
Direct learning in the frequency domain Consider two signals x E Dny E Dn and suppose there exists function : Dn - Dn, mapping x to y, i.e.
y= ((x).
Then, there must also exist another function Dx + Dr that relates the spectra the signals, i.c. Y = (X) being X = (xx) and Y = Tyy. In particular,
(0x) - v ((x) + Tor(== + T(x)
It follows that, from a learning perspective, we can aim to approximate & directly in the k-space rather than & in the n-space. To so, we define a learnable parametric function so DD; - Dj and train it to minimize the approximation error Jo of the output signal spectrum Y in the k-space. Given a distribution p(x) of input signals, is characterized by the following nonlinear program
Ex.y [IT(()) - 11]
min 8
Y
X
Sa
Ju
subject to y fo o T(r))
p(x)
Y
== ((x)
If T DFT, the above turns out to a close approximation equivalent, depending on the function class of fo) to the minimization of lyy -y-Ell ~l in n-space by the Sinevval-Planchere identity.
Theorem 3.1 normalized DFT. Given signal D, and transform T(v), holds ||a|| = [[V]]
Identity (Stein and Shakarchi, 2011, pp. 223) ). Let T be the
This result also applies any other norm-preserrvv transform T, e.g. normalized type-I1 DCT (Oppenheim, 1999, pp. 679). For the linear transforms considered in this work, T(x) = Wx, W E C X condition for Th. 3.1 to hold is W to be orthonormal, i.e. W*W II.
Note that T1 retains, in principle, the same universal approximation properties of FNOs (Kovachki et al., 2021) fe is allowed to operate on the entirety of the input spectrum. Given enough capacity, fg can arbitrarily approximate th, implicitly reconstructing via 7-1 T.
Speedup measurements We provide a concrete example the effect of pruning redundant trans- forms on computational costs. We measure wall-clock inference time speedups of depth d T1 T1(x): == fa o.... ... f2 fi T(x) over an equivalent depth FNO with layers (1). The only difference concerns the application of transforms between layers.
Fig. 3.1 provides the speedups on two-dimensional signals: on the left, we model depth d and investigate the scaling in signal width (i.e. number of channels) and signal resolution. On the

---

right, we fix signal width to be 32 and visualize the interaction of model depth and signal resolution. For common experimental settings e.g. resolutions of 64 or 128, layers and width 32, T1 is at least 10x x faster than other FDMs. It will later be shown ($ 4) that T1 also preserves or improves on predictive accuracy of other FDMs across tasks.
When T1 is not preceded by online preprocessing steps for inputs I, such other neural networks or randomized data augmentations, the transform on T(x) can be done once on the dataset, amortizing the cost over training epochs, and increasing the speed of T1 further.
Speedup on Two- Dimensional Data
Signal Resolution
Signal Resolution
190
1.66
128
128
64
64
Choosing the right transform The transform T in T1 is chosen to be in the class of Discrete Cosine Transforms (DCTs) (Ahmed et al., 1974; Strang, 1999), in particular the normalized DCT-II, which can also be computed O(N log N) via FFTs (Makhoul, 1980). DCTs provide effective repre- sentations of smooth continuous signals (Trefethen, 2019) and are the backbone of modern lossy com- pression methods for digital images.
16
32
48
Signal Width
Model Dopth
10
20
10
20
30
Figure 3.1: Speedup in a forward pass of T1 over FNOs sharing the same transform (DFT) on two-dimensional signals of increasing resolution. The speedup for a given configuration (point on the plane) is shown as background color gradient. The improvement grows with signal width, reso- lution and model depth.
Although other transforms are available, we empir- ically observe DCT-based T1 to perform best in our experiments. This phenomenon can be explained by the sparsity and energy distribution properties of the transformed spaces, an intrinsic property of the specific dataset and chosen transform. This in line with results of classic signal processing and compression literature. Particularly, DCT features are known to have a higher energy compaction than their DFT counterparts in variety of domains, from natural images (Yaroslavsky, 2014) to audio signals (Soon et al., 1998). Energy compaction is often the decisive factor in choosing transform for downstream tasks.
Letting T be real-valued transform in architectures preserves compatibility between fe and existing architectures e.g., models pre-trained on natural image datasets.
3.1 Reduced-Order T1 Model and Irreducible Loss Bound
We seek to leverage structure induced in Dr by T. To this end allow T1, similarly to (1), to modify specific elements of X and consequently trasform only certain frequency components of x (and y).
The reduced-order T1 model is designed to operate only on m < N elements (selected by Sm E Xx)) of the input k-space, i.e. reduced k-space Dm, = RT of lower dimension. Thus, can employ a smaller neural network T/o Dn + D,, for mapping S,,XX to the corresponding m elements Sn of the output k-space. Thus, training involves truncated objective that compares predictions with elements in the output signal spectrum also selected by Smi
X SmX
min Ery [|5m T(v) - Y1] 0
J9
subject to
ý = 79 o Sm o T(x)
(3)
x ~ p p(x) y = (xx)
O
O
Sm Y
Y
y
5

---

How to choose modes in reduced-order FDMs We now detail some intuitions and heuristic choose which modes .o....,kp: should be kept to maximize the information content in the truncated spectrum. For this reason, we evaluate the irreducible loss arising from discarding some N m modes. We recall that the (reduced) kissace training objective Jo(X, Y) reads as
m
Yx ||Sm.. Y 14
Σ
Mi 70,k, (x)))
since only the first predicted output modes Y Ykm. can be compared to Yı. We then consider the total loss Lg of the approximation task, including the N - m elements of the output k-space discarded by model, i.e.
m-
Lo(X,Y) |Y -V-STT ST
1 20 (m))
Y 0|.
(x,YY
Ro(Y)
follows that the overall loss Lg is higher than T1's training objective Jg. i.c. Lg = Jg + Ro > Jo. whilst R, represents the irreducible residual loss due to truncation of the predictions Vk-
Optimal mode selection in auto-encoding T1 In case Y = =X, the reduced-order T1 is tasked with reconstructing the input spectrum, the optimal modes minimizing the irreducible loss are ones with highest magnitude. This can be formalized as follows.
Proposition 3.1 (Top-m modes minimize the irreducible loss). Let Y = X (reconstruction task). Then the choice Ro....,km--
toPs (m) Xxxl minimizes the irreducible loss term Ro
This means that if the spectrum of X monotonically decreasing in magnitude, then low-pass filtering is the optimal mode selection.
Corollary 3.1 (Low pass filtering optimal for monotonic spectrum). If [Xa] is monotonically decreasing in k, then the choice ko,..., km-1 = O,..., minimizes the residual Ro.
However, spectra in practical datasets are commonly monotonoo e.g., the spectrum of solutions of chaotic turbulent systems (Dumont and Brumer, 1988). We show example in Fig. 3.2.
Mode selection criteria in general tasks When Y + X and the task is general prediction task, the simple top analysis is not optimal. Nonetheless, given dataset of input-output signals it is still possible perform an priori analysis on R, inform the choice of the best modes to keep.
Often, we empirically observe the irreducible error R, for reduced-order T1 to be smaller than for non-reduced-order FDMs i.e R, < K-1 || - (b)] with layers of type (1)$.
We also note that the reachable component Jo of the objective cannot always be minimized to regardless of the approximation power of n/. For each h < m, Sn discards m frequency
$ "See Fig. 4.1 and Appendix B for experimental evidence in support of this phenomenon.
Selected Modes Location
Reconstruction
T T(x)
MAE
Topa selec
3.0
15
Low Low-paoe pass fiterting
1.0
03
0.0
Figure 3.2:
Reconstructions after low-pass filtering (first modes) [Bottom] or top-m selection [Top] of ERAS (Hersbach al., 2020) climate data. The non-monotonic structure of the spectrum implies more accurate reconstructions can be obtained with top-m selection.

---

components of the imput signal which, if different than zero, likely contain the necessary information to approximate (X)) exactly. Specifically, the irreducible lower bound on Jo should depend on "how much" the output's s m frequency components depend on the discarded N - input's s elements.
A rough quantification of such bound be obtained by inspecting the mismatch between the gradients of 4/v Ye.k o Sm with respect to X. In particular, it holds
N (X) )ro,a(SMX) ox,
(x)x() a ax,
Ortia (X) ax,
Σ
Σ
ax,
0x;
Unless 0x, dh (X) 0 holds for all j m,...,,N of the ground truth map in k-space on the truncated elements, there will be an irreducible overall gradient mismatch and thus nonzero Ja.
N - and k = I, .... - i.. no dependency
3.2 Weight Initialization for Reduced-Order FDMs
FDMS (Li et al., 2020; Tran al., 2021; Wen al., 2022) opt for standard Xavier-like (Glorot and Bengio, 2010) initialization distribution that takes into account the input channels to a layer i.. NNO, 2) However, well-known variance preserving properties of Xavier schemes do not hold for FDM layers truncating N m elements of the k-space. Notably, Xavier schemes do not scale the variance of the weight initialization distribution based on the number of elements kept after truncation of the spectrum performed by fa, leading to the collapse of the outputs to zero.
To avoid this issue in T1 and other FDMs, develop simple variance- preseevinn (vp) that intro- duces a variance scaling factor based on m and the class of transform.
Theorem 3.2 (Variance Preserving (vp) Initialization). Let
W ST AS,. Wx a k-space
reduced-order layer and W is normalized DCT-II transform. If x: E R is random vector with
E[x] = O,
V[x] = 0²1l
Then,
N Au N (0. m2
V(z].
We report the proof in Appendix A, including some considerations for specific forms of Sg-
Corollary 3.2 (vp initialization for DFTs), Under the assumptions of Theorem 3.2, if W a nor- malized DFT matrix we have Re(Agg), Im((A))) ~ ~N(0,, 2m V[ğl V(z).
The collapse phenomenon is empirically shown in Fig. 3.3 for m = 24, comparing a single layer of FNO and FFNO (with Xavier initialization) with FNO equipped with the proposed vp scheme. Under the assumptions of Corollary 3.2, we sample and compute empirical variances of W ST ((0)Sm.. Wa for several finite batches of input signals x. We repeat the experiment for signals of different lengths N. The vp scheme preserves unitary variances whereas the other layers concentrate output variances towards zero a rate that grows with N m.
Variances of
Ours (DFT)
FNO (DFT)
FFNO (DFT)
21
32
61
128
0.5
1.5
2.5
0.5
1.5
2
0²
Figure 3.3: Output variance histogram in layer outputs è = W ST, A(B)S,, Wx. for finite sample of inputs and single sample of 0. Color indicates signal resolution.
When the learned frequency-domaii transformation fo is obtained, instead of the single low-rank linear layer fe Am as the composition of several layers, preserving variances can be achieved by applying the vp scheme only to the first layer. For some variants of FDMs e.g. FNO that truncate the spectrum at each layer, vp initialization should instead be applied to all.
7

---

Experiments
We validate T1 on learning to approximate solution operators dynamical systems from images.
In $ 4.1, we apply T1 on the standard task of learning solution operators for incompressible Navier-Stokes, comparing against other FDMs. In $ 4.1.1 perform series of ablation exper- iments on each ingredient the Ti recipe, including weight initialization and architecture. In 4..1.2 we provide scaling laws.
In $ 4.2 we deal with fluid-solid interaction dynamics in the form of higher resolution images (128). We consider turbulent flows around varying airfoil geometries, benchmarking against current SOTA (Thuerey et al., 2020).
In $ 4.3 we show how the computational efficiency of T1 allows learning on unwieldy data with- downsampling building low-resolution meshes. We consider learning on high-resolution video (600 x 1062) capturing the turbulent dynamics of smoke (Eckert et al., 2019).
Configuration and model details reported in the supplementary material. The code is available at https: / Mim//DfffEMML/kaal Weights & Biases (wandb) (Biewald, 2020) logs of results are provided.
4.1
Incompressible Navier-Stokes
We show that matches or outperforms SOTA FDMs with less computation on the standard incom- pressible Navier-Stokes benchmark. Losses reported in n-space (signal space) for comparison.
Setup We consider two-dimensional Navier-Stokes equations for incompressible fluid vorticity form as described in (Li et al., 2020). Given dataset of initial conditions, we train all models to approximate solution operator at time 50 seconds for high viscosity (u = le 1e-3) and time 15 for lower viscosity (u = 1e- As metric, we report normalized mean squared error (N-MSE). Both initial condition as well as solution are provided images of resolution 64.
We include as baseline established FDMs, such as Fourier Neural Operators (FNOs) (Li et al., 2020) and Factorized Fourier Neural Operators (FFNOs) (Tran et al., 2021). We indicate with the suffix vp models that employ the proposed variance preserving initialization scheme. All models truncate m 24, except FENOS m
Ground Truth
Ti-yp
PNOvp
PENO
Ground Touh
7itup
FNOxp
FENO
log YY
YY
|8
10
Figure 4.1: [Left] Direct predictions at T 50s on high viscosity Navier-Stokes. [Right] Ground-truth spectrum and absolute emror in k-space (DCT-II). Despite predicting only the first 24 elements, reduced- order T1 models produce smaller emors in other regions of the k-space.
Results We perform 20 training for each model and report mean and standard deviation in
Table 4.1. T1 reduces solution error w.r.t FNOs by over 20% and FFNOs by over 40%. A single for- ward pass of T1 models is average 2x faster than FNO and 10x than FFNOs. We note that FFNOs
Method
Param. (M)
Size (MB)
Step (ms)
high v/
low и
FFNO (Tran et al., 2021)
8.9
35
294
0.99710.003
1.01610.010
FNO (Li et al., 2020)
14.2
56
31
0.37910.006
0.32810.004
FNOvp T1+vp
14.2
56
32
0.35110.000
0.31510.006
10.2
40
19
0.257+0.007
0.240
Benchmarks on incompressible Navier-Stokes. Direct long-range prediction emors (N-MSE) in n-space (signal space) of different models.
Table 4.1:

---

are designed to share parameters between layers and thus require deeper architectures and slower, due more transforms. In particular, training time (500 epochs) for T1 is cut to 20 minutes down from 40 of FNOs, matching the model speedup. Finally, we report improvement in performance for FNOs with parameters initialized following our proposed scheme (FNOvp). Fig. 4.1 provides sample predictions in n-space (left) to contextualize the task, in addition to prediction errors in fre- quency domain (right). Despite being a reduced order model with m 24, T1+vp produces smaller emrors truncated k-space elements (k m) compared to FNOvp and FFNO.
4.1.1 Ablations on weight scheme and architecture
We repeat the previous experiment and report prediction errors for four variants of T1: same architecture and weight initializa- tion scheme as FNOs (T1), T1 with our proposed vp scheme (Tivp), reduced-order variant with k-space model fo defined as a UNet architecture (T1+), and T1+ with variance preserv- ing scheme (T1+vp). The results in Table 4.2 provide empir- ical evidence in support of the vp scheme and its synergistic effect with the proposed architecture. In particular, combining vp scheme and UNet structure in frequency domain reduces error by half compared to the naive T1 approach.
Method
high
low F
0.491
0.449
Tivp
0.304
0.280
T1+
0.295
0.260
Ti+vp 0.257
0.240
Table 4.2:
Ablation on the effect scheme and T1 architecture.
of the proposed weight initialization
4.1.2 Scaling laws
Data Scaling Laws
We verify whether the reduction in predictive error of T1 over neural operator baselines is preserved as the size of training dataset grows. We perform 10 training runs 1e - experiment, each time with a larger dataset size, and report the scaling laws in Fig. 4.2. With additional data, the gaps in errors nar- row slightly, with noticeable improvements obtained by applying the vp scheme to both FNO and T1+.
FNO
Test N-MSE
0.3
FNOvp
T1
0.25
Ti+vp
on the Navier-Stokes
0.2
0.15
230
212
213
Dataset size
4.2 Flow Around Airfoils
Figure 4.2: Scaling laws for N-MSE,
We investigate the performance of T1 in predicting steady-state solutions of flow around airfoils.
Setup We data introduced in (Thuerey al., 2020) in the form of 10000 training pairs of initial conditions, specifying freestream velocities and the airfoil mask, with the target steady-state velocity and pressure fields. This task introduces additional complexity in the form of higher resolution input images (128) and a full k-space due to the discontinuity in the field produced by the mask.
We compare SOTA UNet architecture (DFPNet) introduced by (Thuerey al., 2020) FNOs and T1 with vp initialization schemes. We perform a search on the most representative hyperparameters (detailed in the Appendix). Averages for 5 runs are reported in Table 4.3.
Results All models are able to accurately predict steady-state solutions for different airfoils with small normalized errors. Test N-MSE is comparable as all models are within single standard deviation. Train- ing of T1 is as fast as DFPNets (Thuerey et al., 2020) and accurate as FNOs, evidence of the applicabil- ity of T1 to tasks with signals that are not band-limited (in this case due the airfoil mask).
Method
N-MSE
Time (hrs)
DFPNet
0.023
1.3
FNO
0.020
6.0
Ti+vp 0.024
1.3
Table 4.3: Test N-MSE and total training time on the flow around airfoil task.

---

4.3
Turbulent Smoke
We investigate the performance of predicting iterative rollouts from high-resolution video of real rising smoke plumes.
Ground Truth
Ground Truth
T1+
FNO
Density
0.1
0.2
046 0.5
0.6
-6
-5
-2
Figure 4.3: [[eft]] 10-step rollout predictions on ScalarFlow. FNOs produce nigh-frequency, non-physical artifacts and accumulate eπror more rapidly in time compared to models [Right] Log-absolute values of predictions in k-space (DCT-II). Although T1 is limited to m = 512 and T1+ to m 224 k-space elements, the predictions are overall more physically accurate in re-saace.
Setup We use the ScalarFlow dataset introduced in (Eckert et al., 2019) consisting of 104 quences of 150 frames each collected from video recordings of rising hot smoke plumes. The dataset consists of raw video data high-resolution (600 x 1062) collected at 60 fps. This task scales up complexity by involving real-world high-definition data, capturing highly-turbulent dynamics. We perform rollouts iteratively based on previous predictions: all models are trained 3-step rollouts and evaluated over 10-steps extrapolation to test their generalization in time, We compare FNOs against T1+ and T1+vp of similar model sizes after performing search most representative hyperparameters (Appendix B).
Results Fig. 4.3 provides sample rollout of different model predictions in s-spa (DCT-II). T1+vp accumu- lates smaller errors over the rollout and is less prone to generating non- non-phyn oonyhisical artifacts by performing predic- tion only on subset of the k-space (Table 4.4). No- tably, T1 and T1+ are 4x to 7x faster, providing duction training time from 32.4 hours to 4.7. Ap- pendix B includes additional visualizations, including averaged prediction errors on s-space.
Method
N-MSE
Time (hrs)
FNO
0.232
32.4
T1
0,239
8.1
T1+
0.256
4.7
T1+vp 0.228
4.7
Table 4.4:
Test 10-steps rollout n-space pre- diction errors (N-MSE) and total training time on the ScalarFlow dataset.
Conclusion
We present streamlined class of frequency domain models (FDM): Transform Once (T1).
models are optimized directly in frequency domain, after single transform, and achieve similar or improved predictive performance at a fraction the computational cost (3x to 10x speedups across tasks). Further, a simple truncation-awant weight initialization scheme is introduced and shown to improve the performance of T1 and existing FDMs.
Acknowledgments
This work is supported by NSF (1651565). AFOSR (FA95501910024), ARO (W911NF-21-1-0125)) ONR, DOF, CZ Biohub, Sloan Fellowship and JSPS Kakenhi (21J14546).
References
N. Ahmed, T. Natarajan, and K. R. Rao. Discrete cosine transform. IEEE Iransactions Comput- ers, 100(1):90-92 1974.
Berto, Massaroli, M. Poli, and J. Park. Neural solvers for fast and accurate numerical optimal control. In International Conference on Learning Representations, 2022.
10

---

L. Biewald. Experiment tracking with weights and biases, 2020. URL httpt:h/twww/ wandb. com/. Software available from wandb.com.
W. Chen, Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen. Compressing convolutional neural networks in the frequency domain. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1475-1484, 2016.
L. Chi, B. Jiang, and Mu, Fast fourier convolution. Advances in Neural Information Processing Systems, 33:4479-4488, 2020.
R. S. Dumont and P. Brumer. Characteristics of power spectra for regular and chaotic systems. The Journal of chemical physics, 88(3):1481-1496 1988.
E. Dupont, A. Goliñski, M. Alizadeh, Y. W. Tch, and A. Doucet. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.03122 2021.
M.-L. Eckert, K. Um, and N. Thuerey. Scalarflow: large-scale volumetric data set of real- world scalar transport flows for computer animation and machine learning. ACM Transactions
Graphics (TOG), 38(6):1-16, 2019.
W. Falcon et al. Pytorch lightning. Gitllub. Note: htlps://github. comfP-ytorcdLiignnnnnnptottttt lightming, 3:6, 2019.
R. Feynman. The Character of Physical Law, with new foreword. MIT press, 1965.
K. Fukushima and S. Miyake. Neocognitron: A self-organizing neural network model for mecha- nism of visual pattern recognition. In Competition and cooperation in neural nets, pages 267-285. Springer, 1982.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statisties, pages 249 256. JMLR Workshop and Conference Proceedings, 2010.
J. Guibas, M. Mardani, Z. Li, A. Tao, A. Anandkumar, and B. Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111. 2021.
G. Gupta, X. Xiao, and Bogdan. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems, 34, 2021.
C. R. Harris, K. J. Millman, J. Van Der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, Taylor, S. Berg, J. J. Smith, et al. Array programming with numpy. Nature, 585(7825):357- 362, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08413. 2016.
H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers, et al. The era5 global reanalysis. Quarterly Journal of the 1999-20999 2020.
Royal Meteorological Society, (46(730):1999--1
A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ioneseu, D. Ding, S. Koppula, D. Zoran,
A. Brock, E. Shelhamer, al. Perceiver io: general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 14795, 2021.
H. Jasak, A. Jemoov, Z. Tukovic, et al. Openfoam: A c++ library for complex physics simulations. In International workshop on coupled methods numerical dynamics, volume 1000, pages 1-20. IUC Dubrovnik Croatia, 2007.
G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
D. Kochkov, J. A. Smith, A. Alieva, Q. Wang, M. P. Brenner, and S. Hoyer. Machine learning- accelerated computational fluid dynamics. Proceedings of the National Acudemy of Sciences, 118 (21), 2021.
N. Kovachki, Lanthaler, and S. Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, Art 2021.
11

---

Z. Li, N. Kovachki. K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anand- kumar. Fourier neural operator for parametric partial differential equations. arXiv preprint WwXi:20010.08855, 2020.
M. Lienen and S. Günnemann. Learning the dynamics of physical systems from sparse observations with finite element networks. arXiv preprint arXiv:2203. 08882,, 2022.
Z. Long, Y. Lu, X. Ma, and B. Dong. PDE-net: Learning PDEs from data. In Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3208-3216. PMLR, 10-15 Jul 2018. URL https://proceeddi .ml.nnssssrrr.....mm........eeee mlr. ppress/v00/ooss 8a. html.
L. Lu, P. Jin, and G. E. Karniadakis. Deeponet: Learning nonlinear operators for identifying dif- ferential equations based on the universal approximation theorem of operators. arXiv preprint 1rX,v:9910003193, 2019.
J. Makhoul. A fast cosine transform in one and two dimensions. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(1):77-34, 1980.
Mathiesen, B. Yang. and Hu. Hyperverlet: symplectic hypersolver for hamiltonian systems. In AAAI, 2022.
M. Mathieu, M. Henaff, and Y. LeCun. Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.588 2013.
A. V. Oppenheim. Discrete-time signal processing. Pearson Education India, 1999.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv: /1214, 2022.
T. Pfaff, M. Fortunato, A. Sannhez-Goonzlez, and P. W. Battaglia. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
M. Poli, S. Massaroli, A. Yamashita, H. Asama, and J. Park. Hypersolvers: Toward fast continuous- depth models. Advances in Neural Information Processing Systems, 33:21 10522107-1117, 2020.
M. Poli, W. Xu, S. Massaroli, C. Meng, K. Kim, and S. Ermon. Self-similarity priors: Neural collages differentiable fractal representations. arXiv preprint arXiv:2204.07673, 2022.
H. Pratt, B. Williams, F. Coenen, and Y. Zheng. Fenm: Fourier convolutional neural networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 786 798. Springer, 2017.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics- innsics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-7011 2019.
O. Rippel, J. Snoek, and R. P. Adams. Spectral representations for convolutional neural networks. Advances in neural information processing systems, 28, 2015.
O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241 Springer, 2015.
V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462- 7473, 2020.
Y. Soon, S. N. Koh, and C. K. Yeo. Noisy speech enhancement using discrete cosine transform. Speech communication, 24(3):249-255 1998.
E. M. Stein and R. Shakarchi. Fourier analysis: introduction, volume 1. Princeton University Press, 2011.
G. Strang. The discrete cosine transform. SIAM review, 41(1):135-147, 1999.
H. Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018.
12

---

N. Thuerey, K. Weißenow, L. Prantl, and X. Hu. Deep learning methods for reynolds averagee navier-stokes simulations of airfoil flows. AIAA Journal, 58(1):25-36, 2020.
A. Tran, A. Mathews, L. Xie, and C. S. Ong. Factorized fourier neural operators. arXiv preprini arXiv:2111.13802 2021.
L. N. Trefethen. Approximation Theory and Approximation Practice, Extended Edition. SIAM, 2019.
L. Verlet. Computer" experiments" on classical fluids. i. thermodynamical properties of lennard- jones molecules. Physical review, (5)(1), 1967.
R. Wang, K. Kashinath, M. Mustafa, A. Albert, and Yu. Towards physics-informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1457-1466, 2020.
G. Wen, Z. Li, K. Azizzadenesheli, A. Anandkumar, and S. M. Benson. U-fno an enhanced fourier neural operator- bease deep-learning model for multiphase flow. Advances Water Resources, page 104180, 2022.
I. P. Yaroslavsky. Fast transforms in image processing: compression, restoration, and resampling. Advances in Electrical Engineering, 2014, 2014.
13


Reviews:
[{"review_id": "Tjxjl8sZDHE", "reviewer": "Reviewer_42JU", "summary": "The submission develops a new model architecture under the umbrella of _frequency-domain models_. These models are used to learn reduced order models of dynamical systems. The main development is to apply an integral transform (such as an FFT or DCT) once to the signal and then apply a chain of non-linear operations solely in the spectral domain before transforming back. This saves a lot of computation moving back and forth to the frequency domain at every layer.\n\nThe authors test their model on a number of dynamical systems modelling problems, showing superior performance in terms of normalized mean squared error and wall clock time. The paper is well written and I support acceptance.", "questions": "20-21: The original quote is from Feynman (1965)\n\n39: What is meant be a \"non-trivial number of channels\"? Also, please provide an actual example of how you arrive at hundreds of millions of parameters? If you bandlimit the signals in the frequency domain, as performed in the original Fourier Neural Operator paper, I think the number of learnable parameters is significantly lower than this.\n\nSec 2.1: Regarding learning in the frequency domain, consider also the line of works in the equivariance literature, where the irreducible representations are used for filters.\n\n89: I would be careful not to use the notation $\\mathcal{D}_n$ and $\\mathcal{D}_k$ for signal space and frequency space. It is very easy to assume that these are the same kind of space, just with a different dimensionality, when, for example, one may be a real space and the other a complex one.\n\n100: For the Fourier neural operator, is $g$ not purely local; that is, it is a 1x1 filter?\n\n110: How is the complex-valuedness of the layers related to parameter--inefficiency? Since signals are real, there is conjugate symmetry in the frequency domain, so while needing two real parameters to represent each complex parameter, we only need half the filter taps anyway.\n\n- Please use equation numbers for all display-mode equations\n\n186: What is the necessity for measuring the high frequency loss if your task is not reconstruction? $R_o$ will be non-zero, but I do not see why reconstruction error is a good measure of performance unless a downstream task is specified.\n\n- One of the considerations for using FNO is that convolution with large kernels in the spatial domain becomes more efficient in the spectral domain, since the FFT diagonalizes convolution. With bandlimiting of the spectral representation of the kernel, this becomes very efficient. In this work, $f_\\theta$ is left free-form and thus while adding capacity, does not lead to a computational complexity reduction necessarily. I'm guessing the main complexity reduction is in the fact that you do not need to transform back and forth from the spectral domain with every layer?\n\n- Is the use of N-MSE the best performance metric for rollout quality? In Fig 4.3 it appears the T1+ loses a lot of fidelity compared to the FNO model baseline. This would not be captured by N-MSE, but by a spectral metric.\n\n- Do this model exhibit generalization over resolution, like the original FNO models? It does not appear to me that this would happen.", "limitations": "The authors do not address limitations of their work. This should be fixed if the draft is to published. Conversations about resolution independence, choice of integral transform, and restrictions to regular grids would be welcomed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ 153: I like the insight that $\\mathcal{T}$ can be amortized across training epochs.\n\n+ Section 3.3: The weight initialization scheme for the reduce-order FDMs looks sound. It is nice that this is shown in the main body of the submission.\n\n+ The experimental results look good. The experiments themselves appear to have been conducted properly (+1 for using multiple training runs).\n\n+ I like the idea of transforming once to the spectral domain and then performing all computations there before transforming back to the spatial domain. It saves computation and thus allows one to deploy much more expressive models in the spectral domain, which is exactly what this submission does (they use a U-Net).\n\n+ The submission is well written and nicely typeset.\n\n- The authors have not included limitations of their work.\n\n- Some of the claims about baselines (see questions) are a bit exaggerated---for instance, FNO does not use hundreds of millions of parameters (see Table 4.1) and these shown be toned down.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "NiMUCauqm1f", "reviewer": "Reviewer_kKUQ", "summary": "The authors consider the problem of learning in a frequency domain. They note that multiple Fourier domain layers that perform forward and inverse FFT operations can lead to a high computational cost. They also argue that complex valued layers are inefficient, and propose to use a single forward and inverse DCT throughout the architecture. They also discuss how to initialize such a system.", "questions": "- line 94 : Should it be $\\mathbb{R}^{m\\times n}$ instead of $\\mathbb{R}^{n\\times m}$?\n- in eqn 3 : Is $m$ an optimization parameter -- that is, is $m \\in \\theta$?\n- the eqn between lines 186, 187 : if $m \\notin \\theta$, minimizers of $J_{\\theta}$ and $L_{\\theta}$ are the same. $L_{\\theta}$ is useful only if you want to compare losses across different values. Is that what you're doing? Are you optimizing over $m$ as well?\n- Thm 3.2, line 216, please replace \"Then\" with \"If\", and \"$\\Rightarrow$\" with \"then\". Same comment applies to Corollary 3.1. The statement looks a bit confusing otherwise.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Fourier domain learning is a relatively young but important subfield for machine learning. Therefore the subject is of critical interest. The writing is also clear for most of the manuscript.\n\nHowever, even though the subject is significant, the contribution to the state of the art is modest. Using a DCT, or any other real-valued harmonic transform instead of the FT is fine, but I'm not really convinced of the argument against using complex valued transforms. If one knows that the signal is real-valued, one can make use of symmetry properties of the FT to avoid any redundant operations. Resorting to the Parseval relation to observe that the error is the same in the transform domain is also perfectly fine, but is a straightforward observation. This leaves a theorem about initialization, as the main contribution. This contribution is welcome, but I think that alone is not sufficient for a Neurips paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8u6NxyMga-4", "reviewer": "Reviewer_Q4Qr", "summary": "This paper proposes a frequency domain approach for deep learning that is very intuitively summarized in the title: the goal is to have a single transformation into the frequency domain, perform all necessary operations there, and then transform back. While not an overly expensive operation, it is an interesting goal to reduce the number of frequency domain operations. The paper arrives at this goal via truncation and a DCT.\n\nThe paper is very nicely written, and motivates the approach well. Frequency domain models (FDMs) are an active topic of research, and the paper thoroughly compares to popular variants, the \"fourier neural operators\" and a factorized version. It evaluates the proposed approach with a set of flow-centric set examples. This limits the scope somewhat, but I personally found the examples very interesting, and I believe they cover a nice variety of interesting structures for learning problems.\n\nThe paper also comes with some theoretical results. They look convincing, but I have to admit I didnt check the proofs in detail.\n", "questions": "One open point I noticed was the introduction and later on omission of the T1+vp scheme. Section 4.1.1 first discussed its clear advantages, but the more complex examples in 4.2 and 4.3 seem to only use T1+. Can the authors explain this?\n", "limitations": "As mentioned above, I think the low-pass filtering due to the frequency truncation should be discussed more explicitly.\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In terms of main strengths, I think the paper proposes a very attractive idea. The presentation, derivation and discussion are great, I believe. The paper is clear, and very nice to read. The results and presented in a compelling way, and provide a detailed picture of the proposed approach. I think this is a clear strength of the paper.\n\nThe advantages of the resulting algorithm are probably a bit less than what one would hope for given the initial discussion of positive properties of the single transformation, but I think they're nonetheless substantial. I definitely found the results in terms of MSE errors and resource usage to be attractive steps forward.\n\nOn the side of weaknesses, the results can nonetheless be seen as somewhat underwhelming at times. I think especially the strong low-pass filtering of the frequency truncation is a central limitation of the approach. This is, e.g., visible in the turbulent smoke outputs (fig. 4.3). For this setup, more examples should be provided in the appendix, and I think the implications of inherent low-pass filtering (and the corresponding smoothing effects) should be discussed in the main paper. I hope the authors can also comment on this in the rebuttal, but for now I'm sure the authors could address this in a finalized version of their submission.\n\nOverall, I would argue for accepting this paper to NeurIPS. It proposes an interesting variant of FDMs, thoroughly evaluates and discusses it. Even if some of the results \"only\" result in smoother outputs with similar MSE and faster runtime, I think the approach is suitable for presentation at NeurIPS as a variant that is highly attractive for some learning scenarios.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-E68bTjZON", "reviewer": "Reviewer_Pxxo", "summary": "This paper proposes an efficient operator learning scheme by transforming the input data only once to some frequency domain. Numerical results on turbulence simulation indicate its significant speedup and accuracy improvement in several benchmark problems. ", "questions": "How do you define the architecture of the proposed model in (3)? Also it is not very clear how to define such an architecture which respects symmetry properties of the data, e.g. translation invariance in Navier-Stokes equations. \n\nWhat are the operations f_1,…f_d in T1(x)? When you compute the speedup, are you computing the same function defined by the FNO? It seems to me there is some issue regarding the speedup results in Fig 3.1, if they were convolutional (e.g. in Unets) as it depends on the filter size. \n\nWhat is the definition of N-MSE in Table 4.1?\n\nWhen you talk about the sparsity based on Fourier transforms in the introduction, are you considering turbulence as natural signals? Are they really sparse under DCT transforms? \n", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of computing convolution in the Fourier domain was proposed to speed-up the computational time with respect to the Mathieu et al. 2013, this paper shows that there is some overhead by doing so for each convolution operation: it is more direct to learn some equivalent function in the DCT Frequency domain (section 3.1), without using the common FDM layer structure (eq. 1). The strength seems to be its speedup. A reduced order T1 model is then proposed in 3.2 to further restrict the frequency domain. This seems to be a weakness as one may want to model some energy cascade phenomena in turbulence which would require the whole frequency information in the input signal. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept