WaveBench: Benchmarking Data-driven Solvers for Linear Wave Propagation PDEs
Authors: Tianlin Liu
Jose Antonio Lara Benitez
Florian Faucher
AmirEhsan Khorashadizadeh
Maarten V. de Hoop
Ivan Dokmanić
Venue: Accepted by TMLR ()

Abstract: Wave-based imaging techniques play a critical role in diverse scientific, medical, and industrial endeavors, from discovering hidden structures beneath the Earth's surface to ultrasound diagnostics. They rely on accurate solutions to the forward and inverse problems for partial differential equations (PDEs) that govern wave propagation. Surrogate PDE solvers based on machine learning emerged as an effective approach to computing the solutions more efficiently than via classical numerical schemes. However, existing datasets for PDE surrogates offer only limited coverage of the wave propagation phenomenon. In this paper, we present WaveBench, a comprehensive collection of benchmark datasets for wave propagation PDEs. WaveBench (1) contains 24 datasets that cover a wide range of forward and inverse problems for time-harmonic and time-varying wave phenomena; (2) includes a user-friendly PyTorch environment for comparing learning-based methods; and (3) comprises reference performance and model checkpoints of popular PDE surrogates such as Fourier neural operators and U-Nets. Our evaluation on WaveBench demonstrates the impressive performance of PDE surrogates on in-distribution samples, while simultaneously unveiling their limitations on out-of-distribution samples, indicating room for future improvements. We anticipate that WaveBench will stimulate the development of accurate wave-based imaging techniques through machine learning.

Published in Transactions on Machine Learning Research (02/2024)
WaveBench: Benchmarking Data-driven Solvers for Linear Wave Propagation PDEs
Tianlin Liu" University of Bosel
Jose Antonio Lara Benitez' Rice University
Florian Faucher" Team Makulu, Inria, University of Pau and Pays de I I'Adour, TolalEnergies, CNRS, France
AmirEhsan Khorashadizadeh University of Basel
Maarten V. de Hoop Rice University
Ivan Dokmanié University of Basel
"These authors contributed equally to the work.
Reviewed on OpenReview: https: // openreview. net/ forum? id= 6up Inumas8
Abstract
Wavo-based imaging techniques play critical role in diverse scientific, medical, and indus- trial endeavors, from discovering hidden structures beneath the Earth's surface to ultrasound diagnostics. They rely on accurate solutions to the forward and inverse problems for par- tial differential equations (PDEs) that govern wave propagation. Surrogate PDE solvers based on machine learning emerged as an effective approach to computing the solutions more efficiently than via classical numerical schemes. However, existing datasets for PDE surrogates offer only limited coverage of the wave propagation phenomenon. In this paper, present WAVEBENCH, comprehensive collection of benchmark datasets for wave prop- agation PDEs. WAVEBENCH (1) contains 24 datasets that cover a wide range of forward and inverse problems for time-harmonic and time-varying wave phenomena in 2D; (2) in- cludes a user-friendly PyTTorch environment for comparing learning-bese based methods; and (3) comprises reference performance and model checkpoints of popular PDE surrogates such as U-Nets and Fourier neural operators. Our evaluation on WAVEBENCH demonstrates the improssive performance of PDE surrogates in-distribution samples, while simultancously unveiling their limitations oullof-distribution (OOD) samples. This 00D-generalization limitation is noteworthy, especially since we use stylized wavespeeds and provide abundant training data to PDE surrogates, We anticipate that WAVEBENCH will stimulate the devel- opment of accurate wave-based imaging techniques through machine learning.
1 Introduction
Waves behind imaging modalities diverse reflection seismology, medical ultrasound, and X-ray crystallography. Imaging with waves relies on mathematical models of wave propagation expressed through partial differential equations (PDEs) called wave equations.
Since conventional mumerical PDE solvers are computationally expensive for large-scale problems, recent rescarch has witnessed rapid emergence of machine learning- basod models approximate PDE solutions (Li et al., 2020: Nelsen & Stuart, 2021; Lu et al., 2019; Bhattacharya et al., 2021; Li al., 2021; Huang al., 2021; Wang et al., 2021; Gupta et al., 2021; de Hoop et al., 2022a; Kissas et al., 2022; Brandstetter et al.,

---

Published in Transactions on Machine Learning Research (02/2024)
2023). These models, collectively referred to PDE surrogates, are trained using dataset of ground-truth PDE solutions. Properly trained PDE surrogates can offer faster inference speeds compared to traditional numerical solvers while maintaining accuracy. Realizing this advantage, however, typically requires having access to high-quality PDE solutions for training purposes.
Indeed, is hard to overstate the significance of high-quality datasets in the success of modern machine learning models in computer vision and natural language processing. Similarly, in the case of PDE surrogates, the importance of high-quality datasets has become evident. Recent efforts focus constructing high-quality datasets for PDE surrogates (Lu et al., 2022; de Hoop al., 2022b; Gupta & Brandstetter, 2022; Takamoto et al., 2022). These datasets cover a wide range of equations, especially those related to fluid dynamics, such as Darey flow, shallow-water, and Navier Stokes equations (Li et al., 2021; Gupta & Brandstetter, 2022; Stachenfeld et al., 2022; Holl et al., 2020). However, a comprehensive dataset for a broad family of wave PDEs is still missing.
To address the gap, we present WAVEBENCH, an extensive collection of benchmark datasets designed for wave propagation PDEs. WAVEBENCH includes 24 datasets, encompassing two categories of forward and inverse problems of acoustic waves: time-harmonic problems and time-varying problems. These datasets are constructed using open-source software tools hawen (Faucher, 2021) for time-harmonic waves and j-wave (Stanziola et al., 2023) for time-varying waves. We have made these datasets publicly accessible for re- searchers to access. Moreover, we provide a Py'orrch (Paszke et al., 2019) environment that enables easy training and comparison between various PDE surrogate models. We also include checkpoints and reference results for popular PDE surrogates of Fourier neural operators (Li et al., 2021), U-Nets (Ronneberger al., 2015), and the U-shaped neural operator (Rahman et al., 2023). By providing these resources, we aim to foster the development of machine-learning techniques for wave imaging,
2 Background and related work
ML methods to approximate operators described by PDEs. The solution map of a generic PDE can be written as an operator G : A + Ut where A and U are normed function spaces defined on some bounded subsets in Rdd We focus on 2D domains, thus d = 2. For instance, G can be a mapping that converts PDE coefficients a E A into a PDE solution (a(a) E U. A PDE surrogate Go is a data-driven emulator of the true PDE solution map G with denoting the model parameters. A PDE surrogate is learned from training dataset of input-ouu output pairs {(a ( where aj E A and uj Gt (aj) are prepared using conventional mumerical solvers. The loss quantifying how well PDE surrogate with parameters / fits the data is formulated as
1 Σ Z (Colas))...
LN(0)= - N
where the error function 1(55)) is often chosen as the absolute error 1(ū, u) || u/lur or the relative error ((ū, u) = ||@ ullex ||ul|z (Kovachki et al., 2023). Once trained, the model Go yields approximate PDE solutions, often faster than traditional solvers.
PDEs surrogates. A PDE surrogate Ga can be parameterized in various ways. Popular choices include kernel-based models (Kadri et al., 2016; Gricbel & Rieger, 2017), random feature models (Nelsen & Stuart, 2021), Gaussian processes Chen el al. (2021); Harkonen et al. (2022); Henderson al. (2023), and neural networks (Lu et al., 2019; Kovachki et al., 2023; Bhattacharya et al., 2021; Wang et al., 2021; Kissas et al., 2022; Brandstetter et al., 2023). Among them, neural networks represent the current state-of-the-art in empirical performance. Recently, Lu et al. (2022); de Hoop et al. (2022b); Gupta & Brandstetter (2022); Takamoto et al. (2022) ran comprehensive numerical comparison of various models, showing their relative merits in different scemarios. For PDEs discretized on 2D grid meshes, Fourier neural operators (FNO) and U-Nets are highly performant (Takamoto et al., 2022; Gupta & Brandstetter, 2022); we thus adopt them as baselines.
Benchmarks and datasets for PDEs surrogates. Recent advancements in PDE surrogates have led to the development of standardized datasets that serve as benehmarks (Lu et al., 2019; Kothari et al., 2020;
2

---

Published in Transactions on Machine Learning Research (02/2024)
Li et al., 2021; Lu et al., 2022; de Hoop et al., 2022b; Gupta & Brandstetter, 2022; Takamoto et al., 2022; Benitez et al., 2023). These datasets comprise a wide range of PDFs, usually with particular emphasis on fluid dynamics problems such as Darcy flow, shallow-water, and Navier Stokes problems. Despite this impressive varicty, there is a noticeable gap in the coverage of wave propagation PDEs. While a few work apply PDE surrogates on wave problems (Kothari et al., 2020; de Hoop al., 2022b; Deng et al.; 2022; Benitez al., 2023), each focuses on specialized setting. For instance, Kothari al. (2020) and Deng et al. (2022) focus on certain time-varying wave problems, and Stanziola al. (2021); de Hoop et al. (2022b) and Benitez et al. (2023) concentrate on time-harmonic ones. In contrast, the proposed WAVEBENCH provides comprehensive range of wave propagation datasets by considering both time-varying and time-harmonic problems. Additionally, we cover a wide range of frequency configurations for time-harmonic problems and take different lypes of wave speeds into account for both time-varying and time-aarmnnic problems. To facilitate model comparisons, we also offer a unified PyTorch environment for comparing and benchmarking models.
3 WaveBench datasets
Our proposed WAVEBENCH consists of 24 datasets, divided into two problem categories: time-harmonic wave problems and lime-varying problems. The Lime-harmonic wave problem is further decomposed inlo two with the consideration of acoustic and elastic waves.
3.1 Datasets for Hime-harmonic wave problems
3.1.1 Time-harmonic acoustic waves.
We consider the propagation of time-harmonin acoustic waves (Martin, 2021; Colton et al., 1998; Faucher & Scherzer, 2020) written in terms of the pressure field p. Upon assuming medium with constant density, it results into considering the Helmholtz equation (e.g., Faucher (2017)):
-(At
w² c(x)}2
pla,,w) ((x,w),
(1)
where p w( is pressure field at the angular frequency w, is wavespeed function. Throughout this work, our domain is in 2D, that is, æ E R.2 For the source function f, we consider a delta-Dirac in space 5(y) where y is the position of the source. On the boundary of the domain, we follow a geophysical configuration (Benitez et al., 2023; Faucher & Scherzer, 2020): The boundary divided into two distinct parts. The upper boundary corresponds an interface I'1 where Dirichlet zero conditions (free surface) are enforced. In the remaining portion of the boundary, I2 we assume absorbing boundary condilions to prevent waves from reflecting back to the medium, Engquist & Majda (1977). We have,
p(x, w) 0, on F1 (free surface),
(2a)
dw c(x))
p(x,w) 0, on I2 (absorbing boundary conditions).
(2b)
Experimental setup We use the same domain size and a fixed point source (near surface) for all fre- quencies. To stay in the statistical learning setup, we randomly generate wavespeed as the composition of an affine transformation and Gaussian random field with the Whittle-Matén covariance as described Benitez al. (2023). Using the same notation as Benitez let the coefficients À (Ar, appear in Whittle-Matérn
(2023), we set the smoothness parameter of
the field to be for all the cases. Furthermore, covariance (Benitez et al., 2023) be as follows: (a) for the isotropic case choose À (0.1, 0.1), and (b) for the anisotropic case we have X (0.2, 0.5). We illustrate in Figure wavespeed and the corresponding pressure field
The domain is 2D having the size 1.27 1.27km2
Experiment config.
50 000 GRF wave speeds generated, imposing 1.5km 8 < c(x) < 5kms-1
(3)
The data are that solve (1) at frequency w/(2m) = 10, 15, 20 and 40 Hz.
3

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed C Isotropic GRF
Real parts of pressure fields p(,,w) w) at different frequencies w
10 Hz
15 Hz
20 Hz
40 Hz
0.25
0.2
0.0
0.0
0.00
0.0
-0.2
-0225
-0.2
-0.1
Anisotropic GRF
10 Hz
15 Hz
20 Hz
40 Hz
0.1
0.25
0.2
0.0
0.00
0.0
0.0
2
-0.1
-025
0.2
Figure I: Visualization of samples from the acoustic time-harmonic datasets. At a fixed frequency w, the solver to the Helmholtz equation (1) converts a wavespeed c to a pressure field p(:,w). Two wavespeed are shown in the left-most panels. They are rcalizations Gaussian random field (GRF) with an isotropic kernel and an anisotropic kernel, respectively. The right panels display ground-truth pressure field p(-,w) frequencies w/(2n) 10Hz, 15H2, 20Hz, and 40Hz.
At fixed frequency and source J, the parametric form of the Helmholtz equation describes operator Ghelm that maps the wavespeed to a pressure field
G : p(,w).
(4)
To estimate the operator Gheem from data, dataset contains paired wavespeed and pressure field ((g.P9)h;- We let the wavespeed be realizations of Gaussian random fields (GRF) with an isotropic kernel or an anisotropic kernel. For each wavespeed c, the corresponding ground truth pressure field is ob- lained by solving the PDE using a hybridizable discontinuous Galerkin (HDG) method (Faucher & Scherzer, 2020) implemented in hawen package (Faucher, 2021). We produce time- harmonic datasets corresponding types GRF wavespoeds (isotropic and anisotropic). and frequencics. Each dataset contains 49,000 training samples, 500 validation samples, and 500 test samples. For an overview of these time-harmonic datasets, see Table 1.
3.1.2 Time-harmone elastic waves.
Unlike the acoustic wave problem which models scalar pressure field, the elastic case works with the displacement vector field u, written in two dimensions ju,, uy] Following time-harmonic propagation, each component (u, uy) is vomplex-valued. Under elastic isotropy, the time-harmonic equation has the form, (Carcione, 2007; Faucher, 2017):
- -(())=p((m) v (>(x)V
u(x)) V((a))
V (u(=)) Vu((x) (Vu(z)) '1) g(x),
where w is the angular frequency and g the source. The isotropic elastic medium is characterized by its density p(r), and the Lamé parameters A(x) and μ(x) (in particular, μ(x) is the shear modulus). In elastic media, two body waves propagate, the P-wave (compressional or primary wave) and S-wave (shear or secondary wave), Carcione (2007). Fach wave associated with a wavespeed, respectively ca(2)) and c, which can be used to characterize the medium an alternative to the Lamé parameters, and are given by,
A(x) + 2p(xx)
[(xx)
cx(x) =
cx(2) = 1 p(z)
(6)
p(x)
4

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed e
Frequency w/2r
10 Hz
15 Hz
Isotropic GRF
20 Hz
40 Hz
Acoustic wave
10 Hz
15 Hz
Anisotropic GRF
20 Hz
40 Hz
10 Hz
15 Hz
Elastic wave
Anisotropic GRF
20 Hz
40 Hz
Table Summary of 12 tirm-narmonic datasets. Each dataset corresponds to governing time- harmonic wave equation (acoustic or elastic), a type of wavespeed (isotropic GRF anisotropic GRF) and frequency (10, 15, 20, or 40 Hz). Each dataset consists of 49,000 training samples, 500 validation samples, and 500 test samples. The roles of the wavespeed and frequency in time-harmont wave propagation can be seen in the Helmholtz equation (1).
Experimental sctup Throughout clastic wave experiments, we let P-wavespeods cp, the S- S-wavespeed c be realizations of anisotropic GRFs while the density is kept to p 1. More preciscly, we first generate cp as GRF with values between 2.5 and 5.5 km.s-1, we then generate a GRF function with values between 0.35 and 0.50 that we use as scaling to create C. = ccp. Therefore, Cp. This choice is motivated as Cp and Cs are physical properties of medium that are expected to contain the same geometry of structures (e-g,; Jr and P appear in both Cp and C in (6)). For boundary conditions, we follow the same configuration as for the acoustic case with absorbing boundary conditions on the lateral and bottom boundaries, and free-surface condition for the upper surface where (representing the interface between the ground and the air), see, e.g., Faucher (2017). We visualize the wavespeeds and the real parts of the displacement fields at different frequencies in Figure 2.
is randomly scaled version of
P-wavespeed
U field (real part) 10 Hz
U field (real part) 15 Hz
Ue field (real part) 20 Hz
U field (real part) 40 Hz
2
0.2
2
-2
-2
S-wavespeed
U field (real part)
U field (real part)
U field (real part) 20 Hz
U field (real part)
10 Hz
15 Hz
40 Hz
0.5
2
-1
2
-2
Figure 2: Visualization of samples from the elastic time-harmonic datasets. The wavespeeds are separated into P-waves and S-waves shown in the left-most. column. At a fixed frequency w, the solver converts wavespeed c to a displacement field u = [uz, uzJ. The right panels display these displacement fields frequencics w2xT 10Hz, 15Hz, 20Hz, and 40Hz.

---

Published in Transactions on Machine Learning Research (02/2024)
In the case of elastic propagation, for fixed frequency w and source f, the parametric form describes operator Ge that maps the P- and S-wavespeeds Cp and to displacement vector field u:
Ge (Cp. Co) u(:,w) [u>(,w), us(:,w)]
(7)
For the computation of G elst similar the acoustic case we use the open-source software hawen (Faucher, 2021). We list in Table 1 the frequencies used to generate the dataset.
3.2 Datasets for time-narying wave problems
We now turn to the governing equation for acoustic propagation in the time domain. The acoustic wave equation describes the evolution of pressure g(ar, 1) over time under the influence of the wavespeed
c c(x):
8 gg(xx1) Aq(x, t) c(x)²
(8)
012
This equation is subject to radiating boundary conditions detailed in Appendix A. The time-varying quan- tity (xx t) in (8) and the time-harmonic quantity p(æ, w) in (1) are related: under appropriate assumptions, qx t) can be written as an integration of p(æ, w) in the frequency domain. See Faucher (2017, Section 1.6) for a detailed derivation. We consider two problems that arise from the time-varying wave dynamics: the reverse time continuation and the inverse source problem.
Gaussian Lens wavespeed c
Isotropic GRF wavespeed c
Anisotropic GRF wavespeed c
3
2
1
Initial pressure q(: 0)
Final pressure q(- T)
Final pressure q(:,77 T)
Final pressure q(, T)
In-distri. sample
0.75
0.2
0.2
0.2
0.50
0.0
0.0
0.0
0.25
0.2
0.2
-0.2
0.00
OOD sample
2
1.00
0.5
0.75
0.5
0.50
0.0
0.00
0.0
0.25
005
-025
-0.5
0.00
Figure 3: Visualization of samples from the reverse time continuation (RTC) problems. With a fixed wavespeed c, the goal of RIC is map the final pressure q(. to the initial pressure g(-,)). The top-most panels depict three types of wavespeods pressures O). The remaining panels display the final pressures q(., T), which are influenced by both the initial pressure q(: 0) in its respective row and the wavespeed in its corresponding column.
The left-most panels depict two realizations of initial

---

Published in Transactions on Machine Learning Research (02/2024)
Reverse time continuation (RTC) problem. The goal of this problem is to determine the initial pressure of the wave equation (8) based on the final pressure. Let g(:; 0) be the initial pressure, which propagates over time span I to reach the final pressure q(,,"), Note that the final pressure ((= is influenced by both the initial pressure ((:0) and the wavespood c in (8). With wavespeod c fixed, the ground-truth operalor that solves the RTC lask represented by
Ghe : (lo (): g(::)-
(9)
We construct the RTC dataset in the form of (gi(t, (; 0)) with fixed propagation time T but different wavespeed types. Figure 3 illustrates samples from these datasets. We consider three types of wavespeed (top panels of Figure 3): Gaussian lens and realizations of isotropic and anisot Gaussian random fields. For configuring the initial pressure ((-,),, we use two approaches. In the first approach, we place boxes or thick lines of random sizes, orientations, and locations in the domain, following the approach in Kothari et al. (2020). Additionally, to evaluate the oustrf-diitti (OOD) generalization performance of models, we construct test datasets using MNIST images (LeCun et al., 1998) as the initial pressure. The two bottom rows of Figure display samples from both the in-distribution thick line initial pressure and the OOD MNIST pressure. Appendix A provides detailed configuration information about the RTC dalasets.
Gaussian Lens wavespeed e
Isotropic GRF
Anisotropic GRF wavespeed c
wavespeed c
3
3
2
2
2
1
1
1
0
0
0
Initial pressure g(::0 0)
Sensor record (a( ())ecSteT
Sensor record
Sensor record (g(2,, ()
[1(x, [z, 1). #CSACT
In-distri. sample
0.75
0.2
0.2
0.50
0.0
0.0
0.25
-.2
-02
0.00
OOD sample
2
0.5
0.75
0.25
0.25
0.50
0.0
0.00
0.00
0.25
0.25
-0.25
0.5
0.00
Figure Visualization of samples from the inverse source (IS) problems. a fixed wavespeed e, the goal of IS to map ()a,,)))eSsIET (pressure recorded at sensor locations S and time T) into the initial pressure q(-, 0). The top-most, panels depict three types of wavespeeds c. The left-most panels depict two realizations of initial pressures (). The remaining panels exhibit the sensor records [g(2r, 1))e which influenced by both the initial pressure q(s, 0) in its respective row and the wavespeed in its corresponding column.
Inverse source (IS) problem. In the previous RTC problem, we are given the final pressure q(: T) on the full domain at terminal time T. However, this assumption is often too strong in applications such seismic imaging, where the pressure can only be measured by sensors placed at some portions of the domain

---

Published in Transactions on Machine Learning Research (02/2024)
boundary. To address this limitation, we turn to the inverse source (IS) problem. The IS problem aims to predict the initial pressure g(:, 0) based on pressure measurements taken at certain boundary locations over a time span [0, []. Given a fixed wavespeed c, sensor locations S, and discrete time steps T, the ground-truth operator that solves the IS task can be expressed as:
G : [q(2, (iiett)lesseee + {).0).
(10)
This problem is illustrated in Figure 4. Similar to the RTC dataset, we create random thick lines and use them as initial pressure O). In Figure 4, we visualize samples from IS datasets. The sensor locations S are fixed to be the upper boundary of the domain. Further details on the configuration of the IS datasets are provided in Appendix A.
We summarize our 12 time-varying datasets of RTC and IS in Table 3 in Appendix A. Each dataset based upon the initial pressures of thick lines consists of 9,000 training samples, 500 validation samples, and 500 testing samples. Each dataset based upon the initial pressures of MNIST is for out-of-distribution testing purposes, comprising of 500 testing samples.
3.3 Dataset accessibility and format.
The datasets are in the beton format of FFCV (Leclere et al., 2023), which is open-source software that provides high-throughput data loading for model training. Our datasets are accessible on Zenodo (an open platform for datasets sharing): https: htt////noo..orrrrrrrr///////1111 sible through our GitHub httpp//iooi..///////h/ooo//////a
org/recordds8005545, and the benchmark code is acces- https:/
4 Wavebench Benchmarks
4.1
Baseline models
We provide reference implementations and benchmark performance of PDE surrogates trained on WAVEBENCI datasets. We focus on U-Nets and FNOs as PDE surrogates, which exhibit high performance across various PDE problems (Takamoto et al., 2022; Gupta & Brandstetter, 2022).
Model
parameters
Forward pass runtime
Backward pass runtime (s]
FNO-depth-4
4.2M
0.011
0.018
FNO-depth-8
8.4M
0.019
0.033
U-Net-ch-32
7.8M
0.006
0.012
U-Net-ch-64
31.0M
0.016
0.031
UNO-modes-12
10.1M
0.024
0.032
UNO-modes-16
17.9M
0.024
0.033
Comparison of baseline models. The baseline models include two variants of FNO (Li et al. 2021) and two variants of U-Net (Ronneberger et al., 2015). FNO-depth-4 and FNO-depth-8 are two FNO variants with of 4 or hidden Fourier layers. U-Net-ch-64 stands for the standard U-Net that has 64 channels in its first layer, and U-Net-ch-32 is a smaller variant with all convolutional channels halved. Both UNO- modes-12 and UNO-modes-11 variants have 3 scales that correspond to dommain-diecretization of 128 128, 64 x 64, and 32 32. The UNO-modes-12 model uses [12, 12,,633. Fourier modes for cach scale, while the UNO-modes- 16 uses [16, 8, 4] Fourier modes for each scale. The model runtime was assessed using batch (8, 1, 128, 128), consisting of 8 samples of 128 x 128 array. The benchmarking procedure involved initial 10 dry runs followed by 100 test runs conducted on an 11 GB NVIDIA GeForce RTX 2080 Ti GPU.
Table 2:
FNO. The Fourier neural operator (Li et al., 2021) represents one of the statc-of-the-art models for PDE data on regular grids. We use FNOs in 2D consisting of 64 hidden channels, 16 Fourier modes, and either or 8 Fourier layers as hidden layers; for the input lifting and output projection parts of FNOs, use
8

---

Published in Transactions on Machine Learning Research (02/2024)
MLPs that consisted of 1x1 convolution with 128 channels. The FNO variants with 4 and 8 hidden Fourier layers are referred to as FNO-depth-4 and FNO-depth--, whose parameter count and runtime are summarized in Table 2.
U-Net. The U-Net (Ronneberger et al., 2015) is a convolutional network originally developed for 2D image- to-image regression problems such as image segmentation. However, its versatility has led to its adoption in PDE learning tasks (Kothari al., 2020; Li et al., 2021; Gupta & Brandstetter, 2022; Chen & Thuerey, 2023), both as a component within larger architectures standalone model, U-Nets perform well in several PDE modeling tasks, sometimes matching or surpassing dedicated PDE surrogates such as FÑOs (Takamoto et al., 2022; Gupta & Brandstetter, 2022). In our baselines, we employ either the standard U-Net (Ronneberger et al., 2015), referred to as U-Net-ch-64 as it uses 64 channels in the first hidden layer, or a smaller variant with halved convolutional channels (referred to as U-Net-ch-32). Table 2 summarizes their parameter counts and runtime.
as standalone model. Remarkably, even
UNO. The UNO (U-shaped neural operator) (Rahman et al., 2023) combines U-Net and FNO design elements. Like U-Net, UNO encodes input into smaller domains and decodes to generate output greater domains. UNO's layers are parametrized with Fourier layers in a way identical to FNOs. We consider two UNO variants, UNO-modes-12 and UNO-modes-IIe 16. For both versions, we use 3 scales on domains 128 x 128, 64 x 64, and 32 x 32. UNO-modes-12 uses Fourier modes of [12, 6,3] for each scale, and UNO-modes- 16 with [16, 8, for each scale.
Training protocol. We trained and tested the baseline U-Net, FNO, and UNO models using the 20 datasets described in Section 3 and summarized in Table and Table 3 in the appendix. For all datasets, we trained all models for 50 epochs using the AdamW optimizer (Loshchilov & Hutter, 2019). The learning rates were initially set to 1e-3 and then annealed to 1e-5 using the cosine annealing (Toshchilov & Hutter, 2017). We employed the relative L2 loss for training and evaluation in all our problems, following the approach in Li et al. (2021); de Hoop et al. (2022b). Since all datasets were divided into training, validation, and test splits, we monitored the model's generalization performance on the validation split during training. We selected the model that performed the best on the validation split for the evaluation of the test split. All experiments were conducted on an 11 GB NVIDIA GelForce RTX 2080 Ti GPU.
In the following sections, we present empirical results visualizations and complete qualitative results are provided in Appendix A. Due to space limitations, we present results for PDE surrogates of FNOs and U-Nets here in the main text; UNO variant outcomes are in the appendix. Tn most experiments, UNOs outperform some U-Nets but not the best FNOs.
subset of WAVEBENCH datasets. Additional
4.2 Performance of baseline models on time-harmonic datasets
In-distribution performance. Figure 5 illustrates the performances of models applied to the time- harmonic dataset with isotropic GRF wavespeeds and frequency 40 Hz. All models are trained using the training split of the dataset; the figure presents results based on sample from the test split. All PDE surrogates produce visually comparable predictions. However, the error fields (the third row of Figure 5)
show that two FNO variants produce smaller errors in comparison to U-Nets.
Numerical values of the errors are provided in Table in the Appendix B.
Dut-of-distribution (OOD) performance. While FNOs and U-Nets produce both visually and mu- merically appealing results for in-distribution samples, their out-of-distributio (OOD) test results clearly indicate room for further improvements (Figure 6). In this experiment, we trained models on a time-harmone dataset with isolropic GRF wavespeed and evaluated them on the anisolropic version of the datasel. This setup presents a greater challenge, as models that can successfully solve presumably must capture a greater extent of the physics of wave propagation. The results demonstrate that PDE surrogates generally perform over times worse in OOD tests compared to in-distribution tests in the metric of relative L2 error (Table 5 in the Appendix B).
9

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.2
0.0
2
0.2
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
0.25
Relative error
0.002
0..01
Figure 5: In-distribution test performance of models on a acoustic time-harmonic dataset. The model trained and tested on splits of the acoustic hime-naroonic dataset with isotropic GRF wavespeed and frequency w/2T 40Hz. The input wavespeed and real part of the ground-truth pressure field are shown in the top panels, ef. Equation (1). The middle panels show the predictions from 4 different models. The bottom panels show the relative error (p(,.w) #(,,4) (Ip(:,))lzzz between the predicted pressure p and the ground-truth pressure
4.3 Performance of baseline models on time-harmonic datasets
Recall that WAVEBENCH contains two time-varying wave problems: reverse time continuation (RTC) and the inverse source problem (IS). Here in the main text, we report the model performances on the more challenging IS problem. Full results of both problems can be found in Appendix D.
Figure provides a visual performances comparison of PDE surrogates on the IS dataset that uses ispic Gaussian Random Field (GRF) wavespeeds. Both U-Nets and Fourier neural operators yield comparable rosults in-distribution samples. U-Nets slightly exceed FNOs in performance, though the margin is narrow (Table 7 in Appendix D). The outputs from both models within reasonable expectations.
However, when it comes to OOD samples, both models introduce artifacts, leading to inaccuracies in re- constructing the target. This can be seen in the third row of Figure 7. There, U-Nets generate outputs with box-like patterns, which are derived from the training data; FNOs struggle to accurately reproduce the smooth contour of the OOD target. It is worth noting that similar artifacts can be consistently observed across all time-varying datasets (Appendix D).
10

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
0.00
2
-0.25
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
-0.25
Relative error
0.02
0.01
Figure 6: OOD test performance of models on an acoustic time-harmonic dataset. Here, the model trained on the time-harmonic dataset with isotropic GRF wavespeed and frequency w/2n = 40Hz, but tested on the anisotropic version instead.
5 Discussion
This paper introduces WAVEBENCH, a comprehensive repository 24 benchmark datasets designed for wave PDEs. The datasets cover a broad range of wave-related problems, including time-dependenl prob- lems derived from the wave equation and time-harmonic problems derived from the Helmholtz equation. WAVeBENCH serves two purposes: providing a data source for machine learning for wave problems and providing user-friendly PyTorc environment for training and evaluating PDE surrogate models.
After evaluating PDE surrogates on the WAVEBENCH datasets, we have identified several recurring patterns. Firstly, across all datasets, U-Nets and FNOs showed remarkable capabilities in approximating wave prop- agation for in-distribution data. The larger models, such as ENO-depth-8 and U-Net-ch-66 -4,, outperformed their smaller counterparts like FNO-depth-4 and U-Net-ch-32. This shows the potential advantages em- ploying high-capacity models tackle challenging PDE problems, assuming the existence of abundant and high-quality PDE data.
However, the performance of PDE surrogates declines significantly when moving from in-distribution to out- of-distribution (OOD) samples. This implies that although these models learn statistical patterns from the data, they do not capture the wave-propagation physics present in the data. Given the importance of OOD generalization the practical use of wave- based techniques in exploratory scientific applications like seismic imaging and medical imaging, future PDE surrogates that have the capability to learn wave-propagation physics from data are highly sought-after.
Limitations. While WAVEBENCH contains a broad range of wave propagation PDE problems, we note that all WAVEBENCH datasets are presently simulated on 2D domains. A simulation of wave propagation within 3D domain could provide more realistic and challenging setting for applying PDE surrogates. At
11

---

Published in Transactions on Machine Learning Research (02/2024)
In-distri input
In-distri target
OOD input
OOD target
1.00
1.00
Ground truths
2
0.2
0.75
0.25
0.75
0.0
0.50
0.00
0.50
-0.
0.25
0.25
0.25
0.00
0.00
FNO depth-4
FNO depth-8 8
U-Net ch-32
U-Net ch-64
OOD prediction In-distri prediction
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
1.00
1.00
1.00
1.00
?
2
2
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
Figure 7: Test performance of models on the time-varying IS dataset with isotropic GRF wavespeed. The first shows the input and target samples of the IS dataset; they can be either in-distribution OOD, The second row shows the model predictions the in-distribution sample. The third row shows the model predictions on the OOD sample.
the same time, preparing such dataset is computationally much more demanding. We leave this aspect for future work.
Acknowledgement.
This work was supported by the European Rescarch Council Starting Grant SWING. Numerical experiments were partly performed at the sciCORE (http://scicore www.icorre.iba.....eeeee..o.aaa. wnw.as ch/) scientific computing center at the University of Basel. FF acknowledges funding by the European Research Council Starting with Grant 101116288 INCORWAVE. Views and opinions expressed are however those of the authors only and not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them.
12

---

Published in Transactions on Machine Learning Research (02/2024)
References
Jose Antonio Lara Benitez, Takashi Furuya, Florian Faucher, Xavier Tricoche, and Maarten V de Hoop. Fine- tuning neural-operator architectures for training and generalization. arXiv preprint arXiv:2301.11500
2023. (Cited page 3.)
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. Model reduction and neural networks for parametric PDEs. The SMAI journal of computational mathematics, 7:121 157, 2021. (Cited on pages and 2.)
Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh K Gupta. Clifford neural layers for PDE modeling. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023. (Cited on pages and 2.)
José M Carcione. Wave fields in real media: Wave propagation in anisotropic, anelastic, porous and electro- magnetic media. Elsevier, 2007. (Cited on page 4.)
Li-Wei Chen and Nils Thuorey. Towards high-aceuracy deep learning inference of compressible flows over aerofoils. Computers & Fluids, 250:105707, 2023. (Cited on page 9.)
Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M Stuart. Solving and learning nonlinear PDEs with Gaussian processes. Journal of Computational Physics, 447:110668, 2021. (Cited page 2.)
David Colton, Rainer Kress, and Rainer Kress. Inverse acouslic and elecIromagnelic scollering theory, volume 93, Springer, 1998. ISBN 978-1-4614-4942-3, doi:
(Cited on page 3.)
Maarten de Hoop, Matti Lassas, and Christopher A Wong. Deep learning architectures for nonlinear operator functions and nonlinear inverse problems. Mathematical Statistics and Learning, 4(1):1-86, 2022a. (Cited page 1.)
Maarten V. de Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M. Stuart. The cost-accuracy trade-off operator learning with neural networks. Journal of Machine Learning, 1(3):299 341, 2022b. (Cited on pages 2, 3, and 9.)
Chengyuan Deng, Shihang Feng, Hanchen Wang, Xitong Zhang, Peng Jin, Yinan Feng, Qili Zeng, Yinpeng Chen, and Youzuo Lin. OPENFWI: Large-scale multi-structural benchmark datasets for full waveform inversion. In Proceedings of the 35th Conference in Newral Information Processing Systems (NeurIPS), 2022. (Cited on page 3.)
Björn Engquist and Andrew Majda. Absorbing boundary conditions for numerical simulation of waves. Proceedings of the National Academy of Sciences, 74(5):1765 1766, 1977. (Cited on page 3.)
Florian Faucher. Contributions to Seismic Full Waveform Inversion for Time Hurmonic Wave Equations: Stability Estimates, Convergence Analysis, Numerical Experiments involving Large Scale Optimization Algorithms. Theses, Université de Pau et des Pays de I' Adouur, November 2017. (Cited on pages 3, 4, 5, and 6.)
Florian Faucher. hawen: time-harmonic wave modeling and inversion using hybridizable discontinuous Galerkin discrctization. Journal of Open Source Software, 6657):2699, 2021. (Cited on pages 2, 4, and 6.)
Florian Faucher and Otmar Scherzer. Adjoint-state mothod for hybridizable discontinuous Galerkin dis- cretization, application to the inverse acoustic wave problem. Computer Melhods in Applied Mechanics and Engineering, 372:113406, 2020. (Cited on pages and 4.)
Leon A. Gatys, Alexander S. Eoker, and Matthias Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414 2423, 2016. (Cited on page 38.)
Michael Griebel and Christian Rieger. Reproducing kernel Hilbert spaces for parametric partial differential equations. STAM/ASA Journal on Uncertainly Quantification, 5(1):111-13)) 2017. (Cited on page 2.)
13

---

Published in Transactions on Machine Learning Research (02/2024)
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- based operator learning for differential equa- tions. In Proceedings of the 35th conference of Neural Information. Processing Systems (NeurPPS). 2021.
(Cited on page 1.)
Jayesh K Gupta and Johannes Brandstetter. Towards mulliispatiooeempraa-saalle generalized PDE modeling. arXiv preprint ar arX:::2009.1000 2022. (Cited on pages 3, 8, and 9.)
Marc Harkonen, Markus Lange-Hegermanni and Bogdan Raita. Gaussian process priors for systems of linear partial differential equations with constant coefficients. In Proceedings of the 40th international conference on Machine learning (ICML), 2022. (Cited on page 2.)
Henderson, P Noble, and O Roustant. Covariance models and gaussian process regression for the wave equation. application to related inverse problems. Journal of Computational Physics, pp. 112519, 2023. (Cited on page 2.)
Philipp Holl, Vladlen Koltun, Kiwon Um, and Nils Thnerey. Phiflow: A differentiable PDE solving framework for deep learning via physical simulations. In NewrIPS workshop, volume 2, 2020. (Cited on page 2.)
Xiang Huang, Zhanhong Ye, Hongsheng Liu, Beiji Shi, Zidong Wang, Kang Yang, Yang Li, Bingya Weng, Min Wang, Haotian Chu, et al. Metooerto for solving parametric partial differential equations. Tn Proceedings of the 36th Conference on Newral Information Processing Systems (NeurIPS), 2021. (Cited
page 1.)
Hachem Kadri, Emmanuel Duflos, Philippe Preux, Stéphane Camu, Alain Rakotomamonjy, and Julien Au- diffren. Operator-valued kernels for learning from functional response data. Journal of Machine Learning Research, 17(20):1-55 54, 2016. (Cited on page 2.)
Georgios Kissas, Jacob Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George J Pappas, and Paris Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research, 23 (215):1-63, 2022. (Cited on pages and 2.)
Konik Kothari, Maarten de Hoop, and Ivan Dokmanić. Learning the geometry of wave-based imaging. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS), 33:8318-8329, 2020. (Cited on pages 2, 3, 7, 9, and 16.)
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89):1-97, 2023. (Cited on page 2.)
Guillaume Loclere, Andrew Tyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Mądry. FFCV: Accelerating training by removing data bottlenecks. In Proceedings of the 34th IEEE/CVE /CVF Con- ference on Computer Vision and Pottern Recognátion (CVPR), pp. 12011-12020, June 2023. (Cited on page 8.)
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. (Cited on page 7.)
Zongyi Li, Nikola Kovachki, Kanyar Azizzadeneshell, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Proceedings of the 34th Conference Neural Information Processing Systems (NeurIPS), 33, 2020. (Cited page 1.)
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In Pro- ceedings of the 9th International Conference on Learning Representations (ICLR), 2021. (Cited on pages 1,
2, 3, 8, and 9.)
Iyy Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proceedings to the 5th International Conference on Learning Representations (ICLR), 2017. (Cited page 9.)
14

---

Published in Transactions on Machine Learning Research (02/2024)
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings to the 7th Inter- national Conference on Learning Representations (TCI.R) 2019. (Cited page 9.)
Lu Lu, Pengzhan Jin, and George Em Karniadakis. DeepONet: Learning nonlinear operators for identi- fying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.0811
2019. (Cited on pages and 2.)
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based FATR data. Compuler Methods in Applied Mechanics and Engineering, 393:114778, 2022. (Cited pages 2 and 3.)
Paul A Martin. Time-Domain Scattering, volume 180. Cambridge University Press, 2021. doi: 10.1017/ 9781 108899066 (Cited on page 3.)
Nicholas H Nelsen and Andrew M Stuart. The random feature model for input-output maps between Banach spaces. SIAM Journal on Scientific Computing, 43(5):A3212 A3243, 2021. (Cited on pages 1 and 2.)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorrch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd conference on Neural Information Processing Systems (NeurIPS), pp. 8024-8035. Curran Associates, Inc., 2019. (Cited on page 2.)
Allan D Pierce. Acoustics: An introduction to ils plysical principles and applications. Springer, 2019. (Cited page 16.)
Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli. U-NO: U-shaped neural operators.
Transactions on Machine Learning Research, 2023. ISSN 2835-8856. (Cited on pages and 9.)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of the 18th International Conference on Medical Image Computing and Computer- Assisted Intervention (MICCAI), pp. 234 241.41. Springer, 2015. (Cited on pages 2, 8, and 9.)
Kim Stachenfeld, Drummond Buschman Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Sanchez-Gonzalez. Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanchez- Gonzalez. Learned simulators for turbulence. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022. (Cited on page 2.)
Antonio Stanziola, Simon R Arridge, Ben T Cox, and Bradley E Treeby. A Helmholtz equation solver using unsupervised learning: Application to transcranial ultrasound. Journal of Computational Physics, 441: 110430, 2021. (Cited on page 3.)
Antonio Stanziola, Simon R. Arridge, Ben T. Cox, and Bradley E. Treeby. j-Wave: An open-source differ- entiable wave simulator. SoftwareX, 22:101338, 2023. (Cited on pages 2 and 16.)
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pffüiee, and Mathias Niepert. PDEBench: An extensive benchmark for scientific machine learning. Proceedings of the 36th Conference on Neural Informalion Processing Systems (NeurIPS), 35:1596 1611, 2022. (Cited pages 2, 3, 8, and 9.)
Bradley E Treeby, Jiri Jaros, Alistair P Rendell, and BT Cox. Modeling nonlinear ultrasound propagation in heterogencous media with power law absorption using k-space pseudospectral method. The Journal of the Acoustical Society of America, 131(6):4324 4336, 2012. (Gited on page 16.)
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-infrrmed DeepONets. Science Advances, 7(40):cabi8605, 2021. (Cited on pages and 2.)
15

---

Published in Transactions on Machine Learning Research (02/2024)
WaveBench: Benchmark Datasets for Modeling Linear Wave Propagation PDEs Supplementary Material
A Details of the time-varying datasets
To simulate wave propagation for both Reverse Time Continuation (RTC) and Inverse Scattering (IS) prob- lems, we the open-source j-wave package (Stanziola et al., 2023). The j-wave package simulates the wave dynamics in (8) by an equivalent system of first-order equations (Treeby et al., 2012; Pierce, 2019):
Ou 1
Vq. Ōt bo
(momentum conservation)
0b
Ot bo V
u,
(mass conservation)
= bb.
(pressure-density relation)
where u = u(xx,) t) is called the acoustic particle velocity and by is ambient density, Radiating boundary conditions are enforced with a perfectly matched layer (PMT.), following the default setting of j-wave (Stanziola al., 2023).
In simulation, the domain is represented square grid, with dimensions of 1.024 km x 1.024 km discretized into 128 x 128 array. Recall that for the time-varying experiments, the wavespeed e can be a realization of an isotropic GRF, anisotropic GRF, or Gaussian lens. In the case of isotropic and anisotropic GRF, the wavespeeds are taken from the the wavespeed is a point mass situated at the grid coordinates of (50, 55) blurred by a Gaussian filter with a standard deviation 50 in both spatial directions. Across all types of wavespoods considered in our simulations, the minimum wavespeed is normalized to value of 1.4 km s-1, while the maximum wavespeed is normalized to kms-1. The propagation time for both the RTC and TS simulations set to be T 0.2
datasets; in the case of the Gaussian-lens wavespeed,
For both RTC and IS datasets, the initial pressure datasets q(- 0) can either be thick lines or MNIST images, represented by 128 x 128 arrays; see Figure 3 and Figure 4. The thick lines represent pressures that are used as in-distribution samples for training and evaluating the model. They are box-like patterns of random sizes, orientations, and locations in the domain, following the approach Kothari et al. (2020). Each sample contains 5 10 boxes, uniformly distributed. The box centroids are sampled on the discretized grid of the domain. Dimensions of boxes are sampled from uniform distributions: length from [50, 100), width from [20, 40), and orientation from [0, x]. The dataset consists of 9000 training samples, 500 validation samples, and 500 test samples. Additionally, there are 500 oul-of-distribution (OOD) MNIST pressure samples for testing.
The IS problem is more challenging than RTC. This is because in IS we only get to measure the wave pressure at the top of the domain. That is, the sensor locations in the measurements [ą(x,, ()]mES,ET7 correspond the topmost coordinates of the domain (excluding the size of PML). The time steps consist of 128 equidistant intervals within the range of [O, These settings result in the sensor record [q(xx, ()] in Figure 4. To make the IS problem more tractable to solve, we
having a square image-like appearance use the following way to prepare the initial pressure q(:, 0). We resize thick line and MNIST images that with an original size 128 x 128 into the size of 64 x 64 and put them on the top center of the domain. The remaining entries are filled with zeros. Consequently, all nonzero entries of the initial wave pressure g(, are concentrated in the top center of the domain. This configuration, with the sensors positioned at the top, allows for better reception of propagated waves and helps mitigate the ill- ill-posedness of the problem.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Problem
Wavespeed
Initial pressure g(: 0)
Thick lines
Gaussian lens
MNIST
Reverse time continuation (RTC)
Thick lines
Isotropic GRF
MNIST
Thick lines
Anisotropic GRF
MNIST
Thick lines
Gaussian lens
MNIST
Inverse source (IS)
Thick lines
Isotropic GRF
MNIST
Thick lines
Anisotropic GRF
MNIST
Table 3: Summary of the 12 time-varying datasets. Each datasel corresponds to specific problem types (reverse time continuation or inverse source), wavespeed variations (Gaussian lens, isotropic GRF, or anisotropic GRF), and initial pressure characteristics (thick lines or MNIST). The thick line initial pressure datasets consist of in-distribution samples: they contain 9000 training samples, 500 validation samples, and 500 testing samples. The MNIST pressure dataset consists of out-of-distribution (OOD) samples used exclusively for testing and comprises 500 samples.
17

---

B
Full experimental results of the acoustic time-harmonic datasets
Published in Transaction fonss on Machine Learning Research (02 22044)
Wavespeed c
Freq. w/2T
FNO-depth-4
FNO-depth-8
U-Net-ch-32
U-Net-ch-64
UNO-modes- 12
UNO-modes-16
10 Hz
0.063
0.040
0.073
0.063
0.064
0.054
15 Hz
0.093
0.057
0.116
0.087
0.106
0.081
Isotropic GRF
20 Hz
0.122
0.070
0.157
0.106
0,147
0.114
40 Hz
0.283
0.165
0.286
0.191
0.407
0.301
10 Hz
0.059
0.025
0.144
0.119
0.074
0.051
15 Hz
0.098
0.039
0.204
0.165
0.123
0.093
Anisotropic GRF
20 Hz 0.135
0..60
0.230
0.176
0.171
0.129
40 Hz 0.315
0.172
0.321
0.231
0.422
0.343
Table 4: In-distribution performance comparison of models the test. folds of the time-harmonic dalasets. The error metric the relative L2
lp 2/ /|plz between the ground-truth and prediction p.
Wavespoed c
Freq. w/2n
FNO-depth-4
FNO-depth-8
U-Net-ch-32
U-Net-ch-64
UNO-modes-12
UNO-modes-16
18
10 Hz
0.485
0.379
0.527
0.506
0.489
0.458
15 Hz
0.633
0,464
0.638
0.620
0.674
0.618
Isotropic GRF
20 Hz
0.758
0.533
0.751
0.717
0.770
0.747
40 Hz
1.152
0.895
0.883
0.891
0.893
0.951
10 Hz
0.560
0.388
0.541
0.527
0.376
0.382
15 Hz
0.771
0.518
0.671
0.656
0.498
0.483
Anisotropic GRF
20 Hz
0.812
0.612
0.754
0.725
0.607
0.599
40 Hz
1.018
0.887
0.905
0.898
0.803
0.950
Table 5: OOD performance comparison of models on the test folds of the time-harmonic datasets. The table layout similar to Table 4. The wavespeed reported in the left column shows the OOD wavespeed used in data; for instance, the three "Isotropic GRF rows are based on models trained the corresponding Anisotropic GRF versions, and vice versa.

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.05
0.00
-0.05
FNO 4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.05
0.00
-0.05
Relative error
0.00075
0.00050
0.00025
Figure 8:
In-distribution test performance of models on an acoustic hameone dataset. The time-
harmonic dataset is configured with isotropic GRF wavespeed and frequency w/2n = 10Hz. The figure layout same as Figure in the main text.
C Full experimental results of the elastic hime-harmonic datasets
Freq. w/2T
FNO-depth-4
FNO-depth--8
U-Net-ch-32 U-Net-ch-64 UNO-modes-12 12
UNO-modes-16
10 Hz
0.080
0.039
0.154
0.141
0.103
0.076
15 Hz
0.130
0.072
0.229
0.217
0.205
0.127
20 Hz
0.225
0.124
0.267
0.225
0.264
0.204
40 Hz
0.504
0.365
0.497
0.470
0.534
0.490
Table 6:
In-distribution performance comparison of models the test folds of the elastic time-harmonic datasets. Wavespeeds are anisotropic GRFs for all frequencies. The error metric is the relative T.2 error u @ /22/ / between the ground-truth u and prediction â.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
4
0.00
-025
FNO 4epth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
-0.25
Relative error
0.00075
0.00050
0.00025
Figure 9: In-distribution test performance of models acoustic time-harmonic dataset. The time-
harmonic dataset is configured with with isotropic GRF wavespeed and frequency w/2m = 15Hz. The figure layout is same as Figure 5 in the main text.
20

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
4
0.00
2
-0.25
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
-0.25
Relative error
0.0010
0.0005
Figure 10:
In-distribution test performance of models on acoustic time-hame harmonic dataset. The time-
harmonic dataset is configured with isotropic GRF wavespeed and frequency w/2n 20Hz. The figure layout same Figure 5 in the main text.
21

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.00
2
-0.05
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.05
0.00
-0.05
Relative error
0.003
0.002
0.001
Figure 11: In-distribution test performance of models on acoustic time-harmonic dataset. The time-
harmonic dataset configured with anisotropic GRF wavespeed and frequency w/2n = 10Hz. The figure layout same as Figure in the main text.
22

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.5
4
0.0
2
-0.5
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
0.5
Prediction
0.0
0.5
Relative eiror
0.0050
0.0025
Figure 12:
In-distribution test performance of models on acoustic time- harmonic dataset. The time-
harmonic dataset is configured with anisotropic GRF wavespoed and frequency w/2n = 15Hz. The figure layout same as Figure in the main text.
23

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.5
0.0
2
0.5
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.0
0.5
Relative error
0.0075
0.0050
0.0025
Figure 13: In-distribution test performance of models on acoustic time-harmonic dataset. The time-
harmonic dataset configured with anisotropic GRF wavespeed and frequency w/2n = 20Hz. The figure layout same as Figure in the main text.
24

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
4
0.00
2
-0.25
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
-0225
Relative error
0.010
0..05
Figure 14: In-distribution test performance of models on acoustic time-harmonic dataset. The time-
harmonic dataset configured with anisotropic GRF wavespeed and frequency w/2n = 40Hz. The figure layout same as Figure in the main text.
25

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.05
0.00
2
-0.05
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.05
0.00
-0.05
Relative eiror
0.015
0.010
0.005
Figure 15: OOD test performance of models on a time-harmonic dataset. The model is trained on the time- harmonic dataset with isotropic GRF wavespeed and frequency w/2n = 10Hz, but tested on the anisotropic version instead. The figure layout follows from Figure in the main text.
26

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
0.00
-0.25
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.00
-0.25
Relative error
0.015
0.010
0.005
Figure 16: OOD test performance of models on a time-harmonic datasct. The model is trained on the time- harmonic dataset with isotropic GRF wavespeed and frequency w/2n = 15Hz, but tested on the anisotropic version instead. The figure layout follows from Figure in the main text.
27

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
0.00
0.25
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
-0.25
Relative error
0.015
0.005
Figure 17: OOD test performance of models time-harmonic dataset. The model is trained the time- harmonic dataset with isotropic GRF wavespeed and frequency w/2n = 20Hz, but tested on the anisotropic version instead. The figure layout follows from Figure in the main text.
28

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.05
0.00
00055
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.05
0.00
-0.05
Relative error
0.0050
0.0025
Figure 18: OOD test performance of models on a time-harmonic dataset. The model is trained on the time- harmonic dataset with isoloppcc GRF wavespeed and frequency w/2n 10Hz, but tested on the anisotropic version instead. The figure layout follows from Figure in the main text.
29

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
0.00
-025
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
-0.25
Relative error
0.010
0.005
Figure 19: OOD test performance of models time-harmonic dataset. The model is trained the time- harmonic dataset with anisotropic GRF wavespeed and frequency w/2n = 15Hz, but tested on the isotropic version instead. The figure layout follows from Figure in the main text.
30

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
0.00
-025
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
-0.25
Relative error
0.015
0.010
0.005
Figure 20: OOD test performance of models on a time-harmonic datasct. The model is trained on the time- harmonic dataset with anisotropic GRF wavespeed and frequency w/2n = 20Hz, but tested on the isotropic version instead. The figure layout follows from Figure in the main text.
31

---

Published in Transactions on Machine Learning Research (02/2024)
Wavespeed
Pressure (real part)
0.25
0.00
-0225
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.25
0.00
-0.25
Relative error
0.02
0.01
Figure 21: OOD test performance of models time-harmonic dataset. The model is trained the time- harmonic dataset with anisotropic GRF wavespeed and frequency w/2n = 40Hz, but tested on the isotropic version instead. The figure layout follows from Figure in the main text.
32

---

Published in Transactions on Machine Learning Research (02/2024)
P-Wavespeed
U field (real part)
-2
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
2
Prediction
0
2
Relative error
0.002
0.001
Figure 22: In-distribution test performance models on an elastic time-harmonic dataset. The time-
harmonic dataset is configured with anisotropic GRF wavespeed and frequency w/2m = 10Hz.
33

---

Published in Transactions on Machine Learning Research (02/2024)
P-Wavespeed
U field (real part)
1
0
3
-1
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
1
0
Relative error
0.003
0.002
0.001
Figure 23: In-distribution test performance models on an elastic time-harmonic dataset. The time-
harmonic dataset is configured with anisotropic GRF wavespeed and frequency w/2m = 15Hz.
34

---

Published in Transactions on Machine Learning Research (02/2024)
P-Wavespeed
U field (real part)
3
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0
Relative error
0.004
00.002
Figure 24: In-distribution test performance models on an elastic time-harmonic dataset. The time-
harmonic dataset is configured with anisotropic GRF wavespeed and frequency w/2m = 20Hz.
35

---

Published in Transactions on Machine Learning Research (02/2024)
P- P-Wavespeed
U field (real part)
5
0.0
0.5
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
Prediction
0.5
0.0
0055
Relative error
0.02
0.01
Figure 25:
In-distribution test performance of models elastic time-harmonte harmonic dataset. The time-
harmonic dataset is configured with anisotropic GRF wavespeed and frequency w/2n = 40Hz.
36

---

D
Full experimental results of the time-varying datasets
Published in Transaction ions on Machine Learning Research (02 )
Problem
Wavespeed
Init. press.
FNO-depth-4
FNO-depth-8
U-Net-ch-32
U-Net--ch-4
UNO-modes- 12
UNO-modes-
Thick lines
0.421
0.381
0.393
0..71
0.466
0.433
Gaussian lens
MNIST
0.461
0.410
0.525
0.520
0.478
0.434
Thick lines
0.365
0.329
0.451
0.430
0,411
0.378
RTC
Iso. GRF
MNIST
0.342
0.349
0.535
0.517
0.337
0.329
Thick lines
0.348
0.308
0.432
0.414
0.395
0.362
Aniso. GRF
MNIST
0.378
0.377
0.469
0.491
0.364
0.362
Thick lines
0.550
0.500
0.446
0.443
0.551
0.545
Gaussian lens
MNIST
0.695
0.686
0.629
0.627
0.652
0.667
Thick lines
0.436
0.380
0.383
0.356
0.462
0.441
1S
Iso. GRF
MNIST
0.489
0.479
0,511
0.511
0.463
0.465
Thick lines
0.415
0.351
0.359
0.327
0.436
0.419
Aniso. GRF
MNIST
0.471
0.449
0.537
0.514
0.413
0.414
Table 7: Performance comparison of models on the test folds of the time-varying datasets.
The error metric the relative T,2 error
lp 21 /lll between the ground-truth and prediction p.

---

Published in Transactions on Machine Learning Research (02/2024)
In-distri imput
In-distri target
OOD input
OOD target
1.00
1.00
Ground truths
0.2
2
0.75
0.25
0.75
0.0
0.50
0.00
0.50
0.25
-.25
0.25
-.2
0.00
0.00
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
OOD prediction In-distri prediction
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
1.00
1.00
1.00
1.00
2
2
2
2
0.75
0.75
0.75
0.75
0.50
0,50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
Figure 26: Test performance of models on the time-varying RTC dataset with isotropic GRF wavespeed.
The first row shows the input and target samples of the RIC dataset; they can be either in-distribution OOD. The second row shows the model predictions the in-distribution sample. The second row shows the model predictions on the OOD sample.
E
Case study: Gradually challenging the OOD generalization
We have seen the limitation PDE surrogates on OOD samples. We note that the comparison "in- distribution vs OOD" is a simplified, binary notion, as implies that a sample either within a distribution or outside of it. More fine-grained OOD notions are helpful, as intuitively the performance of models on a sample may depend the degree to which that sample differs from those in the training distribution.
In this section, we present a case study where we vary wavespeeds from near in-distribution samples (*less OOD") to distant ones ("more OOD"). See Figure 31 for visualization. We employ neural style transfer (NST) (Gatys et al., 2016) to create '0' digits of different OOD levels. These images are then used as wavespeeds in time-varying problems.
Recall that the NST algorithm separates and recombines the content and the style of images. Our content of interest is '0' digit and the style of interest. box-like strokes. The style and content of an image are balanced by content weighting factor; see Gatys et al. (2016) for details. By adjusting the content weighting factor (referred to as OOD weight in Figure 31) from small to large, we generate images that resemble in- distribution thick lines (left panels of Figure (1() and OOD MNIST (right panels of Figure 31(A)). We use the open-source NST implementation' for data generation.
Using different OOD degrees for wavespeed samples, we assess various PDE surrogates on RTC and tasks. Figure (B) displays the numerical results, As anticipated, higher OOD degrecs' wavespeeds are harder recover, giving higher relative errors.
38

---

Published in Transactions on Machine Learning Research (02/2024)
In-distri input
In-distri target
OOD imput
OOD target
1.00
1.00
Ground truths
2
0.2
0.75
0.0
0.50
0.50
-0.2
0.25
-0.5
0.25
0.00
0.00
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
OOD prediction In-distri prediction
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
1.00
2
1.00
?
1.00
2
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
Figure 27: Test performance of models on the time-varying RTC datasot with Gaussian lens wavespood.
The figure layout is the same Figure 26.
In-distri input
In-distri target
OOD input
OOD target
1.00
1.00
Ground truths
2
0.2
0.75
0.75
0.0
0.50
0.50
-0.2
0.25
0.25
0.5
0.00
0.00
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
In-distri prediction
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.00
OOD prediction
1.00
1.00
1.00
1.00
2
2
2
2
0.75
0.75
0.75
0.75
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
Figure 28: Test performance models on the Lime-varying RTC datasel with anisotropic GRP wavespeed. The figure layout is the same Figure 26.
39

---

Published in Transactions on Machine Learning Research (02/2024)
In-distri input
In-distri target
OOD input
OOD target
1.00
1.00
Ground truths
0.5
2
0.2
0.75
0.75
0.0
-02
0.25
0.25
0.5
0.00
0.00
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
OOD prediction In in-distri prediction
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0,50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
Figure 29: Test performance of models on the time-varying 1S dataset with Gaussian lens wavespeed. The figure layout is the same Figure 7.
In-distri input
In-distri target
OOD input
OOD target
1.00
1.00
Ground truths
0.2
0.75
0.25
2
0.75
0.0
0.50
0.00
0.50
-.2
0.25
-025
0.25
0.00
0.00
FNO depth-4
FNO depth-8
U-Net ch-32
U-Net ch-64
In-distri prediction
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
OOD prediction
1.00
1.00
1.00
1.00
2
2
2
2
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
Figure 30: Test performance of models on the time-varying IS dataset with anisotropic GRF wavespeed. The figure layout is the same Figure 7.
40

---

Published in Transactions on Machine Learning Research (02/2024)
(A)
OOD weight: 0.001
OOD weight:
OOD weight:
OOD weight: 0.2
0.01
0.02
Less OOD (closer to thick lines)
More OOD
(closer to MNIST)
(B)
RTC
IS
0.475
0.60
0.450
0.55
relative error
0.425
0.50
0.400
0.375
0.45
0.350
0.40
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
OOD weight
OOD weight
FNO--deth-88
UNet-ch-64
UNO-mode-16
Figure 31: Case study: Model performance and OOD degree. Panel (A): Samples from neural style transfer, with content factor weights 20 OOD weights ranging from 0.001 to 0.2. Panel (B): Models' performance diverse wavespeed samples. More OOD wavespeed (closer to MNIST than thick lines) results in higher errors.
41


Reviews:
[{"review_id": "KCo4S7Cw6d", "reviewer": "Reviewer_LgzV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-- Presentation of the main text is clear and straightforward.\n\nWeaknesses:\n-- Several technical details should be further clarified. Specific questions are stated in the “Requested Change” section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission presents a collection of dataset of wave propagation PDEs for time-harmonic wave equations and time-varying wave equations respectively. \nThe time-harmonic wave dataset concerns acoustic wave problems and elastic wave problems. \nAnd the time-varying wave dataset concerns final-value problems where the termination condition is given as local or global information and the target is to infer the initial condition.\n\nThe source dataset is generated by hybridizable discontinuous Galerkin method implemented in ‘hawen’ package.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions:\n\n\n-- Why not submit to venues like NeurIPS 2023 Datasets and Benchmarks Track [https://nips.cc/Conferences/2023/CallForDatasetsBenchmarks]? This work will receive wider exposure to the ML community when published in a dataset-specific venue.\n\n-- In equation (1), there is an external driving force term $f(\\mathbf{x},\\omega)$. There is no specification in the entire manuscript about this term.\n\n-- Specify how the Gaussian random fields are generated for the wave speed $c(\\mathbf{x})$.\n\n-- What are $\\lambda$ and $\\mu$ in equations (8) and (9) in appendix section A.2?\n\n-- Considering there are multiple hyperparameters in setting up the PDE problem, authors should make a table (for each subtype of problem, or for a general type) detailing the combination of the parameters.\n\n-- Does the notation $\\dagger$ imply adjoint operator in any case in the manuscript?\n\n-- For section 4, what loss functions are used to set up training? There should be detailed explanation of training setup.\n\n-- Can authors comment on if the solution accuracy results by the selected methods (FNO, Unet and UNO) are up to expectation considering reported performance in precedent work which also leverages these methods?\n\nMinor comments:\n\n\nTypo: section 2 Background, paragraph bold font: ‘bechmarks’", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eSEbNJ7yIj", "reviewer": "Reviewer_Bern", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents certainly a valuable contribution as a collection of datasets for PDE surrogate learning. The focus on the wave propagation PDEs sounds reasonable given the brief literature review in the paper (though I am not really qualified to assess how the current contribution is different from existing benchmarking datasets). The given datasets seem sufficiently challenging as operator learning problems.\n\nSome minor points were unclear as listed below:\n\n(1) For the time-harmonic elastic waves, what is the typical form of the operator $G^\\dagger$ to be estimated? It is missing only for this particular case.\n\n(2) Are there any reasons behind the particular sizes of the datasets provided? e.g., do they match with the numbers in other similar datasets?\n\n(3) Is it correct that the codes for **generating** (and not benchmarking) the data will not be published? Or are they included in the \"benchmark code\" to be in a github repository?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A number of benchmarking datasets for surrogate learning for wave propagation PDEs are proposed. The collection comprises not only different wave PDEs but also multiple problem settings. Moreover, some baseline methods of operator learning are examined on these datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest no major changes. It would be great if the authors could check the minor points that I listed in the previous form and provide relevant information or make changes accordingly.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XjDcJxx7le", "reviewer": "Reviewer_2kwj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stengths:\n - The paper is certainly interesting for a growing subcommunity in machine learning. \n - The machine learning part seems well done. Experiments are performed on reasonable baseline models and clearly described. The results are adequately discussed and strength and weaknesses of the baseline models are highlighted. These results are very believable.\n - The visualizations of the data are well done.\n - Details of the data generation are clearly. \n \nWeaknesses:\n\nThere is one major weakness: The 24 datasets relating to the wave equation are far from covering all possible applications of the wave equation.\n- The main restriction is to have Gaussian random fields as right hand sides. Some applications have zero right hand sides, which are not covered this way. Some applications have much stronger structured right hand sides.\n- The restriction to d=2 might sound reasonable when trying to minimize computation time. However, ML methods for PDEs are supposed to shine in comparison to numerical solvers particularly in higher dimensions. Or the rather relevant Maxwell's equations need the wave equation in d=2.\n- \"Each component (ux or uy ) is complex-valued.\" This is a restriction, as the real wave equation is arguably at least as important as the complex one.\n- For linear PDEs and the wave equation, there are some recent GP models, see later. These are asking different problems regarding the wave equations and it seems your datasets do not (obviously) cover their applications.\n 1) Härkönen et al., Gaussian Process Priors for Systems of Linear Partial Differential Equations with Constant Coefficients, ICML 2023\n 2) Henderson et al., Covariance models and Gaussian process regression for the wave equation. Application to related inverse problems, Journal of Computational Physics 2023\n 3) Henderson et al., Wave equation-tailored Gaussian process regression with applications to related inverse problems, preprint\n 4) Henderson et al., Stochastic Processes Under Linear Differential Constraints: Application to Gaussian Process Regression for the 3 Dimensional Free Space Wave Equation, preprint\n\nThere are some minor problems on the level of typos, see below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new dataset for various forms of the wave equation. \tThe author(s) provide a detailed description of how the data was generated, suggest example applications, and provide and discuss results for various state of the art models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I could certainly live with a paper that is restricted in its applications. However, I would like more information perferably in the abstract (or if necessary in the introduction) about what kind of data is given and what kinds of problems are supposed to be solved by this data. See all points above, and this is me (not an expert about the wave equation) just thinking about spontaneously while writing this review. A lot more care should go into clearly describing the place of these datasets in the (wide!) field (in cs, engineering, math, and physics) about the wave equation. None of this literature is even hinted at, and virtually only ml papers are being cited.\n - Why do you introduce the Helmholtz equation before even writing down the wave equation? The connection between Helmholtz and wave equation is long known in math, but probably not so much in ML. Hence, a hint of their connection (or even the full derivation, which is not long) might be helpful.\n - title: When reading \"wave propagation PDEs\" I am not sure whether the linear wave equations are considered more the non-linear PDEs modeling e.g. ocean waves.\n - p. 2: \"WaveBench includes 24 datasets\" vs. page 3: \"Our proposed WaveBench consists of 20 datasets\"", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper makes a serious effort to establish a first benchmark that will allow future assessments of algorithms for differential operator learning. As far as I know, a benchmark for the case of wave propagation does not exist despite its practical application.
Decision: Accept (poster)