Unsupervised Learning of Full-Waveform Inversion: Connecting CNN and Partial Differential Equation in a Loop
Authors: Peng Jin
Xitong Zhang
Yinpeng Chen
Sharon X Huang
Zicheng Liu
Youzuo Lin
Venue: ICLR 2022 Poster (2022)
Keywords: Unsupervised Learning
Full-Waveform Inversion
Partial Differential Equation
Physics-Informed Machine Learning

Abstract: This paper investigates unsupervised learning of Full-Waveform Inversion (FWI), which has been widely used in geophysics to estimate subsurface velocity maps from seismic data. This problem is mathematically formulated by a second order partial differential equation (PDE), but is hard to solve. Moreover, acquiring velocity map is extremely expensive, making it impractical to scale up a supervised approach to train the mapping from seismic data to velocity maps with convolutional neural networks (CNN).We address these difficulties by $\textit{integrating PDE and CNN in a loop}$, thus shifting the paradigm to unsupervised learning that only requires seismic data. In particular, we use finite difference to approximate the forward modeling of PDE as a differentiable operator (from velocity map to seismic data) and model its inversion by CNN (from seismic data to velocity map). Hence, we transform the supervised inversion task into an unsupervised seismic data reconstruction task. We also introduce a new large-scale dataset $\textit{OpenFWI}$, to establish a more challenging benchmark for the community. Experiment results show that our model (using seismic data alone) yields comparable accuracy to the supervised counterpart (using both seismic data and velocity map). Furthermore, it outperforms the supervised model when involving more seismic data.

Published as a conference paper at ICLR 2022
UNSUPERVISED LEARNING OF FULL- FULL-WAVEFORM INVERSION: CONNECTING CNN AND PARTIAL DIFFERENTIAL EQUATION IN A LOOP
Peng Jiin 1.2x, Xitong Zhang Yinpeng Chen', Sharon Xiaolei Huang", Zicheng Liu and Youzuo Lin'
Farth and Environmental Sciences Division, Los Alamos National Laboratory
Coolleee of Information Sciences and Technology, The Pennsylvania State University
DDepartment of Computational Mathematics, Science and Engineering, Michigan State University
*Microsoft Research
[pq]5125, sub972 /2 8 epsu.ce edu, zhangxit anangxittmms.. Cmsu. edú,, {yiche, 2 www.:eeicrooott.com ylin 1aeeeaan...e . gov
ABSTRACT
This paper investigates unsupervised learning of Full-Waveforn Inversion (FWI), which has been widely used in geophysics to estimate subsurface velocity maps from scismic data. This problem is mathematically formulated by second order partial differential equation (PDE), but is hard to solve. Moreover, acquiring ve- locity map is extremely expensive, making it impractical scale up supervised approach to train the mapping from seismic data to velocity maps with convolu- tional neural networks (CNN). We address these difficulties by integrating PDE and CNN in a loop, thus shifting the paradigm to unsupervised learning that only requires seismic data. particular, we use finite difference approximate the forward modeling of PDE as differentiable operator (from velocity map to seis- mic data) and model inversion by CNN (from seismic data to velocity map). Hence, we transform the supervised inversion task into unsupervised seismic data reconstruction task. We also introduce new large-scale dataset Open/ to establish more challenging benchmark for the community. Experiment results show that our model (using seismic data alone) yields comparable accuracy to the supervised counterpart (using both seismic data and velocity map). Furthermore, it outperforms the supervised model when involving more seismic data.
Input Sesmic Data
Pradicted
Recanstructed Sesmic Data
Velocity Map
Modeling
Acoustic Wave Equation
Ppp(r. wgp de
s(r
Reto Diforncce
Retonstruction E(p.f)
Figure 1: Schematic illustration of our proposed method, which comprises CNN to learn an inverse mapping and a differentiable operator to approximate the forward modeling of PDE.
INTRODUCTION
Geophysical properties (such as velocity, impedance, and density) play an important role in various subsurface applications including subsurface energy exploration, carbon capture and sequestration,
"Equal contribution.
Dataset is availahle httph:://ppenfiiiil/// www.aaann..g........

---

Published as a conference paper at ICLR 2022
Offiet (m)
150
300
450
903
150
300
450
600
900
9600
150
5400
04
5200
Time (m0))
150
Degth
600
00
-04
(a)
(b)
Figure 2: An example of (a) velocity map and (b) seismic measurements (named shot gather in geophysics) and the ID time-series signal recorded by a receiver.
estimating pathways of subsurface contaminant transport, and earthquake early warning systems to provide critical alerts. These properties can be obtained via seismic surveys, i.c., receiving re- flected/refracted seismic waves generated by controlled source. This paper focuses on recon- structing subsurface velocity maps from seismic measurements. Mathematically, the velocity map and seismic measurements correlated through an acoustic-wave equation (a second-order partial differential equation) as follows:
1 pp(r,t) p(r,rt) - v(r)2 0t2
= s(r,t),
(1)
where p(r, t) denotes the pressure wavefield spatial location and time t, v(() represents the velocity map, and 6(r, 1) is the source term. Full-Waveform Inversion (FWI) is a methodology that determines high-resolution velocity maps v(r) of subsurface via matching synthetic seismic wave- forms to raw recorded seismic data p [, t), where F represents the locations of seismic receivers.
velocity map describes the wave propagation speed in the subsurface region interest. An exam- ple in 2D scenario is shown in Figure 2a Particularly, the x-axis represents the horizontal offset of a region, and the y-axis stands for the depth. The regions with the same geologic information (veloc- ity) are called a layer in velocity maps. In sample of seismic measurements (termed shot gather in geophysics) as depicted in Figure 26] each grid in the x-axis represents receiver, and the value in the y-axis is a ID time-series signal recorded by each receiver.
Existing approaches solve FWI two directions: physics-driven and data-driven. Physics-driven approaches rely on the forward modeling of Equation which simulates seismic data from velocity map by finite difference. They optimize velocity map per seismic sample, by iteratively updating velocity map from an initial guess such that simulated seismic data (after forward modeling) is close to the input seismic measurements. However, these methods are slow and difficult to scale up as the iterative optimization is required per input sample. Data-driven approaches consider FWI problem translation task and apply convolution neural networks (CNN) to learn the
as an image-to-iman mapping from seismic data velocity maps (Wu & Lin (019).. The limitation of these methods is that they require paired seismic data and velocity maps to train the network. Such ground truth velocity maps are hardly accessible in real-world scenario because generating them extremely lime-consuming even for domain experts.
In this work, we leverage adattages of both directions (physics + data driven) and shift the paradigm to unsupervised learning of FWI by connecting forward modeling and CNN in loop. Specifically, as shown in Figure ! a CNN trained to predict velocity map from seismic data, which is followed by forward modeling to reconstruct seismic data. The loop is closed by applying reconstruction loss on seismic data to train the CNN. Due to the differentiable forward modeling, the whole loop can be trained end-to-end. Note that the CNN is trained in unsupervised manner, the ground truth of velocity map is not needed. We name our unsupervised approach as UPFWI (Unsupervised Physics-informed Wave-oraeform Inversion).
Additionally, we find that perceptual loss (Johnson et al. 2016) is crucial to improve the overall quality of predicted velocity maps due to its superior capability in preserving the coherence of the reconstructed waveforms comparing with other losses like Mean Squared Error (MSE) and Mean Absolute Error (MAE).
2

---

Published as a conference paper at ICLR 2022
Mean Squared Error (MSE)
Structural Similarity (SSIM)
1c2
0.960
Moan Squared Error
Structureo Sinilanty
0.965
4-
0.960
0.075
0.930
H
PGN
1
0.965
1.2
2,4
Size of Training Set
le4
Size of Training
1e4
Figure 3: Unsupervised UPFWI (ours) vs. Supervised H-PGNN+ (Sun et al. 2021). Our method achieves better performance, e-g. lower Mean Squared Error (MSE) and higher Structural Similar- ity (SSIM), when involving more unlabeled data (>> 24k).
To encourage fair comparison introduce a new synthetic dataset named OpenFWI, which contains 60,000 labeled data (velocity map and seismic data pairs) and 48,000 unlabeled data (seismic data alone). 30,000 of those velocity maps contain curved layers that are more challenge for inversion. We also add geological faults with various shift distances and tilting angles to all velocity maps.
large dataset with more complicate geological structures,
We evaluate our method on this dataset. Experimental results show that for velocity maps with flat layers, our UPFWI trained with 48,000 unlabeled data achieves 1146.09 in MSE, which 26.77% smaller than that of the supervised baseline H-PGNN+ (Sun et al., [02I]) and 0.9895 in Struc- tured Similarity (SSIM), which is 0.0021 higher; for velocity maps with curved layers, our UPFWI achieves 3639.96 96 in MSE, which is 28.30% smaller, and 0.9756 in SSIM, which is 0.0057 higher.
Our contribution is summarized follows:
We propose to solve FWI in an unsupervised manner by connecting CNN and forward modeling in a loop, enabling end-to-end learning from seismic data alone.
We find that perceptual loss is helpful to boost the performance comparable to the super- vised counterpart.
We introduce large-scale dataset as benchmark encourage further research on FWI.
2 PRELIMINARIES OF FULL--WAVEFOR INVERSION (FWI)
The goal of FWI in geophysics is to invert for velocity map E WxH from seismic measure- ments p E S where W and denote the horizontal and vertical dimensions of the velocity map, S is the number of sources used to generate waves during data acquisition process, T de- notes the number of samples in the wavefields recorded by each receiver, and R represents the total number of receivers.
In conventional physics-drive driven methods, forward modeling is commonly referred to the process of simulating seismic data from given estimated velocity maps 6. For simplicity, the forward acoustic-wave operator f can be expressed as
p = f()).
Given this forward operator f, the physics-driven FWI can be posed as a minimization prob- lem (Virieux & Operto 2009)
E' min {IIp f(0)) x
where ||p 1 is the the & distance between true seismic measurements and the corre- sponding simulated data f()), À is a regularization parameter and R(6) is the regularization term which is often the lg or é, norm of 6. This requires optimization per sample, which is slow as the optimization involves multiple iterations from an initial guess.
Data-driven methods leverage CNNs to directly learn the inverse mapping as (Adler al. 2021
go (p) (p),
3

---

Published as a conference paper at ICLR 2022
where (e(-) is the approximated inverse operator of f()) parameterized by 8. In practice, ge is usually implemented as CNN (Adler et al. 2021 Wu & Lin 2019). This requires paired seismic data and velocity maps for supervised learning. However, the acquisition of large volume of velocity maps in field applications can be extremely challenging and computationally prohibitive.
3 METHOD
In this section, present Unsupervised Physics-informed solution (named UPFWT), which connects CNN and forward modeling in a loop. It addresses limitations of both physics-driven and data-driven approaches, it requires neither optimization at inference (per sample), nor velocity maps supervision.
3.1 UPFWI: CONNECTING CNN AND FORWARD MODELING
As depicted in Figure I our UPFWI connects a CNN go and differentiable forward operator to form a loop. particular, the CNN takes seismic measurements p as input and generates the corresponding velocity map 6. We then apply forward acoustic-wave operator f (see Equation 2 on the estimated velocity map 0 to reconstruct the seismic data p. Typically, the forward modeling employs finite difference (FD) to discretize the wave equation (Equation DT. The details of forward modeling will be discussed Section 3.3 The loop is closed by the reconstruction loss between input seismic data and reconstructed seismic data p fge (p)). Notice that the ground truth of the velocity map v is not involved, and the training process is unsupervised. Since the forward operator differentiable, the reconstruction loss be backpropagated (via gradient descent) to update the parameters 0 in the CNN.
3.2 CNN NETWORK ARCHITECTURE
We use an encoder-decoue structured CNN (similar to Wu & Lin (0119) and Zhang & Lin (2020)) model the mapping from seismic data p E R velocity map R" The encoder compresses the seismic input and then transforms the latent vector to build the velocity estimation through a decoder. See the implementation details in Appendix A.T A.T
3.3 DIFFERENTIABLE FORWARD MODELING
We apply the standard finite difference (FD) in space domain and time domain to discretize the original wave equation. Specifically, the second-order central finite difference in time do- main ppr(r.) in Equation is approximated as follows:
012
p(r dt
++1
2pr + ) ((
+((A)))
(5)
A1)
where of. denotes the pressure wavefields at timestep i, and p and v are the wavefields at + At and t At, respectively. The Laplacian of p(r, t) can be estimated in the similar way on the space domain (see Appendix A.2). Therefore, the equation then be written as
+ = (2 va )2 - pt - 0² (f
(6)
where 2 here denotes the discrete Laplace operator.
The initial wavefield at the initial timestep is set to zero (i.e. pf 0). Thus, the gradient of loss L with respect to estimated velocity at spatial location r can be computed using the chain rule as
ac Ou(r)
ac Op(r,t) Op(r,t) du(r)
where T' indicates the total number of timesteps.

---

Published as a conference paper at ICLR 2022
3.4 LOss FUNCTION
The reconstruction loss of our UPFWI includes a pixel-wise loss and a perceptual loss as follows:
C(p. p) = Lpiuet (p, p) + perosptual (p. p),
(8)
where and p are input and reconstructed seismic data, respectively. The pixel-wise loss Cyivel combines & and lg distance as:
Cpizel (p. p) (p. + Xe(z(p.p).
where A1 and 22 are two saprr--prameterss control the relative importance. For the perceptual loss perceptuall we extract features from conv5 in a VGG-16 network (Simonyan & Zisserman 2015 pretrained on ImageNet (Krizhevsky et al. 2012) and combine the E and &g distance as:
Sperecptual (p. p) Agli (p)(), (())
((((()))(()))
(10)
where p()) represents the output of conv5 in the VGG-16 network, and λ and \ are two hyper- parameters. Compared to the pixel-wise loss, the perceptual loss is better to capture the region-wise structure, which reflects the waveform coherence. This is crucial to boost the overall accuracy of velocity maps (e.g. the quantitative velocity values and the structural information).
OPENEWI DATASET
We introduce a new large-scale geophysics FWI dataset OpenFWI, which consists of 108K seismic data for two types of velocity maps: one with flat layers (named FlatFault) and the other one with curved layers (named CurvedFault). Each type has 54K seismic data, including 30K with paired velocity maps (labeled) and 24K unlabeled. The 30K labeled pairs are splitted as 24K/3K/3K for training. validation and testing respectively. Samples are shown in Appendix A..3
The shape of curves in dataset follows sine function. Velocity maps in CurvedFault are de- signed to validate the effectiveness of FWI methods on curved topography. Compared to the maps with flat layers, curved velocity maps yield much more irregular geological structures, making in- version more challenging. Both FlatFault and CurvedFault contain 30,000 samples with 2 to 4 layers and their corresponding seismic data. Each velocity map has dimensions of 70x 70, and the grid size is 15 meter in both directions. The layer thickness ranges from 15 grids to 35 grids, and the veloc- ity in each layer is randomly sampled from uniform distribution between 3,000 meter/second and 6,000 meter/second. The velocity is designed to increase with depth to be more physically realistic. We also add geological faults every velocity map. The faults shift from 10 grids 20 grids, and the tilting angle ranges from -1339 to 123".
To synthesize seismic data, five sources are evenly placed on surface with a 255-meter spacing, and seismic traces are recorded by 70 receivers at each grid with an interval of 15 meter. The source is a Ricker wavelet with a central frequency of 25 Hz (Wang) 2015). Each receiver records time-series data for 1 second, and we use a millisecond sample rate to generate 1,000 timesteps. Therefore, the dimensions of seismic data become 5x 1000x Compared to existing datasets (Yang & Ma 2019 Moseley et al.) (2020), OpenFWI is significantly larger. It includes more complicated and physically realistic velocity maps. We hope it establishes a more challenging benchmark for the community.
5 EXPERIMENTS
In this section, we present experimental results of our proposed UPFWI evaluated on the OpInFWI.
5.1 IMPLEMENTATION DETAILS
Training Details: The input seismic data are normalized to range [-1, We employ AdamW (Loshchlloy & Hutter 2018) optimizer with momentum parameters 31 = 0.9, 3r = 0.999 and weight decay of 1 x 10 to update all parameters of the network. The initial learning rate is set to 3.2 x 10 10-4, and we reduce the learning rate by a factor of 10 when validation loss reaches a plateau. The minimum learning rate is set to 3.2 10-6 6 The size of mini-batch is set 128. All
5

---

Published as a conference paper at ICLR 2022
FladFault
CurvedFault
Supervision
Method
MAE
MSE. 4
SSIM 7
MAE.
MSE 4
SSIM
InversionNet
5285.38
Supervised
VelocityGAN
16.15
1770.31
0.9857
25.83
5076.79
0.9699
H-PGNN+
12.91
1565.02
0,9874
24.19
5139.60
0,9885
(our implementation)
UPFW1-24K (ours)
16.27
1705.35
0,9866
29.59
5712.25
0,9652
Unsupervised
UPFWI-48K (owns)
14.60
23.56
Quantitative results evaluated on OpenFWI in terms of MAE, MSE and SSIM. Our UPFWI yields comparable inversion accuracy comparing to supervised baselines. For H-PGNN+, we use our network architecture to replace the original one reported in their paper, and an additional perceptual loss between seismic data added during training.
Table 1:
Ground Truth
InversionNet
VelocityGAN
H-PGNN+
UPFWI-24K 21 (Ours)
UPFW1-488 48K (Ours)
Denerter 60
100
Figure 4: Comparison of different methods on inverted velocity maps FlatFault (top) and CurvedFault (bottom). For FlatFault, our UPFWI-48K reveals more accurate details at layer boundaries and the slope of the fault in deep region. For CurvedFault, our UPFWI reconstructs the geological anomalies on the surface that best match the ground truth.
trade-off hypee-parameters À in our loss function are set to 1. and train them on 8 NVIDIA Tesla V100 GPUs. All models are randomly initialized.
We implement our models in Pytorch
Evaluation Metries: We consider three metrics for evaluating the velocity maps inverted by our method: MAE, MSE and SSIM. Both MAE and MSE have been employed in existing methods (Wul & Lin 2019 Zhang & Lin 2020) to measure pixel-wise errors. Considering the layered-structunee velocity maps contain highly structured information, degradation or distortion can be easily per- ceived by a human. To better align with human vision, we employ SSIM to measure perceptual similarity. Note that for MAE and MSE calculation, we denormalize velocity maps to their original scale while we keep them in normalized scale [-1, 1] for SSIM according the algorithm.
We compare our method with three state-of-ther algorithms: two pure data-driven methods, i.e., InversionNet (Wu & Lin 2019) and VelocityGAN (Zhang & Lin 2020), and physics- informed method H-PGNN (Sun et al. 2021). We follow the implementation described these papers and search for the best saper-parameters for OpenFWI dataset. Note that improve H- PGNN by replacing the network architecture with the CNN in our UPFWI and adding perceptual loss, resulting in significant boosted performance. We refer our implementation as H-PGNN+ which strong supervised baseline. Our method has two variants (UPFW1-24K and UPFWI- 48K), using 24K and 48K unlabeled seismic data respectively.
Comparison:
5.2 MAIN RESULTS
Results on FlatFault: Table [ shows the results of different methods on FlatFault. Compared to InversionNet and VelocityGAN, our UPFW1-24K performs better in MSE and SSIM, but is slightly worse in MAE score. Compared to H-PGNN+, there is gap between our UPFWI-24K and H- PGNN+ when trained with the same amount of data. However, after we double the size of unlabeled data (from 24K to 48K), significant improvement is observed in UPFWI-48K for all three metrics, and it outperforms all three supervised baselines in MSE and SSIM. This demonstrates the potential of our UPFWI for achieving higher performance with unlabeled data involved.

---

Published as a conference paper at ICLR 2022
Ground Truth
pixel-F2 vdo0
pixel-Ey -all--yfe Fa
pisel-C Ea+ perceptual
300
6X0
$00
500
300
vo30
500
300
y000
900
-0.
0.0
6.2
800
400
600
64
100
220
303
800
000
500
800
-12
-24
300
250
130
600
300
350
300
300
400
030
750
190
799
750
502
500
(a)
(b)
Figure 5: Comparison of UPFWI with different loss functions on (a) waveform residual and their corresponding inversion results (ground truth provided in the first column), and (b) single trace residuals recorded by the receiver at 525 m offset. Our UPFWI trained with pixel-wise loss (l1+22 distance) and perceptual loss yields the most accurate results. Best viewed in color.
The velocity maps inverted by different methods are shown in the top row of Figure 4 Consistent with our quantitative analysis, more accurate details are observed in the velocity maps generated by UPFWI-48K. For instance, we find in the visualization results that both InversionNet and Ve- locityGAN generate blurry results deep region, while H-PGNN+, UPFWI-24K and UPFWI-48K yield much clearer boundaries. We attribute this finding as the impact of seismic loss. We further observe that the slope of the fault in deep region different from that in the shallow region, yet only UPFWI-48K replicates this result highlighted by the green square.
Table d shows the results of CurvedFault. Performance degradation is ob- served for all models, due to the more complicated geological structures in CurvedFault. Although our UPFWI-24K underperforms the three supervised baselines, our UPFWI-4SK significantly boosts the performance, outperforming strates the power of unsupervised learning in our UPFWI that greatly benefits from more unlabeled data when dealing with more complicated curved structure.
Results on CurvedFault:
supervised methods in terms of all three metries. This demon-
The bottom row of Figure 4 shows the visualized velocity maps in CurvedFault obtained using dif- ferent methods. Similar the observation in FlatFault, our UPFWI-48K yields more accurate details compared to the results of supervised methods. For instance, only our UPFWI-24K and UPFWI- 48K precisely reconstruct the fault beneath the curve around the top-left corner as highlighted by the yellow square. Although some artifacts are observed in the results of UPFWI-24K around the layer boundary in deep region, they are eliminated in the results of UPFW1-48K. More visualization results shown in Appendix A.3
5.3 ABLATION STUDY
We study the con- tribution of each loss term in our
Loss Terms:
Velucity
Seismii
pixel-f2 pixel-fz perceptual
MAE
MSE 1
SSIM
MAE
MSE L SSIM f
loss function: (a) pixel-wise lz distance (MSE), (b) pixel-wise & distance (MAE), and (c) per- ceptual loss. All experiments are conducted on FlatFault using 24,000 unlabeled data.
32.61
10014.47
0.9735
0.0167
0.0023
0.9978
21.71
2999.55
0.9775
0.0155
0.0025
0.9977
16.27
1705.35
0.9866
0.0140
0.0021
0.9984
Table 2: Quantitative results of our UPFWI with different loss
function settings.
Figure 5a shows the predicted velocity maps for using three loss combinations (pixel-(2, pixel-C bg. pixel-fi 1 perceepptllll in UPFWI. The ground truth seismic data and velocity map are shown in the left column. For each loss option, we show the difference between the reconstructed and the input seismic data (on the top) and predicted velocity (on the bottom). When using pixel-wise loss in le distance alone, there are some obvious artifacts in both seismic data (around 600 millisecond) and velocity map. These artifacts are mitigated by introducing additional pixel-wise loss in l distance.
7

---

Published as a conference paper at ICLR 2022
Marmousi
Salt
Network
MAEI
MSE
SSIMT
Method
MAE1
MSE.
SSIM1
MAEL
MSF.
SSIM
CNN
16.27
1705.35
0.9866
InversionNet
149.67
45936.23
0.7889
8669.95
0.9764
VIT 41.44
11029.01
0.9461
UPPWI
221.93
125825.75
0,7920
150.34
164595.28
0.7837
MLP-Mixer
22.32
4177.37
0.9726
Table 3: Quantitative results of our UPFWI evaluated on Marmousi and Salt datasets.
Table 4: Quantitative results of our UP- FWI with different architectures.
With perceptual loss added, more details are correctly retained (e.g. seismic data from 400 millisec- ond 600 millisecond, velocity boundary between layers). Figure [5b] compares the reconstructed seismic data (in terms of residual to the ground truth) a slice of 525 meter offset (orange dash line in Figure 5a Clearly, the combination of pixel-wise and perceptual loss has the smallest residual.
The quantitative results are shown in T They are consistent with our observation in qualitative analysis (Figure 5a).. In particular, using pixel-wise loss in & distance has the worst performance. The involvement of é distance mitigates velocity errors but is slightly worse on MSE and SSIM of seismic error. Adding perceptual loss boosts the performance in all metrics by a clear margin. This shows perceptual loss is helpful to retain waveform coherence, which is correlated to velocity boundary, and validates our proposed loss function (combining pixel-wise and perceptual loss).
More Challenging Datasets: We further evaluate our UPFWI on two more challenging tests including
FlatFoult
CurvedFault
(10
PSNR
MAE]
MSEJ
SSIMF
PSNR
MAEL
MSEJ
SSIM
0.5
61.60
15.68
1343.21
0.9888
61.72
23.78
3704.00
0.9751
Marmousi and Salt (Yang & Ma 2019) datasets and achieve solid re- sults. For Marmousi dataset, we fol- low the work of Feng et al.] (2021) and employ the Marmousi velocity map as the style image to construct a low-resolution dataset. Table 3 shows the quantitative results on both datasets. Although our UPFWI achieves good results on Salt dataset with preserved sub- surface structures, it has clearly larger errors than the supervised InversionNet. This is due to two reasons: (a) Salt dataset has a small amount of training data (120 samples), which is very chal- lenging for unsupervised methods; (b) the variability between training and testing samples is small, providing significantly larger favor to supervised methods than the unsupervised counterparts. The visualization of results on Marmousi dataset and Salt data are shown in Appendix A.4
1.0
58.70
24.84
4010.78
0.9733
58.70
24.84
4010.78
0.9733
5.0
51.58
44.33
7592.57
0.9681
51.68
46.90
10415.38
0.9441
Table Quantitative results of our UPFWI tested on seismic inputs with different noise levels.
Other Network Architectures: We fur-
Missing Traces
FlaFault
CurvedFanlt
ther conducted experiments by using Vision
MAET
MSEJ
SSIM
MART
MSEL
SSIM
Transformer Dosoviiskiyyy et al. (2020)
(5%)
21.23
1772.05
0.9868
41.33
6914.12
0.9622
and MLP-Mixer (Tolstikhin et | 22021) to replace CNN as the encoder. Table further
7 (10%)
33.66
3504.25
0.9814
61.72
12445.90
0.9453
17 (25%)
85.21
16731.69
0.9457
121.06
36770.77
0.8853
shows the quantitative results. Solid results are obtained for both network architectures, indicating our proposed method is model- agnostic. Visualization results are shown in Appendix A.4
Table Quantitative results of our UPFWI tested on seismic inputs with missing traces.
We validate the robustness of our UPFWI models by two additional tests: (I) testing data contaminated by Gaussian noise and (2) testing data with missing traces. The quan- titative results are shown in Ta and Table 6 respectively. We observe that in both experiments our model is robust to a certain level of noise and irregular acquisition. Visualization results are shown in Appendix A.4
Robustness Evaluation:
6 DISCUSSION
Our UPFWI has two major limitations. Firstly, it needs further improvement on small number of challenging velocity maps where adjacent layers have very close velocity values. We find that the lack of supervision is not the cause as our UPFWI yields comparable or even better results compared to its supervised counterparts. Another limitation is the speed and memory consumption for forward modeling, the gradient of finite difference (see Equation need to be stored for backpropagation. We will explore different loss functions (c.g. adversarial loss) and methods that can balance the requirement of computation resources and the accuracy in the future work. We believe the idea
8

---

Published as a conference paper at ICLR 2022
of connecting CNN and PDE to solve full-waveform inversion has potential to be applied other inverse problems with a governing PDE such as medical imaging and flow estimation.
RELATED WORK
Physies-driven Methods:
In the past few decades, many regularization techniques have been pro- posed to alleviate the ill-posedness and non-linearity of FWI (Hu et al. [2099] [ursttdde & Ghattas
2009
Ramírez & Lewis 2010 Lin & Huang. 2017 2015b al Guitton [012] Treister & Haber
2016 Other researchers focused on multi-scale techniques and decomposed the data into different
frequency bands (Bunks al. 1995 Boonyasiriwat et al. 2009).
Recently, some researchers employed neural networks to solve FWI. Those methods can be further divided into supervised and unsupervised methods.
Data-driven Methods:
One type of supervised methods require labeled samples to directly learn the inverse mapping, and they can be formulated as:
Supervised:
t (p) = go- (p) s.t. 0* ($.) = arg min
Σ E
(gg) vz),
(11)
where p denotes the seismic measurements, is the velocity map, 0 represents the trainable weights in the inversion network (-). f()) is the forward modeling, and L(: .) is a loss function. One exam-
ple of supervised methods is the fully connected network proposed by AAayy-Polo al. (2018). Wu
& Lin (2019) developed an encoder-decoder structured network to handle more complex velocity
maps. Zhang Lin (020) adopted GAN and transfer learning to improve generalizability. Li al.
2020) designed SeisInvNe Net to solve misaligned issue when dealing sources from different locations.
In Yang Ma (2019), a U-Net architecture was proposed with skip connections. [Feng et al. (2021) proposed a multi-scale framework by considering different frequency components. CCojas-Góme
al..( (220) developed an adaptive data augmentation method to improvegeneralizabliiy [Sun et al.
2022 combined the data-driven and physics-based methods and proposed H-PGNN model.
Another type supervised methods GANs to learn a distribution from velocity maps in training set as prior (Richardson) 2018, Mosser et al. 2020). They can be formulated as:
0(21 = go (z*) s.t.
(p) arg min ((f(ge- (z)), p).
(12)
0- ( arg min
Σ LcAN(go(ax), vz),
where o. is training dataset including numerous velocity maps. = and ag tensors sampled from the normal distribution. The iterative optimization is then performed on z to draw a velocity map sampled from the prior distribution.
The existing unsupervised methods follow the iterative optimization paradigm and perform FWI per sample. They employ neural networks to reparameterize velocity maps. The networks serve as an implicit regularization and are required to be pretrained on an expert initial guess. Those methods can be formulated as:
Unsupervised:
90 (p) (a) s.t. (p) = arg min
(13)
where random tensor. Different network architectures have been proposed including CNN- domain FWI (Wu & McMechan 2019) and DNN-FWI (He & Wang 2021). Zhu et al. (202T) developed NNFWI which does not need pretraining ahead, but the initial guess is still required be fed into the PDE with estimated velocity maps.
8 CONCLUSION
In this study, we introduce an unsupervised method named UPFWI to solve FWI by connecting CNN and forward modeling in loop. Our method can learn the inverse mapping from seismic data alone in an end-to-end manner. We demonstrate through a series of experiments that our UPFWI trained with sufficient amount of unlabeled data outperforms the supervised counterpart our dataset. The ablation study further substantiates that perceptual loss is critical component in our loss function and has great contribution to the performance of our UPFWI.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by the Center for Space and Earth Science Los Alamos National Labora- tory (LANL), and by the Laboratory Directed Research and Development program under the project number 20210542MFR at LANL.
REFERENCES
Amir Adler, Mauricio Araya-Polo, and Tomaso Poggio. Deep learning for seismic inverse problems: toward the acceleration of geophysical analysis workflows. IEEE Signal Processing Magazine, 38(2):89-119, 2021.
Mauricio Araya-Polo, Joseph Jennings, Amir Adler, and Taylor Dahlke. Deep-learning tomography. The Leading Edge, 37(1):58 66, 2018.
Chaiwoot Boonyasiriwat, Paul Valasek, Partha Routh, Weiping Cao, Gerard T Schuster, and Brian Macy, An efficient multiscale method for time-domain waveform tomography. Geophysics, 74 (6::CCC5-WCC66, 2009.
Carey Bunks, Fatimetou Saleck, Zaleski, and G Chavent. Multiscale seismic waveform inversion. Geophysics, 60K5):1457-1473, 1995.
Carsten Burstedde and Omar Ghattas. Algorithmic strategies for full waveform inversion: ex- periments. Geophysics, 74(6):37-46 2009.
Francis Collino and Chrysoula Tsogka. Application of the perfectly matched absorbing layer model to the linear elastodynamic problem in anisotropic heterogeneous media. Geophysics, 66(1); 294-307, 2001.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Shihang Feng. Youzuo Lin, and Brendt Wohlberg. Multiscale data-driven scismic full-waveform inversion with field data study. IEEE Transactions Geoscience and Remote Sensing, pp. 1-14, 2021. doi: 10. 20091
Antoine Guitton. Blocky regularization schemes for full waveform inversion. Geophysical Prospecting, 60:870-884, 2012.
Qinglong He and Yanfei Wang. Reparameterized full-waveform inversion using deep neural net- works. Geophysics. 86(1):V1-V13 2021.
Wenyi Hu, Aria Abubakar, and Tarck Habashy. Simultancous multifrequency inversion of full- waveform seismic data. Geophysics, 74(2):1-14, 2009.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learning, pp. 448-450 456.. PMLR, 2015.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pp. 694-711. Springer, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convo- lutional neural networks. Advances in Neural Information Processing Systems, 25:1097-1105. 2012.
Shucai Li, Bin Liu, Yuxiao Ren, Yangkang Chen, Senlin Yang. Yunhai Wang, and Peng Jiang. Deep- learning inversion of seismic data. IEEE Transactions on Geoscience and Remote Sensing, 58(3): 2135-2149, 2020. doi: 10.1 10110099GRS 209.
10

---

Published as a conference paper at ICLR 2022
Youzuo Lin and Lianjie Huang. Acoustic- and Variation regularization scheme. Geophysical Journal International, 200(1):489-500 2015a. doi:
inversion using modified Total-
10.1093/gi//ggu.33.
Youzuo Lin and Lianjie Huang. Quantifying subsurface geophysical properties changes using waveform inversion with modified VariatVai regularization
double-difference seismic-waveform scheme. Geophysical Journal International, 203(3)212552149) 2015b. doi: 10.1093/gii/ggw429,
Youzuo Lin and Lianjie Huang. Building subsurface velocity models with sharp interfaces using interface-guider seismic full-waveform inversion. Pure and Applied Geophysics, 174(11) 1):4035- 4055, 2017. doi: 10.1007/300024-07-116333
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
Ben Moseley, Tarje Nissen-Meyer, and Andrew Markham. Deep learning for fast simulation of seismic waves in complex media. Solid Earth, 11(4):1527-1549 2020.
Lukas Mosser, Olivier Dubrule, and Martin J Blunt. Stochastic seismic waveform inversion using generative adversarial networks as a geological prior. Mathematical Geosciences, 52(1):53-79, 2020.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), 807-814, 2010.
Adriana Ramírez and Winston Lewis. Regularization and full-waveform inversion: A two-step approach. In 80th Annual International Meeting, SEG. Expanded Abstracts, pp. 2773-2778, 2010.
Alan Richardson. Generative adversarial networks for model order reduction in seismic full- waveform inversion. arXiv preprint FeXiiv18806008800 2018.
Renán Mojas-Gómez, Jihyun Yang, Youzuo Lin, James Theiler, and Brendt Wohlberg. Physics- consistent data-driven waveform inversion with adaptive data augmentation. IEEE Geoscience and Remote Sensing Letters, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Jian Sun, Kristopher A Innanen, and Chao Huang. Physics-guided deep learning for seismic inver- sion with hybrid training and uncertainty analysis. Geophysics, 86(3):R303-R317, 2021.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. MIp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021
Eran Treister and Eldad Haber. Full waveform inversion guided by travel time tomography. SIAM Journal Scientific Computing, 39:8587-S609, 2016.
Jean Virieux and Stéphane Operto. An overview of full-waveform inversion exploration geo- physics. Geophysics, 746):WCCI-CC26 2009.
Yanghua Wang. Frequencies of the Ricker wavelet. Geophysics, 80(2):A31-A37 2015.
Yue Wu and Youzuo Lin. InversionNet: An efficient and accurate data-driven full waveform inver- sion. IEEE Transactions on Computational Imaging, 6(1):4194433, 2019.
Yulang Wu and George A McMechan. Parametric convolutional neural network-domain full- waveform inversion. Geophysics, (16):R881-886 2019.
Fangshu Yang and Jianwei Ma. Deep-lcarning inversion: A next-generation seismic velocity model building method. Geophysics, 84(4))R583-R59) 2019.

---

Published as a conference paper at ICLR 2022
Zhongping Zhang and Youzuo Lin. Data-driven seismic waveform inversion: A study the robust- ness and generalization. IEEE Transactions on Geoscience and Remote Sensing, 58:6900-668 6913, 2020.
Weiqiang Zhu, Kailai Xu, Eric Darve, Biondo Biondi, and Gregory C Beroza. Integrating deep neural networks with full-waveform inversion: Reparametrization, regularization, and uncertainty quantification. Geophysics, 87(1):1-103, 2021.
A APPENDIX
A.1 NETWORK ARCHITECTURE
Since the number of receivers R and the number of timesteps T in seismic measurements unbal- anced (T > R). we first stack 7x and six 3 convolutional layers (with stride 2 every the other layer to reduce dimension) extract temporal features until the temporal dimension is close to R. Then, six 3x3 convolutional layers are followed to extract spatial-temporal features. The resolution down-sampled every the other layer by using stride 2. Next, the feature map is flattened and fully connected layer is applied to generate the latent feature with dimension 512. The decoder first repeats the latent vector by 25 times to generate x12 tensor. Then is followed by five x convolutional layers with nearest neighbor upsampling in between, resulting in a feature map with size x0x80x33 x2. Finally, we center-crop the feature map (70x 70) and apply 3x3 convolution layer to output single channel velocity map.
All the aforementioned convolutional and upsampling layers are followed by batch normaliza- tion (Toffe & Szegedy 2015) and leaky ReI.U (Nair & Hinton 201O) as activation function.
A.2 DERIVATION OF FORWARD MODELING IN PRACTICE
Similar to the finite difference in time domain, 2D situation, by applying the fourth-order central finite difference in space, the Laplacian of p(r, t) can be discretized as
ppp ppp
t(r,
+
Ox²
022
Σ
Σ 1=-2
(14)
(Ax)² C: (A=)²
+ O((A)) (Az)*1
where 6 = - 01 12 C C_i. and and = stand for the horizontal offset and the depth of 2D velocity map, respectively. For convenience, assume that the vertical grid spacing A is identical to the horizontal grid spacing Az.
Given the approximation in Equations 5 and 14 we rewrite the Equation T
++1 pz. = (2 - 5a)p'.. - P.. - aas +
aip'eti + +
(15)
i=
4y10
where a (1A)))
During the simulation of the forward modeling, the boundaries of the velocity maps should be care- fully handled because they may cause reflection artifacts that interfere with the desired waves. One of the standard methods to reduce the boundary effects is to add absorbing layers around the original velocity map. Waves are trapped and attenuated by a damping parameter when propagating through those absorbing layers. Here, follow Collino & Tsogka (2001) and implement the damping parameter as
3uv
K d(u)
In (n(R),
(16)
2r
where denotes the overall thickness of absorbing layers, u indicates the distance between the current position and the closest boundary of the original velocity map, and R is the theoretical
12

---

Published as a conference paper at ICLR 2022
reflection coefficient chosen to be 10-? written as
With absorbing layers added, Equation 6 Caan be ultimately
2 (2 5a K)D'e.s K) -Ax)) 2,2 +
cl + Pe (17)
A.3 OPENFWI EXAMPLES AND INVERSION RESULTS OF DIFFERENT METHODS
Seismie Measurements in Five Channels
Velocity
Channel
Channel
Channel Channel 4
Channel 5
400
805
000
089
900
900
400
m30
099
600
000
099
804
000
CN
176
4153
600
405
804
000
:54
000
359
000
089
800
6X
Figure 6: More examples of velocity maps and their corresponding seismic measurements in Open- FWI dataset.
13

---

Published as a conference paper at ICLR 2022
UPFW1-24K
UPFWI-48K
Ground Truth
InversionNet
VelocityGAN
H-PGNN+
Ourss)
(Ours)
0PMr 344
0Ther BM
03 FM
0Phr
nBv 94
18
60-
20e 10
2hx 94
00
0lud 19)
201 191
Figure Comparison of different methods on inverted velocity maps of FlatFault. The details revealed by our UPFWI are highlighted.
14

---

Published as a conference paper at ICLR 2022
UPFWI-24K Ourss)
UPEWI-48K
Ground Truth
InversionNet
VelocityGAN
H-PGNN+
(Ours)
0The P1
2014 4M1
18w 94
10
40-
10
806 94
004134
00et
201 0m1
40-
Figure 8: Comparison of different methods on inverted velocity maps of CurvedFault. The details revealed by our UPFWI are highlighted.
15

---

Published as a conference paper at ICLR 2022
A.4
ADDITIONAL EXPERIMENT RESULTS
Ground Truth
Ground Truth
Prediction
Prediction
Oltxt Iml
Otbek [m]
Strsct ont
Ofbuk Ini
X0
70
200
751
XX
300
750
400
2803
3830
Otbel [m)
Ofbal [m]
Stsel (11
Othak (m1
3500
Dago 00
3000
Dapm
25:2
2003
Offect [m]
Othakt
200
1000
Dogn 1n0
40
400
Deean
01-
Figure 9: Results of low-resolution Marmousi Dataset. This dataset contains low-resolution ve- locity maps generated using style tranfer with the Marmousi velocity map as the style images. Our UPFWI model yields good results shallow regions, and also captures some geological structures in deeper regions. Similar phenomenon is also observed in the prediction of the smoothed Marmousi velocity map (bottom-right corner).
(m)
Offece (m)
Offict (M))
Office (N)
203
ICC
430
200 102
3.2
1.3
Dayydn
200
X39
200
uni
403
300
300
200
302
300
300
600
300
200
200
30
Inc
400
300
140
322
332
Choith mine
300
Deeth
223
220
200
200
2000
Figure 10: Results salt bodies dataset. This dataset contains more complicated velocity maps. Our UPFWI model yields good velocity map prediction (bottom) on both salt bodies and background geological structures compared to the ground truth (top).
16

---

Published as a conference paper at ICLR 2022
Ground Truth
CNN
MI.P-Mixer
Offien
[m]
Offect 2m1
700
600
900
100
700
100
157
159
839
509
80-
603
890
600
603
48000
3700
800
3000
Officet
OPsit (m))
Offset
000
150
400
250
909
450
750
3000
Officet
[m]
Offect 2m1
Offict
100
203 193
km
190
100
123
159
200
130
/02
500
4890
1710
603
100
600
660
4150
4500
Uttxd
Olbect Ins
Offeet
500
100
300
Degen
699
750
1020
Figure 11: Results of UPFWI with different network architectures. We replace the CNN in our model with Vision Transformer (VIT) and MLP-Mixer as the encoder and test them on the FlatFault dataset. Both models yield reasonable velocity maps. This demonstrates that our proposed learning paradigm model-agnostic.
17

---

Published as a conference paper at ICLR 2022
Ground Truth
Clean
PSNR=61.600
PSNR=58. 70JB
PS 588B
Scismic Inpat
Velocity Map
Scismic Input
Velocity Map
Figure 12: Results of adding Gaussian noise FlatFault. The model is trained on the clean data (without noise) and tested on different levels (PSNR) of Gaussian noises. Clearly, our method is robust the noise although slight degradation is observed when noise level increases.
Ground Truth
Clean
PSNR=61,7206
5S 50BBB 70dB
PSNR=51 6648
Seismic Input
Velocity Map
Seismic Input
2h
Velocity Map
Figure 13: Results adding Gaussian noise to CurvedFault. The model is trained on the clean data (without noise) and tested different levels (PSNR) of Gaussian noises. Similar to the results of FlatFault, our method robust to the noise although slight degradation is observed when noise level increases.
18

---

Published as a conference paper at ICLR 2022
Ground Truth
Clean
Missing
10 Missing
17 Missing
Seismic Input
Velocity Map
Seismic Input
Velocity Map
Figure 14: Results of randomly missing traces on FlatFault. The model trained on the clean data (without missing traces) and tested multiple missing rates from 5% to 25%. Our method is robust to the missing traces. Although the higher missing rate leads to shifts in velocity values, the geological structures are well preserved.
Ground Truth
Clean
Missing
10 Missing
17 Missing
Seismic Input
Velocity Map
Scismic Input
204
Velocity Map
Figure 15: Results of randomly missing traces on CurvedFault. The model is trained on the clean
data (without missing traces) and tested multiple missing rates from 5% to 25%. Similar to the results of FlatFault, our method is robust to the missing traces. Although the higher missing rate leads to shifts in velocity values, the geological structures well preserved.
19

---

Published as a conference paper at ICLR 2022
Additional experiments to investigate generalization. We conducted two additional experiments: (1) training our model on the CurvedFault dataset and further testing on the FlatFault dataset (visu- alization results are listed in Figure T6 and quantitative results are shown in Table 7h: (2) testing model on time-lapse imaging problems (visualization results are listed in Figure 17). The results demonstrate that our proposed model yields generalization ability to certain degree.
Training Dataset
Dataset
MAE]
MSE
SSIMT
FlatFault
FlatFault
14,60
1146.09
0,9895
CurvedFault
FlatFault
50.80
17627.65
Table 7: Quantitative results of our UPFWI models evaluated on FlatFault.
Ground Truth
Prediction
Figure 16: Results on generalization across datasets. The test is performed on FlatFault by apply- ing UPFWI model that is trained on CurvedFault dataset. Although the artifact is not negligible, the fault structures and velocity values are well preserved. This demonstrates that our model has generalizability certain degree.
Ground Truth
Prediction
Obatl
Ground Truth
Prediction
Results on generalizability over geological anomalies. The test is performed on dataset where add additional geological anomalies to simulate lime-lapse imaging problems. The velocity maps containing those anomalies are not included during training. However, our model captures the spatial and temporal dynamics of anomalies in prediction. This demonstrates that model has generalizability to certain degree.
Figure 17:
20


Reviews:
[{"review_id": "kGJ33dx_ag3", "reviewer": "Reviewer_ZRwQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nPropose an unsupervised formulation for FWI\n\nWeakness:\n\n**(1) Sim2Real gap between OpenFWI and seismic inversion:**\n\nOpenFWI obtains the velocity map and raw shot signals through simulation.  In this simulation, OpenFWI assumes uniform acquisition geometry, where the receivers are positioned at each grid with an interval of 15 meters. In real seismic inversion for exploration, the acquisition geometry may be non-uniform. Moreover, dataset such as (Yang & Ma, 2019) includes velocity maps with more complicated salt bodies.  It seems that OpenFWI provides more simplified velocity maps.  Therefore, it remains unknown how models trained on OpenFWI generalize to real scenarios.\n\n**(2) Convolution on raw seismic traces:**\n\nAs shown in the paper, the raw seismic traces could be simulated through forward modeling of PDE. Under the irregular acquisition geometry, the raw seismic traces are not image-like as we do not have receivers in a grid. In these scenarios, convolution operations that require matrix input could not be applied. Therefore, the convolution networks may be restricted to synthetic datasets.\n\n**(3) The role of PDE forward modeling:**\n\nIt seems that the PDE forward modeling is regarded as a blackbox during unsupervised training. Is there any illustration of how the forward modeling helps the training of CNN?\n\n**(4)The novelty of the unsupervised learning paradigm:**\n\nIs the proposed learning paradigm model-agnostic? How does it perform when we switch the backbone model to transformer (Dosovitskiy et al 2020) or MLP (Tolstikhin et al 2021)?\n\n\n\n\nFangshu Yang and Jianwei Ma. Deep-learning inversion: A next-generation seismic velocity model building method. Geophysics, 84(4):R583–R599, 2019.\n\nDosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020.\n\nTolstikhin I, Houlsby N, Kolesnikov A, et al. Mlp-mixer: An all-mlp architecture for vision[J]. arXiv preprint arXiv:2105.01601, 2021.\n", "summary_of_the_paper": "This article studies the Full-Waveform Inversion (FWI) problem. The major contributions of this article could be summarized as\n\n(1) Formulate the FWI in an unsupervised way\n\n(2) Propose a learning strategy with perceptual loss\n\n(3) Introduce a dataset for benchmark\n", "summary_of_the_review": "The topic is of general interest and the paper is mostly well written. However, there exist concerns about a lack of novelty and incremental improvements.\n\nThe authors are strongly encouraged to address these shortcomings by:\n\ni) provide a better justification of the OpenFWI dataset, especially on how OpenFWI bridges the gap between simulation and real seismic inversion\n\nii) motivate the idea Neural Network+PDE  better\n\niii) improve the experimental section by studying if the proposed method is model-agnostic\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RA9Fdo8HeWG", "reviewer": "Reviewer_q6o6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I find their approach very interesting. The paper is also well structured and well written. However, I have some comments and points that I feel authors could consider.\n\nGeneric comments:\n\n- The dataset is generated for fixed measurement setup. Would this be practical or can it be extended for practical measurement procedures? They mention that physics driven approaches can be slow, but a data driven approach is very slow if a new neural network (and possibly dataset) should be generated for each measurement. \n\n- Have authors considered extending their work for semi-supervised setting? Would labels of some samples improve results if included to the loss function?\n\n- The improvement in their results seems to come from the fact that more samples are used (48K vs 24K). This is reasoned as this is unsupervised approach. But still it makes me wonder if somehow the other half has \"better\" more representative samples, i.e. more similar sample to the ground truths in their tests? To verify this, ave authors e.g. tried to switch the unlabelled part to be the labelled part (should be possible as this is simulation)?\n\n\nMore specific comments:\n\n- The wave equation employed in their study assumes elastic medium and does not hold, for example, for porous materials. This is fine for this study, but could be commented. \n\n- Introduction: it could be mentioned that OpenFWI is a simulated dataset \n\n- Introduction, ending: MSE and SSIM values are mentioned, but especially MSE values do not have so much meaning to a reader if there is no comparison, information about typical scale etc. Relative improvement should be fine there. \n\n- Eq (8) and (9): the loss function compares the measurement (/label) to modelled pressure signals. For \"perfect models\", the difference is noise. I am wondering if noise distribution could be somehow taken into account? \n\n- It is not mentioned how dense discretization was used (spatially and temporally)? Did authors check that numerical scheme has converged (e.g. denser discretization does not change results)? Is the same discretization used both in training and test sets? Also to avoid, inverse crime type of thing, it could be beneficial to use different discretization level in training and test sets. This would avoid, for example, a case in which the neural network learns artefacts of the numerical scheme. \n\n- Experiments: Is there any noise included to the numerical results?\n\n- Experiments: how lambdas in Eqs (8) and (9) are chosen?\n\n- Experiments: \"batch size 16\" this sounds quite small especially if 8 GPUs are used for training. Or is this per GPU batch size (in which case total batch size would be 8*16)?\n\n- Discussion mentions \"challenging velocity\", which has not commented in results section. This is bit odd.\n", "summary_of_the_paper": "The paper introduces a novel way to solve Full-Waveform Inversion problem which is a common problem in geological surveys. Their method is based on CNN, giving a reconstruction of velocity field for measured seismic data, and the loss function which connects CNN to a discretized version of the governing partial differential equations (the wave equation). The power of the method lies in unsupervised learning which allows one to use more data without expensive data labelling. This is demonstrated using numerical results using their simulated seismic dataset.\n", "summary_of_the_review": "I am very positive about the paper and would recommend acceptance, but authors could also considers comments/points mentioned above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "P0UR3KDbI20", "reviewer": "Reviewer_vGXF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- Combining the CNN with the physics-based model helps to alleviate the need for ground truth velocity maps, which is convenient.\n\n- The authors introduce a new large-scale dataset which could be of interest to the community working on data-driven solutions to full-waveform inversion (FWI).\n\n- The authors compare to other learning-based approaches and perform an ablation study to evaluate their proposed method.\n\n**Weaknesses**\n\n- There is already a very large body of work on combining neural networks (e.g., CNNs) and physics-based models for solving PDEs, including for FWI. Yet, in the introduction the authors cite only a single paper to put their work in context. I recognize that there is a more extensive review in the related work (at the end), but reading the paper I was confused if the authors are actually claiming the method as a contribution or if they are only applying it to FWI.\n\n- Indeed there are a few papers that ought to be cited which also combine a neural network with the physics-based model for FWI. For example, Mosser et al. (2020) combine a generative model with a physics-based model in order to improve FWI inversion, and He and Wang propose a very similar approach that uses a CNN to output the velocity field, and then they optimize the CNN through the physics-based forward model. This is actually extremely close to what the authors propose; the main difference seems to be that He and Wang perform test-time optimization to overfit to a single scene (so also unsupervised) while the authors propose to train across a dataset. This works should also be discussed, and the authors should clarify what the novelty of the proposed approach is.\n\n    He, Qinglong, and Yanfei Wang. \"Reparameterized full-waveform inversion using deep neural networks.\" Geophysics 86.1 (2021): V1-V13.\n\n    Mosser, Lukas, Olivier Dubrule, and Martin J. Blunt. \"Stochastic seismic waveform inversion using generative adversarial networks as a geological prior.\" Mathematical Geosciences 52.1 (2020): 53-79.\n\n- As with any other learned approach, generalization is likely a challenge with the proposed method. How does the method perform on slightly different datasets? For example can the model that is trained on CurvedFault generalize to FlatFault? Or can these models generalize to standard FWI datasets like the Marmousi dataset or the 2004 BP model (ref below)? These are standard datasets and so these results would likely be of interest to the FWI community.\n\n    Billette, F. J., and Sverre Brandsberg-Dahl. \"The 2004 BP velocity benchmark.\" 67th EAGE Conference & Exhibition. European Association of Geoscientists & Engineers, 2005.\n\n**Typos/Misc.**\n\n\nSection 1\n- It should be clarified that the proposed dataset is a simulated 2D dataset.\n\n- It's not clear what supervised method is being referred to here: \"...26.77% smaller than that of the supervised method\"\n\n- Full Waveform Inversion or Full-Waveform Inversion? Both are used (title, intro text, Section 2 header)\n\nSection 3\n\n\"Physics-informed\", but in section 1 \"Physical-informed\"\n", "summary_of_the_paper": "The paper presents a method for full-waveform inversion (an inverse problem in seismic imaging) that combines a convolutional neural network (CNN) with a physics-based forward modeling operator. As opposed to other work which directly learns the inverse mapping from measurements to the sought-after velocity parameter, the authors propose an \"unsupervised\" approach. Here, the CNN takes the seismic measurements as input and predicts a velocity field. Then, the physics-based forward operator produces measurements which can be compared to the original measurements to train the network. The authors evaluate their method on a new simulated dataset and show that it can outperform other supervised approaches.\n", "summary_of_the_review": "Overall I lean slightly negative on the paper. My main criticism is that the novelty of the approach seems to be low. Moreover, the closest related work that I am aware of does not appear to be cited or discussed. Finally, while the method performs well in simulation, there are no results that clearly evaluate generalization and/or performance on more challenging datasets. I would be open to revising my score if the authors can clearly address these concerns.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an unsupervised method for learning Full-Waveform Inversion in geophysics, by combining a differentiable physics simulation with a CNN based inversion network.

The reviewers agreed that the paper was well written and described an important advance but were concerned about limited novelty and a potential sim2real gap. The authors responded to their critique with significant new experiments and clarified the novelty of their method relative to prior work.

Based on the author responses, I recommend acceptance.
Decision: Accept (Poster)