Fully Convolutional Approach for Simulating Wave Dynamics
Authors: Mario Lino Valencia
Chris D Cantwell
Eduardo Pignatelli
Stathi Fotiadis
Anil Anthony Bharath
Venue:  (2021)
Keywords: Convolutional neural network
spatio-temporal forecasting
data-driven physics
wave dynamics

Abstract: We investigate the performance of fully convolutional networks to predict the motion and interaction of surface waves in open and closed complex geometries. We focus on a U-Net type architecture and assess its ability to capture and extrapolate wave propagation in time as well as the reflection, interference and diffraction of waves.  We investigate how well the network generalises both to long-time predictions and to geometric configurations not seen during training. We demonstrate that this neural network is capable of accurately predicting the height distribution of waves on a liquid surface within curved and multi-faceted open and closed geometries, when only simple box and right-angled corner geometries were seen during training. We found that the RMSE of the predictions remained of order $1\times10^{-4}$ times the characteristic length of the domain for at least 20 time-steps.

Under review as a conference paper at ICLR 2021
FULLY CONVOLUTIONAL APPROACH FOR SIMULATING WAVE DYNAMICS
Anonymous authors Paper under double-blind review
ABSTRACT
We investigate the performance of fully convolutional networks predict the mo- tion and interaction of surface waves in open and closed complex geometries. We focus U-Net type architecture and assess its ability to capture and extrapolate wave propagation in time as well as the reflection, interference and diffraction waves. We investigate how well the network generalises both long-time predie- tions and to geometric configurations not seen during training. We demonstrate that this neural network is capable of accurately predicting the height distribu- tion of waves geometries, when only simple box and right-angled corner geometries were seen during training. We found that the RMSE of the predictions remained of order 10-4 times the characteristic length of the domain for at least 20 time-steps.
liquid surface within curved and multi-faceted open and closed
INTRODUCTION
Predicting the spatio-temporal dynamics of physical sys- tems is recurrent problem in many areas of science and engineering. well-established process consists de- scribing the physical phenomena by human-engineered mathematical models, which capture our current under- standing of the physical laws governing the systems, but whose complexity may prevent finding analytical solu- tions. Scientists therefore frequently turn numerical solvers to simulate such mathematical models and gener- ate accurate approximations their solution.
The huge progress in machine learning (ML) algorithms and increased availability computational power during the last decade has motivated significant growth in the popularity of data-driven physics. In this field, the inter- polation capabilities of neural networks (NNs) have been mostly used two ways: first, to solve partial differen- tial equations (PDEs) in unsupervised manner (Dis- sanayake & Than-Thien 1994; Lagaris et al. 1998 22000] Raissi al.] 2019) and second, predict the physical dynamics from previous observations without knowledge of the underlying equations (Guo al., 2016 Farimani et al. 2017 Thuerey et al.) 2018] Lee & You 2019). Un- like the first approach, the latter does not lead an ana-
Figure I: Rollouts of our U-Net. It
simulates wave motion fluid sur- face with the possible existence of solid walls vvideo].
lytical representation of the physical dynamics, however, it makes feasible to produce predictions for a diversity of physical domains, boundary conditions and initial conditions without needing to re-train the network. provided that the physical laws are unaltered. Recent studies applying con- volutional neural networks (CNNs) to simulate fluid dynamics have reported a speed-up of up to four orders of magnitude, in comparison to traditional numerical solvers, while keeping similar accuracy (Guo et al. 2016). The major shortcoming of NNs are their often poor generalization to unseen configurations and poor long-time predictions in unsteady simulations.

---

Under review as a conference paper at ICLR 2021
We investigate the application of fully convolutional neural networks to the problem of forecasting surface wave dynamics, the motion of which is described by the shallow water equations, a system of three non-linear PDEs (Ersoy et al. 2017). Computational modelling of surface waves is widely used in seismology, computer animation and flood modelling (Ersoy et al.] 2017] Garcfa--avarro et al. 20T9). Our network learnt simulate range of physical phenomena including wave propa- gation, reflection, interference and diffraction at sharp corners. This kind of NN could supplement or potentially replace numerical algorithms used solve the shallow water PDEs, reducing the in- ference time by several orders of magnitude and allowing for real-time solutions. This has particular relevance in iterative design scenarios and potential applications such as tsunami prediction.
Contribution. We demonstrate that our U-Net architecture is able to accurately predict surface wave dynamics in complex straight-sided and curved geometries, even when trained only on datasets with simple straight-sided boundaries. The generalisation to different initial conditions and longer- time predictions are also evaluated. Additionally, we show how including the MSE of the wave gradient into the loss function significantly reduces spurious oscillations in predicted solutions and may help improve the prediction of the position the wavefronts. Our network is able to simulate wave dynamics four orders of magnitude faster than state-of-the-as spectral/hp element numerical solver (Karniadakis & Sherwin (2013), so it could be an effective replacement for numerical solvers in applications where performance is critical.
RELATED WORK
Physics-informed NNs for solving PDEs. The use of NNs for the solution PDEs has been investi- gated since the early 1990s. Most of the relevant research at that time built on the idea of exploiting the universal function approximator property of multi-layer perceptrons (MLPs) (Dissanayake & Phan-Thien 1994 Dissanayake & Phan-Thie Thien 1994 Lagaris al. 1998). In such an approach, the solutions to the PDES are approximated as MLPs whose only inputs are the spatio-temporal coordi- nates. These MLPs are trained in unsupervised way to satisfy the governing PDEs well as the initial and boundary conditions. The main ad advantage of this paradigm is that the solution obtained in differentiable, closed analytic form, easily usable in any subsequent calculations. Nevertheless, these networks cannot extrapolate to different domains, boundary conditions or initial conditions; making necessary to re-train the network for every slight modification of the problem. These tech- niques inspired the more modern physics-informed neural networks (PINNs) (Raissi et al.) [217]] Yazdani et al. 2018 Raissi et al. 2019 Lu et al. 2019) which include deeper MLPs and random collocation points.
CNNs for simulating steady physics. During the last five years, most of the networks used to pre- dict continuous physics have included convolution layers. For instance, CNNs have been used to solve the Poisson's 's equation (Tang et al., 2018: Özbay al. 2201), and to solve the steady Navier- Stokes equations (Guo et al. 2016 Miyanawala & Jaiman [018] Yilmaz & German] 2017] Farimani al. 2017 Thuerey et al. [0188] [ Zhang et al. 2018). The use of CNNs allows for visual inputs rep- resenting physical information, such the domain geometry the initial condition, and for visual outputs representing the solution fields. In contrast to PINNs, the network predictions are purely based observation, without knowledge of the underlying governing equations. This paradigm has proven to extrapolate well to domain geometries, boundary conditions and initial conditions not seen during training (Thuerey et al. 2018). The evaluation of these networks for prediction is con- siderably faster than traditional PDE solvers, allowing relatively accurate solutions to be predicted between one and four orders of magnitude faster (Guo al., 2016 Farimani al., 2017]. These reasons make CNNs perfect for developing surTogate models, complementing expensive numerical solvers (Guo al. 2016) Miyanawala Jaiman 2018). or for real-time animations (Kim et al., 2019). Our work takes inspiration from Guo et al. (2016) in the use of a binary geometry field to represent the physical domain. Although Guo et al. (2016); Farimani et al. (2017) 7) and Thuerey et al. (0118) proved the generalisation of their networks to domain geometries not seen during training, these unseen domains contain elementary geometrical entities included within the training data. We go one step further by training the network with exclusively straight boundaries and demonstrat- ing the network is able to generalise to domains incorporating boundaries with varying radius of curvature.
2

---

Under review as a conference paper at ICLR 2021
CNNs for simulating unsteady physics. Unsteady physics have also been explored from the com- puter vision perspective (Lee You 2019 Sorteberg et al. 2018: Wiewel et al. 2019; Kim et al, 2019 Fotiadis et al. 2020), although to lesser extent than steady physics. Here, the input to the net- work is sequence of past solution fields, while the output a sequence of predicted solution fields future times. When predicting unsteady phenomena there is additional challenge: keeping the
predictions accurate along time. To address this, [Sorebbrgg et al. 2018); WWewell et al. (2019) and Kim al. (2019) proposed use pesooderpropagattrrdeeooooo
architectures, whereas Lee & You
(2019) and Fotiadis et al. (2020) continued to use incoder-decode architectures similar to those used for steady problems. Inspired by /Fotiadis al. (2020). which showed that feed-forward net- works perform at least well as recurrent networks in wave forecasting, we opt to use U-Net architecture (Ronneberger et 2015) to perform each time-step the simulations.
METHOD
3.1
WAVE DYNAMICS DATASETS
The datasets used during training and testing were generated by solving the inviscid, two- dimensional shallow water equations with Nektar++, high-order spectral/hp element solver (Cantwell et al. 2015). conservative form, these equations are given by hu
lv hvu hu² + +gh//2
hu h
+ + hu? + gh hwv
0. (x
Øt
where y 9.80665 m/s² is the acceleration due to gravity and DD E RR² denotes the domain under consideration. Unknown variables in this system are the water depth h(x,, y. t) and the components of the wwo-dimensional velocity vector u(x:, t) and v(2, v,
imposed two forms of boundary conditions: solid wall boundaries, which result wave reflec- tion and diffraction; and open boundaries, which allow waves to exit the domain. As initial condi- tions we considered droplet, represented mathematically by localized two-dimensional Gaussian superimposed unitary depth:
hg = I exp
( xx) (y y²)3)
where / is set to 0.1 m, C is randomly sampled from uniform distribution between 400 and 1000 and the droplet centre, (Ze, Be), randomly sampled from uniform distribution in D.
(a) Square box (tr.)
(b) Corner (tr.)
(c) Double comeer
(d) Convex circle
07
24
89
11
27
19
(e) Concave circle
(0 B-Splinne
(g) Ellipse ares
Figure 2: Flow domains on our training (tr.) and testing sets. Dimensions in metres.
Each simulation the datasets is associated with a binary geometry field, 2(x, v), which satisfies
0, if (x, DD otherwise.
a(z, v)
(3)

---

Under review as a conference paper at ICLR 2021
Therefore, Ω = inside the fluid domain and f2 1 in the solid boundaries (Guo et al. 2016). This geometry forms an additional imput to the network, required to provide information about the walls location. Figure S shows the geometry field for the seven categories of fluid domains included in the datasets. The table below summarises the training and testing sets. The sequences in each dataset contain 100 snapshots of the height field sampled at intervals of At 0.003 seconds. For full details of the datasets, see Appendix A
Table 1: Training and testing datasets
ID
Dataset
Purpose
Geometry
Initial Condition
Sequences
A
Box.Single_DDa Drop
Training
Figure 2a
Single Drop (eq.
500
Corner SingleeDro _Drop
Training
Figure
2b
Single Drop (eq.
500
C
Drop
Testing
Figure
2c
Single Drop (eq.
200
D
Sonvex.Single.
Testing
Figure
2d
Single Drop (eq.
250
E
Sancave_Singlee Drop
Testing
Figure
2e
Single Drop (eq.
2
500
Spline Single Drop
Testing
Figure
2
Single Drop (eq.
2
200
G
Testing
Figure
2g
Single Drop (eq.
(
200
3.2
U-NET AS A SIMULATION ENGINE
Our neural network is based on a U-Net architecture (Ronneberger et al. 2015), which has been tensively used for amage-to-image translation tasks (Farimani et al. 2017 Isola et al. 2017 Thuerey al. 2018 Fotiadis al.) 2020). This architecture consists of fully convolutional feed-forward encoder-decoder network with skip connections between the encoder and the decoder. In wave dy- namics forecasting, the input sequence and the target share an important amount of information at different length scales, this makes the U-Net a particularly appropriate architecture for our problem. Our U-Net receives six fields as input: the geometry field, Ω and a sequence of five consecutive height fields, {h,, hut1, h,+2, h,+3, h.44). It generates as output prediction of the subsequent height field, h,+5, at the next time point. Hence, each evaluation of the network corresponds to performing a single time-step, and the network re-fed with past predictions to make further pre- dictions. See Appendix B for more details about our U-Net architecture.
3.3 GRADIENT LOSS
The loss function used in the present work is given by
Oh, Oh, Or dr
Oh;
ah; Oy Oy
L = (1 - A)MMM((,, hi) + A MSE
+ MSE
(4)
where À is ryper-parameter and MSE is the mean-squared error. The derivatives in equation were computed using second-order finite differences. This loss function penalises oscillations in the predicted fields and favours smooth solutions. This is especially important for temporal forecasting by consecutively re-feeding the network, since the spurious oscillations are amplified in each new prediction. In the case of wave dynamics, the gradients have a large magnitude on the leading and trailing edges of the wavefront. Capturing such small-scale information enables the network to accurately learn and reproduce the wave speed and width of wavefront.
3.4 TRAINING
Our U-Ne was trained against the simple closed box and open corner geometries shown in Figures 2a and 2b with a single droplet as initial condition. The time step was set to At 0.012 seconds and the spatial resolution was set 128 pix/m. A series of transformations
applied to perform data augmentation and normalisation (see Appendix C). We trained for 500 epochs with the Adam
All dataset are available on to be revealed.
GitHuu repository available to he revealed.
4

---

Under review as a conference paper at ICLR 2021
00010
1,03
Loss
20
00000
MSE Loss
1,02
00006
1.01
00004
1.00
0.99
120
0.90
10
12
34
10
20
Ground
function
Time steps ahead
(a)
(h)
Figure 3: (a) Predictions obtained for the fifth time-step (colour values in metres). (b) RRMSE be-
tween target and prediction vs. time-step into the future.
optimiser and its standard parameters (ingmma & Ba 2015) and the loss function given by equation 4 with À 0.05. The learning rate was set to 10 and decreased by a factor of 10 every 100 epochs. The time origin for the input sequences is not at t O, instead it is randomly selected, and five time-steps are performed before updating the network weights in an auto-regressive fashion.
RESULTS AND DISCUSSION
In this section we evaluate the ability of the trained model to extrapolate to different domain geome- tries and initial conditions. We also assess the quality of long-time predictions.
4.1 GRADLOSS VS MSE
Figure 3a compares the ground truth and two predictions obtained five time steps ahead of the last temporal height frame used as input. The height field in the central image was predicted the U-Net trained with the plain MSE, whereas the prediction on the right was obtained after training with the loss function given by equation (4). Here, the suppression of the oscillations on the wavefront when using the GradLoss instead of the MSE is clearly apparent. To quantify this difference, Figure 36 shows the root-mean- root-mean-squared error (RMSE), averaged over the 500 simulations dataset A (see Table [T) during 20 time-steps for both loss functions. We can conclude that, even for the first predicted time-step, the network trained with the MSE is considerably less accurate. This difference is amplified linearly with each new predicted frame.
4.2 GENERALISATION DIFFERENT DOMAIN GEOMETRIES
Our network proved to generalise to complex domain geometries not seen during training, some including curved walls, when only the closed domain and open-corner domain were used for training (Figures 2a and 2bp. The three columns of Figure 4b depict the ground truth, prediction and the absolute value of their difference every five time-steps (the input height fields are depicted in Figure 44). This Figure shows that our network is able to accurately predict the wave speed and correctly infer the reflection on solid walls and diffraction on sharp edges. This is evidenced by the predicted height field at frame 25 (after 20 time-steps), where we observe that the position and wavelength of the predicted and true waves are coincident. The main discrepancy between ground truth and predictions is the height of the wavefronts diffracted on the edges, which are of lower magnitude in the predicted fields. The reason for this may be that the network is less exposed to wave diffraction than to wave propagation and reflection during training.
The previous domain a straightforward extension of the domain in Figure 2b which was used during training, so the ability of the network extrapolate to this configuration is not unexpected. We now explore the generalisation of the network to curved boundaries, which is potentially more challenging. Figures and depict the inputs and outputs for simulation in the domain depicted By visual inspection, these results suggest that the network is able to accurately predict
in Figure 2d
Links to animations comparing the ground truth and the U-Net predictions can he found in http://tto.orr//0000644////hhhhh 13 13182623.v1
5

---

Under review as a conference paper at ICLR 2021
(a)
Frame Fumber
Framm number
Framne number
Framme number
Framm numberr
Ground truth
Prediction
Truth - Prediction
1.03
1.02
1.01
Frame number 10
1.00
0.98
0.0016
0.0014
0.0012
0.0010
Frame number
0.0008
0.0006
0.0002
0,0000
Frame number
Figure 4: Predictions for previous unseen partially open geometry, incorporating multiple corners. (a) Input sequence to the network. (b) Ground truth (left column), predictions (centre) and absolute difference (right) for such inputs. Colour values in metres [video].
wave reflections circular convex walls, and errors are of the same magnitude to those in Figure 4b. The same conclusion was obtained for circular concave walls.
The network is also able produce good-qualine predictions for configurations involving reflec- tions on walls whose radii of curvature is not uniform, although in these cases more substantial differences between targets and predictions illustrate the good generalisation to a fluid domain surrounded by spline-sided walls. The successful generalisation this kind of domain is likely due the localised support the convolution kernels and the architecture. When the convolutions are applied, the network may interpret curved walls as polygonal walls made up of lots of many straight walls of size equal to the stride size. In that case, the higher the image resolution, the smaller the radius of curvature that could be handled by the network.
be seen. The temporal frames depicted in Figure
4.3 GENERALISATION TO A HIGHER NUMBER OF DROPLETS
Our U-Net demonstrated satisfying accuracy on simulations with two or more droplets in the initial condition. For instance, Figure shows the solution field after 20 time-stemp steps domain surrounded by curved walls. Here, the resemblance between the ground truth and the network prediction evi- dences the proper propagation and reflection of the initial excitations while they interfere with each
shows

---

Under review as a conference paper at ICLR 2021
(a)
Frame Fumber
Framm number
Framn number
Fiamm number
Fremne numberr
(b)
Ground truth
Prediction
Truth - Prediction
1.03
1.02
1.01
Frame number 10
1.00
0.98
0.0016
0.0014
0.0012
Frame number
0.0010
0.0008
0.0006
0.0004
0.0002
0,0000
Frame number
Figure 5: Prediction for previously unseen partially open geometry, incorporating curved bound- ary. (a) Input to the network. (b) Ground truth (left column), predictions (centre) and absolute difference (right) for such inputs. Values in metres [video].
1.02
0.015
0.010
0.005
0.98
0.000
Frame number 25
Figure 6: Ground truth (left column), predictions (centre) and absolute difference (right) after 20 time-step and two initial droplets. Values in metres (video)
other. The height field depicted in Figure 7 was obtained predicting the evolution of four droplets for 20 time-steps. While this prediction far from exact, we can observe that the network reproduces well the complex height patterns originated in this chaotic simulation.
4.4 LONG-TIME PREDICTIONS
Firstly, we would like to highlight that the size of the time-steps performed by our network are considerably bigger than in previous work. For instance compared the work of Fotiadis et al.

---

Under review as a conference paper at ICLR 2021
0.015
1.02
0.010
0.005
0.98
0.000
Frame number 25
Figure 7: Ground truth (left column), predictions (centre) and absolute difference (right) after 20 time-step and four initial droplets. Values in metres [video].
2020), our ratio At/te is four times bigger (we consider the characteristic time scale of the shallow water equation defined te g/).). This challenges the network accuracy, but also decreases the number of time steps required to simulate certain time interval, which decreases the computational time. This another advantage of NN-based solvers, since traditional PDE solvers must keep a low time-step size due to stability constraints.
We assessed the ability the network to make accurate predictions over longer time periods by simulating in domain four times longer. Figure g shows predictions and ground truth after 30, 55 and 80 time-steps. The initial condition is single droplet placed in the centre of the narrow channel. The network is able to predict accurately the speed of the leading wavefront, however, the wave magnitude is not retained and it is not able to maintain the propagation of the wavefronts originating from reflections within the narrower channel, which be seen quickly dissipate. This problem could be addressed by training the U-Net for longer output sequences and possibly increasing the depth of the network.
Frame number 35
1.00
Frame number 60
0.98
Frame number 85
Figure 8: Ground truth (left) and predictions (right) for simulations in channel with sudden expansion. The initial condition is a droplet placed at the middle of the narrower segment. Values in metres.
4.5 BASELINE
U-Net with the proposed by |Fotiadis al. (2020) In their work, they compared a U-Net, a LSTM, ConvLSTM and a Casual LSTM for the task of forecasting surface waves; and they demonstrated the superiority of the U-Net in achieving high accuracy and low computational time. Our U-Net has a smaller depth than the one in Fotiadis et al. (2020), resulting 4.2 times less learnable parameters and faster network evaluations. However, each forward pass of their U-Net returns predictions for the next 20 time-pointe instead of the single time-step performed by our network. As result our simulations are one order of magnitude slower, but also about one order of magnitude more accurate. We trained the U-Net in Fotiadis et al.) (2020) in similar way to ours (see section(3.4). but with the MSE as loss function and predicting 20 frames as they did their original work. Figure 9 shows the RMSE between the ground truth and these two U-Nets predictions for our datasets in Table I We can appreciate that the RMSE our U-Net predictions increases linearly with time, whereas the RMSE of the U-Net in Fotiadis al. (2020) remains approximately constant after the 5th time-point, but considerably higher. The decrease in the RMSE the datasets B, C and D is explained by the wavefronts leaving the fluid domain due the presence of open
We contrast
8

---

Under review as a conference paper at ICLR 2021
boundaries. From these results conclude that the network depth in Fotiadis al. (2020) is
unnecessarily big, and more important, that training in an auto-regressive fashion for five time-steps is better than training for 20 frames predicted in single evaluation of the network.
0,0020
(B)
3.0035
0.0030
Fotiadis al. [A]
0.0015
Fotietls el
3.0025
Ours
(C)
3.020
Fotiadic al. (E)
0.0020
Fotialls et [C]
(D)
Fotiadis al.
0.0005
Fotiads et al. (D)
3,0000
Fotiadis al. (G)
0.0000
step
step
Figure 9: RMSE between target and prediction for the datasets Table
4.6 PERFORMANCE
Performing one time step with our network is 62 times faster than simulating that same time-interval in Nektar++ when the
single-core CPU used. On the other hand, performing the network evaluation on a Tesla T4 results in 21765x speed up with respect to Nektar++ running on the CPU. This improvement, and the low RMSE reported in the different datasets, should be enough to justify the use of our model for fost-solution applications.
5 CONCLUSION
In this work, we investigated the application of fully convolutional deep neural networks for fore- casting wave dynamics on fluid surfaces. In particular, we focused on U-Net architecture with two skip connections, and we trained the network to predict the spatio-temporal evolution of wave dynamics, including: wave propagation, wave interference, wave reflection and wave refraction. We demonstrated that including the MSE between the gradients of the predicted height and the truth height in the loss function significantly reduces spurious oscillations in the solution and helps to predict the position of the wavefronts more accurately. This suggests that loss functions which capture discrepancies in the predicted spatial gradients provide valuable information when training networks forecast wave dynamies, especially over long-time intervals. The domains considered during training only included closed box and a single right-angled corner. However, our U-Net was able extrapolate curved walls with varying radii of curvature. The RMSE remained of order 10-1 times the characteristic length for at least 20 time steps for both the training and testing datasets. When run on GPU, these simulations are around 10 times faster than the equivalent nu- merical simulation used for generating our datasets. These findings highlight the potential for neural networks to accurately approximate the evolution of wave dynamics with computational times sev- eral orders of magnitude smaller than conventional numerical simulation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Chris D Cantwell, David Moxey, Andrew Comerford, Alessandro Bolis, Gabriele Rocco, Gianmarco Mengaldo, Danicle De Grazia, Sergey Yakovlev, J-E Lombard. Dirk Ekelschot, al. Nektar++: An open-source spectral/hp element framework. Computer physics communications, 192:205- 219, 2015.
M. W. G. Dissanayake and N. Phan-Thien. Neural-network--aaee approximations for solving partial differential equations. Communications in Numerical Methods in Eingineerigg 10(3):195- 201. 1994. doi: 10. http9/cccm..401000000 moi/abs//00 1002 cnm 1640100303
com/
M. W.M.G. Dissanayake and N. Phan-Thies Neural-nctwork--usee approximations for solving partial differential equations. Communications in Numerical Methods in Engineering, 10(3): 195-201, 1994. ISSN 10990887. doi: 10. 1000//cm.16401000000.
Mehmet Ersoy, Omar Lakkis, and Philip Townsend. A Saint- Mennt-Venant shallow water model for over- land flows with precipitation and recharge. arXiv preprint 170i 754000 2017.
Amir Barati Farimani, Joseph Gomes, and Vijay S Pande. Deep learning the physics of transport phenomena. arXiv preprint 1709: 02432, 2017.
Stathi Fotiadis, Eduardo Pignatelli, Mario Lino Valencia, Chris Cantwell, Amos Storkey, and Anil A. Bharath. Comparing recurrent and convolutional neural networks for predicting wave propa- gation. ICLR Workshop on Deep Neural Models and Differential Equations, 2020. URL
wtw.bb//2002..o http: http:://axxvv...o 200p
Pilar García-Navarie J Murillo, J Fernández-Pato, I Echeverribar, and Mario Morales-Hemánnee The shallow water equations and their application to realistic cases. Environmental Fluid Me- chanics, 19(5): (155):223-111) 1252, 2019.
Xiaoxiao Guo, Wei Li, and Francesco Torio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 481-490, 2016.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-11344 2017.
George Karniadakis and Spencer Sherwin. Spectral/hp element methods for computational fluid dynamics. Oxford University Press, 2013.
Byungsoo Kim, Vinicius C Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. In Computer Graphics Forum, volume 38, pp. 59-70. Wiley Online Library, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
/ wwwxiv.org/aal /1obb///41... 6980
Isaac E. Lagaris, Aristidis Likas, and Dimitrios I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 5:987-1000, 1998.
Isaac Elias Lagaris, Aristidis C. Likas, and Dimitrios G. Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, (5):1041-1049, 2000. ISSN 10459227. doi: 10. 10.1109/72.870037.
Sangscung Lee and Donghyun You. Data-driven prediction of unsteady flow over circular cylinder using deep learning. Journal of Fluid Mechanics, 879:217-254, 2019.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Karniadakis. Deepxde: A deep learning library for solving differential equations. arXiv preprint arXiv: 2019.
10

---

Under review as a conference paper at ICLR 2021
Tharindu P Miyanawala and Rajeev K Jaiman. A novel deep learning method for the predictions of current forces on bluff bodies. In Proceedings of the ASME 2018 37th International Conference on Ocean, Offshore and Arctic Engineering. June 2018.
Ali Girayhan Özbay, Sylvain Laizet, Panagiotis Tzirakis, Georgios Rizos, and Bjöm Schuller. Pois- son CNN: Convolutional Neural Networks for the Solution of the Poisson Equation with Varying Meshes and Dirichlet Boundary Conditions, 2019. URL http:///axxxvvv....aaa// 08613
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differen- tial equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 10902716. doi: 2018 045
jop.006/...p.2211...0. 10.045. /
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learn- ing (part I): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10500 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejan- dro F. Frangi (eds.), Medical Image Computing and Compute-Assisted Intervention MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing.
William Sorteberg, Stef Garasto, Alison Pouplin, Chris Cantwell, and Anil A. Bharath. Approxi- mating the Solution to Wave Propagation using Deep Neural Networks. In NeurIPS Workshop on Modeling the Physical World: Perception, Learning, and Control, December 2018. URL
Wei Tang, Tao Shan, Xunwang Dang, Maokun Li, Fan Yang, Shenheng Xu, and Wu. Study on Poisson's equation solver based on deep learning technique. 2017 IEEE Electrical Design of Advanced Packaging and Systems Symposium, EDAPS 2017, 2018-Janua:1-3) 2018. doi: 10.
P109EEDAPS.2017.8277000
Nils Thuerey, Konstantin Weissenow, Lukas Prantl, and Xiangyu Hu. Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations Airfoil Flows. AIAA Journal, 58(1):15 oct 2018. ISSN 0001-1452.
doi: 10.2514/ 1405,201 URL http: /ttp:///axii..m wwtp:/i..rrr/b////1//11
08217
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent space physics: Towards learning the temporal evolution of fluid flow. In Computer Graphics Forum, volume 38, pp. 71-82. Wiley Online Library, 2019.
Alireza Yazdani, Maziar Raissi, and George Karniadakis. Hidden fluid mechanics: Navier-stokes informed deep learning from the passive scalar transport. Bulletin of the American Physical Society, 63, 2018.
Emre Yilmaz and Brian German. convolutional neural network approach training predictors for airfoil performance. In 18th AIAASISSMO multidisciplinary analysis and optimization conference, pp. 3660, 2017.
Yao Zhang, Woong Je Sung, and Dimitri N Mavris. Application of convolutional neural network to predict airfoil lift coefficient. In 2018 A/AA/ASCE/AHIS/SSC Structures, Structural Dynamics, and Materials Conference, pp. 1903, 2018
11

---

Under review as a conference paper at ICLR 2021
A DATASETS
Datasets A-C contain only straight-sided fluid domains. Dataset only includes wall boundary conditions, whereas B and C include open boundaries. Datasets D-G contain curved-sided domains. The domains in dataset F were generated randomly given four random control points and using B- Splines to create closed domain. The domains dataset were also generated randomly given concave quarter of an ellipse and a convex quarter of an ellipse, whose minor axes follow uniform distribution between 0.25 and 0.5 m. Only the datasets A and B were used during training. The remaining datasets were used to demonstrate the ability of the network to generalise to unseen fluid domain geometries.
B U-NET ARCHITECTURE
The diagram in Figure TO depicts the U-Net architecture used in the present work. The network re- ceives six fields as input: the geometry field Q and sequence of five consecutive height fields (hg.ha+1, h242211944:100 het3, het1)- The output is prediction of the subsequent height field hy45- Whereas some recent studies (Fotiadis et al., 2020; Thuerey et al. 2018) have used bi-linear in- terpolation perform the up-sampling, we opted and stride 2. This increases the number of trainable parameters to 1,864,577, but we also noticed a significant improvement in the quality of the predictions.
transpose convolutions with 2x2 kernel
Com ReLU
Mex pool 2x2
Up-corv
256
Skp
Figure 10: Our U-net architecture with 1,864.577 trainable parameters. It receives six fields as input: the geometry field and a sequence of five consecutive height fields. The output is prediction the height field in the subsequent time step.
C NORMALISATION AND DATA AUGMENTATION
To improve generalisation across a range of wave dynamics, the height fields were re-scaled accord- ing h +- (h / //mxx()x) h)), where for visualising the network predictions. In order to avoid over-fitting and improve the generalisation capabilities the network, we apply two sets of transformations to the training datasets. For dataset A such transformations consist random rotations of 90, 180 and 270 deg as well as hori- zontal and vertical flips. For the dataset B random rotations by multiples of 15 deg are applied and the physical size of the frames is reduced to X by cropping the original frames to domains whose center position follows an uniform distribution from 0.9 to 1.1 m in both spatial directions. Finally, the images of all these sequences satisfy the 128 pix/m requirement. Regarding the testing datasets, sets E, F and G were augmented in the same manner as the dataset A, since they contain only closed boundaries; and datasets C and D are augmented like B, as they include open boundaries.
and max(h) (h) 1.1. This re-scaling was reversed
linearly interpolated to 128 128 resolution
12


Reviews:
[{"review_id": "kiSDIXGAtOb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a methodology for simulating wave dynamics is presented based on convolutional neural networks. A standard analytic wave dynamics solver was used to generate a large dataset of 2D wave simulations. A deep net based on U-Net was trained to predict the next state of the wave field given the five previous states. The training was done with both standard MSE loss and also GradLoss, which placed a loss on the gradients of the field as well. The results show that the network trained with GradLoss performs significantly better over successive rollouts than the one trained with MSE. The results also show that the trained network is able to generalize to qualitatively different environments from the training set.\n\nIf there was one word I would use to describe this paper, it is thorough. The introduction and related work do a good job going over related contributions in the literature and giving the reader, particularly a reader with an ML background and not necessarily a fluid dynamics background, a good understanding of what has been done to this point. The same can be said for the results. The paper offers thorough analysis of the technique. The results first show why GradLoss is necessary. They then go on to show the performance of the model in a variety of scenarios qualitatively different than the scenarios seen during training, including curved surfaces and multiple droplets. The model performs well in these scenarios. Additionally, the paper presents a scenario that the model fails at (long term predictions with two droplets in a narrow channel) and briefly discusses why the model fails. The strength of this paper is in the thoroughness of its results, which includes presenting results where the model fails.\n\nThe weakness of this paper is that it doesn't present any novel techniques. It's an existing architecture (U-Net) applied in a new domain (wave simulation). However, experimental papers provide value as well, and the field is grossly lacking in them. To that end, this paper could be a useful contribution. The results are thorough and even show the limitations of the model. For a full experimental paper, it would be nice to do even more analysis, although with limited number of pages that may not always be possible. However I would have liked to see have seen data showing how much faster this model is over the standard solvers. The paper claims in the intro and conclusion that their model is 10^4 times faster than the standard method, but no where in the paper is data provided to back this up. This seems like one of the main reasons for using this methodology, so not having data to show this is a big oversight. Still overall, while there is room for improvement, this is a quality paper and should be accepted.\n\nA couple small things to fix up for the camera ready version:\n-A related paper that should probably be cited \"Accelerating Eulerian Fluid Simulation With Convolutional Networks\" that does very similar things with deep nets and fluid dynamics.\n-Fig 7 says predictions are on the left and ground truth is on the right, but I believe that is flipped (predictions are on the right, ground truth is on the left).", "extra_scores": {}, "extra_text": {}}, {"review_id": "SpTcO33ccM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper applies a fully convolutional U-Net model for next step prediction of the height field for 2d wave dynamics. On the domains tested, the predictions remain accurate for 20 time-steps and the method seems to provide considerable speed-ups compared to a state-of-the-art spectral/hp element numerical solver. Some encouraging results on generalization to new domain shapes and larger scale domains are also presenting in the experimental session. \n\n\nStrong points:\nThe manuscript is very well written, the methods are clear and the generalization results are encouraging. \n\nWeakness: \n1) The paper lacks a novel contribution from the architectural and application side: UNets have been previously used on forward dynamics predictions (e.g. [1] Thuerey et al.) and there are other works using neural networks on wave prediction (e.g. [2] Fotiadis et al.). The architecture is a straightforward application of a Unet architecture + a loss function with MSRE of the predictions and its gradients.\n\n2) There are no comparisons to other baselines. The code for the UNet architecture from [1] Thuerey et al. is open sourced, so this would be a natural baseline to include. \n\n3) The MSRE errors are not reported on the full dataset, instead plots for single trajectories are presented.\n\n4) Many of the modeling choices are not well motivated or explained, for example\n\na) the training set dt=0.03s and for unrolling it's set dt=0.12s, but no insight on this choice was provided. \nb) a history of the previous 5 time-steps is used as an input to the network, no ablation or justifications are provided.\nc) in the training details, authors state \"The time origin for the input sequences is not at t = 0, instead it is randomly selected, and five time-steps are performed before updating the network weights\" but nothing else is discussed.\nd) authors claim that their model is 4 order of magnitude faster than a state-of-the-art spectral/hp element numerical solver, but the actual runtimes per dataset are not provided, and there is no discussion of terms of scalability, e.g, how does it scale with the space resolution. \ne) In the generalization to a large domain experiment (section 4.4), the authors justify the relatively poor performance of the approach by stating \"This problem could be addressed by training the U-Net for longer output sequences and possibly increasing the depth of the network.\" Since this is a simple hypothesis, I feel it should have been tested. \n\n5) Most experiments are showing results for 20 time-steps unrolls, and while for longer sequences dissipation was observed -- which is a strong limitation of the model. \n\n\nBecause of the points above, I think this manuscript does not pass the ICLR threshold for acceptance, although I believe this would be a good workshop paper submission. \n\n[1] Nils Thuerey, Konstantin Weissenow, Lukas Prantl, and Xiangyu Hu. Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations of Airfoil Flows\n[2] Stathi Fotiadis, Eduardo Pignatelli, Mario Lino Valencia, Chris Cantwell, Amos Storkey, and Anil A. Bharath. Comparing recurrent and convolutional neural networks for predicting wave propagation", "extra_scores": {}, "extra_text": {}}, {"review_id": "CQ0h1kfg1MV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary and Contributions\nThe paper proposes a generative model for the motion of surface waves in open and closed geometries. While neural networks have been applied to simulate fluid dynamics before, they are purported to suffer from poor generalization to unseen geometries for long-time predictions. The paper proposes a U-net based model which is trained using a modified loss function incorporating the gradient into the loss function.  The results demonstrate generalization to unseen geometries with good predictions upto 80 time steps in the future. \n\n### Detailed Review \nThe following is the detailed review of the paper, organized into strengths and weaknesses subsections. \n\n### Strengths \n\n#### Relevance and Significance \nThe utilization of DNNs for modeling physical phenomena is compelling and gaining steam. In particular, the modeling of spatiotemporal phenomena like unsteady fluid dynamics for complex geometries should be of interest to the ML community.\n\n#### Clarity \nThe paper Is written well and is easy to understand. \n\n#### Reproducibility \nShould be reproducible.  \n\n### Weaknesses \n#### Relation to Prior Art \nThe paper does a reasonable job of presenting the prior art but fails at identifying the unmet need that the presented work fulfills. The claimed speedup over numerical solvers has been achieved before (Guo et al, 2016). Further, DNNs have been used to model surface waves (Fotiadis et al, ICLR Workshop 2020, Sorteberg et al, NeurIPS Workshops, 2018, Kim et al Eurographics 2019,  etc.). (Sorteberg 2018) claim to use LSTM for modeling up to 80 time steps into the future on a dataset not seen during training. It is not clear what advantages the present work is supposed to have over the SOTA and why. \n\n#### Methodology\nThe proposed approach is a straightforward application of U-net to predict a spatial field given past few spatial fields (stacked together). However, U-Nets, LSTMs, conv-LSTMs and other architectures have been tried before. It is unclear what the novel contribution in this paper is (gradient augmented loss function?) and why it would be instrumental in handling unseen geometries over longer periods of time.\n\n#### Novelty \nThe presented approach is a straight-forward application of known techniques. Further, these approaches have been tried before (see prior art) \n\n#### Empirical Evaluation \nThere is no evaluation against the state of the art. It is important to compare the performance against (Fotiadis et al, ICLR 2020 Workshops),  (Sorteberg et al, NeurIPS Workshops, 2018), (Kim et al Eurographics 2019),  etc. \n\n### Assessment \nThough the problem seems relevant and of significance to the research community, the paper suffers from a lack of novelty and a non-existent comparison against the state of the art. It fails to identify the unmet needs it is addressing and what novel contributions allows them to achieve this. I do not recommend the publication of this paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BkN1KevZyJ8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors use a U-Net architecture network to predict the motion and interaction of surface waves in an open and closed complex. The network trained on data with a simple box and right-angled corner geometries and generalizes well to other complex geometric configurations. The neural network-based method runs much faster than the standard numerical simulation by directly solving the PDE.  \n\nStrengths: \n1. Much faster than directly solving the inviscid, two-dimensional shallow water equations using standard numerical solver.\n2. The network can generalize to geometric configurations and initial conditions not seen during training.\n\nWeaknesses:\n1. The predictions over more extended periods are not very accurate. Although training the U-Net for longer output sequences and increasing the network's depth may help, this may also increase the difficulty in training. \n2. Some previous works also used the U-net to predict wave dynamics as [3]. It is not clear what is the novelty (if any) in the proposed network architecture.\n3. Not enough Experiments. How does the model generalize with more complicated initial conditions, for example, five or ten droplets?  Furthermore, there is no comparison to other existing work.\n\nA further question:\nThe model generalizes well to other boundary geometries. Is this because the wave reflection at the boundary is a local phenomenon, so the convolutional network only needs to learn the local reflection? \n\n\nSome relative works: \n[1] Zhu, Weiqiang, Yixiao Sheng, and Yi Sun. \"Wave-dynamics simulation using deep neural networks.\" (2017).\n[2] Sorteberg, Wilhelm E., et al. \"Approximating the solution to wave propagation using deep neural networks.\" arXiv preprint arXiv:1812.01609 (2018).\n[3] Fotiadis, Stathi, et al. \"Comparing recurrent and convolutional neural networks for predicting wave propagation.\" arXiv preprint arXiv:2002.08981 (2020).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviews were somewhat mixed here, but the consensus is to reject, with at least one voice (R2) urging rejection. Across reviewers, the recommendation to reject is primarily based on the level of originality with the proposed U-Net architecture and on weakness of experiments, especially in comparing to baselines.

Reviewers found strengths in the paper's writing and in its demonstration of generalization to unseen geometries.

However, reviewers noted that the architecture does not win originality/significance points (including R3, the most positive reviewer):
* R3: "The weakness of this paper is that it doesn't present any novel techniques. It's an existing architecture (U-Net) applied in a new domain (wave simulation)."
* R2: "The proposed approach is a straightforward application of U-net to predict a spatial field given past few spatial fields (stacked together). However, U-Nets, LSTMs, conv-LSTMs and other architectures have been tried before. It is unclear what the novel contribution in this paper is [...] and why it would be instrumental in handling unseen geometries over longer periods of time."
* R2 post-response: "This paper is a clear reject. None of the contributions are novel [...]"
* R4: "The paper lacks a novel contribution from the architectural and application side"
* R1: "Some previous works also used the U-net to predict wave dynamics [...] It is not clear what is the novelty (if any) in the proposed network architecture"

Reviewers also noted weaknesses in the experiments (acknowledged by R3, the most positive reviewer, though that review did not consider them a fatal flaw):
* R1: "Not enough Experiments. How does the model generalize with more complicated initial conditions, for example, five or ten droplets? Furthermore, there is no comparison to other existing work."
* R2: "There is no evaluation against the state of the art [...]"
* R2 post-response: "Application of DNNs to this problem, speed-ups over numerical solvers, etc. have all been explored by SOTA works which have not been compared against. There is no clear articulation of the claimed novel contributions over the SOTA and empirical validation (or theoretical reasoning) of the same."
* R4: "There are no comparisons to other baselines [...] "
* R3: "Reviewer 4 brings up some fair points [about experimental issues]. I'm not as concerned about the lack of a baseline comparison; that doesn't seem to be the point of this paper [...] there is only so much that can be done in an 8 page conference paper [...] However, given that the other 2 reviewers think the paper could use more work, it would be completely reasonable for the chairs to reject it based on those reviews."

Based on this consensus of reviews, my recommendation is to reject. I hope the feedback from the reviews is helpful to the authors.
Decision: Reject