Towards Physics-informed Deep Learning for Turbulent Flow Prediction
Authors: Rui Wang
Karthik Kashinath
Mustafa Mustafa
Adrian Albert
Rose Yu
Venue:  (2020)

Abstract: While deep learning has shown tremendous success in a wide range of domains, it remains a grand challenge to incorporate physical principles in a systematic manner to the design, training, and inference of such models. In this paper, we aim to predict turbulent flow by learning its highly nonlinear dynamics from spatiotemporal velocity fields of large-scale fluid flow simulations of relevance to turbulence modeling and climate modeling. We adopt a hybrid approach by marrying two well-established turbulent flow simulation techniques with deep learning. Specifically, we introduce trainable spectral filters in a coupled model of Reynolds-averaged Navier-Stokes (RANS) and Large Eddy Simulation (LES), followed by a specialized U-net for prediction. Our approach, which we call Turbulent-Flow Net (TF-Net), is grounded in a principled physics model, yet offers the flexibility of learned representations. We compare our model, TF-Net, with state-of-the-art baselines and observe significant reductions in error for predictions 60 frames ahead. Most significantly, our method predicts physical fields that obey desirable physical characteristics, such as conservation of mass, whilst faithfully emulating the turbulent kinetic energy field and spectrum, which are critical for accurate prediction of turbulent flows.

Under review as a conference paper at ICLR 2020
TOWARDS PHYSICS DEEP LEARNING FOR TURBULENT FLOW PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
While deep learning has shown tremendous success in a wide range of domains, it remains a grand challenge to incorporate physical principles in systematic man- ner to the design, training and inference of such models. In this paper, we aim to predict turbulent flow by learning its highly nonlinear dynamics from spatiotem- poral velocity fields of large-scale fluid flow simulations of relevance to turbulence modeling and climate modeling. We adopt a hybrid approach by marrying two well-established turbulent flow simulation techniques with deep learning. Specif- ically, we introduce trainable spectral filters in a coupled model of Reynolds- averaged Navier-Stoke (RANS) and Large Eddy Simulation (LES), followed by a specialized U-net for prediction. Our approach, which we call Turbulent-Floo Net (TF-Net), is grounded in a principled physics model, yet offers the flexibility of learned representations. We compare our model, TF-Net, with state-of-the- art baselines and observe significant reductions in error for predictions 60 frames ahead. Most importantly, our method predicts physical fields that obey desirable physical characteristics, such as conservation of mass, whilst faithfully emulat- ing the turbulent kinetic energy field and spectrum, which are critical for accurate prediction of turbulent flows.
INTRODUCTION
Modeling the dynamics of physical processes that evolve over space and time and vary over a wide range of spatial and temporal scales is fundamental task in science. Computational fluid dynam- ies (CFD) is at the heart of climate modeling and has direct implications for understanding and predicting climate change. However, the current paradigm in atmospheric CFD is purely physics- driven: known physical laws encoded in systems of coupled partial differential equations (PDEs) solved over space and time via numerical differentiation and integration schemes. These meth- ods are tremendously computationally innteeesii,,,, requiring significant computational resources and expertise. Recently, kata-driven methods, including deep learning, have demonstrated great success in the automation, acceleration, and streamlining of highly compute-intensive workflows for science (Reichstein et al. 2019). But existing deep learning methods underlying physical knowledge incorporated, and are yet to be proven to be successful in capturing and predicting accurately the properties of complex physical systems.
mainly statistical with little or no
Developing deep learning methods that can incorporate physical laws in a systematic manner is key element in advancing AI for physical sciences (Steven Brunton 2019), Towards this goal, we inves- tigate the challenging problem of predicting a turbulent flow, governed by the high-dimensional non- linear Navier-Stokes equations. Recently, several studies have attempted incorporating knowledge about physical system into deep learning. For example, /Emmanuel de Bezenac (2018) proposed a warping scheme to predict the sea surface temperature, but only considered the linear advection- diffusion equation. Xie et al. (2018) and Jonathan Tompson (2017) developed deep learning models in the context of fluid flow animation, where physical consistency is less critical. Wu et al. (2019) and Tom Beucler (2019) introduced statistical and physical constraints in the loss function regu- larize the predictions of the model. However, their studies only focused on spatial modeling without temporal dynamics, besides regularization being ad-hoc and difficult to tune the ppraeepparmetrrss
In this work, we propose a hy learning paradigm that unifies turbulence modeling and deep repre- sentation learning. We develop novel deep learning model, TuubuulentFFlo Net (TF (TF-Net)), that en-

---

Under review as a conference paper at ICLR 2020
hances the capability of predicting complex turbulent flows with deep neural networks. TF-Net ap- plies scale separation to model different ranges of scales of the turbulent flow individually. Building upon promising and popular CFD technique, the RANS-LES coupling approach (E. Labourasse 2004), our model replaces priori spectral filters with trainable convolutional layers. We decom- pose the turbulent flow into three components, each of which is approximated by specialized U-net to preserve invariance properties. To the best of our knowledge, this is the first hybrid framework of its kind for predicting turbulent flow. We compare our method with state-of-the-ar baselines for forecasting velocity fields up to 60 steps ahead given the history. We observe that TF-Net is capa- ble of generating accurate and physically meaningful predictions that preserve critical quantities of relevance. In summary, our contributions are as follows:
1. We study the challenging task of turbulent flow prediction as a test bed to investigate incor- porating physics knowledge into deep learning in a principled fashion.
2. We propose novel hybrid learning framework. TF-Net, that unifies popular CFD tech- nique, RANS-LES coupling, with custom-designed deep neural networks.
3. When evaluated on turbulence simulations, TF-Net achieves 11.1% reduction in predic- tion RMSE, 30. 1% improvement in the energy spectrum, 21% turbulence kinetic energy RMSEs and 64.2% reduction flow divergence in difference from the target, compared to the best baseline.
2 BACKGROUND IN TURBULENCE MODELING
Most fluid flows in nature are turbulent, but theoretical understanding of solutions to the governing equations, the Navier-Stokes equations, is incomplete. Turbulent fluctuations occur over a wide range of length and time scales with high correlations between these scales. Turbulent flows are characterized by chaotic motions and intermittency, which are difficult to predict.
Figure 1: A snapshot of the Rayleigh-Bénane Bénard convection flow, the velocity fields along r direction (top) and y direction (bottom) (Chirila 2018). The spatial resolution 1792 256 pixels.
The physical system we investigate is two-dimensional Rayleigh-Bénne Béngad convection (RBC), a model for turbulent convection, with a horizontal layer of fluid heated from below so that the lower surface at a higher temperature than the upper surface. Turbulent convection a major feature of the dynamics of the oceans, the atmosphere, as well as engineering and industrial processes, which has motivated numerous experimental and theoretical studies for many years. The RBC system serves idealized model for turbulent convection that exhibits the full range of dynamics of turbulent convection for sufficiently large temperature gradients.
Let be the vector velocity field of the flow with components v),, velocities along and directions, the governing equations for this physical system are:
V
Continuity Equation
dw
1
+ (w-
V)w
Vp + w
Momentum Equation
Øt
PO
0T
Ø1 + (w V)T = =V
Temperature Equation
where p and T pressure and temperature respectively, is the coefficient of heat conductivity, Pn is density at temperature at the beginning, a is the coefficient of thermal expansion, / is the kinematic viscosity, the body force that is due to gravity. In this work, we particular approach
2

---

Under review as a conference paper at ICLR 2020
to modeling RBC that uses a Boussinesq approximation, resulting in divergence- flow, V.w should be zero everywhere (Chirila 2018). Figure shows a snapshot in our RBC flow dataset.
CFD allows simulating complex turbulent flows, however, the wide range of scales makes it very challenging to accurately resolve all the scales. More precisely, fully resolving a complex turbulent flow numerically, known direct numerical simulations DNS, requires very fine discretiza- tion of space-time, which makes the computation prohibitive even with advanced high-performance computing. Hence most CFD methods, like Reynolds-Averagee Navier-Stokes and Large Eddy Simulations (McDonough 2007a Pierre Sagaut 2006 McDonough (0076), resort to resolving the large scales whilst modeling the small scales, using various averaging techniques and/or low-pass filtering of the governing equations (Eqn. [ However, the unresolved processes and their interac- tions with the resolved scales are extremely challenging to model. CFD remains computationally expensive despite decades of advancements in turbulence modeling and HPC.
Deep learning (DL) poised accelerate and improve turbulent flow simulations because well- trained DL models can generate realistic instantaneous flow fields with physically accurate spa- tiotemporal coherence, without solving the complex nonlinear coupled PDEs that govern the system (Tompson al. 2017 Maziar Raissi 2019; 2018). However, DL models are hard to train and are often used as "black boxes" " in physical science as they lack knowledge and are very hard to interpret. While these DL models may achieve low prediction errors they often lack scientific consistency and do not respect the physics the systems they model. Therefore, is critical infusing known physics and design efficient turbulent flow prediction DL models that not only accurate but also physically meaningful.
the underlying physics
3 TURBULENT-FLOW NET
Inspired by techniques used CFD to separate scales this multi-scale system, the global idea behind TF-Net is to decompose the flow into three components of different scales with trainable modules for simulating each component. First, we provide a brief introduction of the CFD tech- niques which are built this basic idea.
E(W)
U mmd
Temporal Filter
E (R)
d_eer
D
w
U_net
Spatial Fiftor
E(w)
U_net
120
W
(WW)
Figure 2: Turbulent Flow Net: three identical encoders to learn the transformations the three components of different seales, and shared decoder that learns the interactions among these three components to generate the predicted 2D velocity field at the next instant. Each encoder-decoder pair can be viewed as U-net and the aggregation weighted summation.
Reynolds-averaged Navier-Stokes (RANS) decomposes the turbulent flow w into two separable time scales: aveereee mean flow and fluctuating quantity w'. The resulting RANS equations contain a closure term, the Reynolds stresses, that require modeling, the classic closure problem of turbulence modeling. While this approach is good first approximation to solving turbulent flow, RANS does not account for broadband unsteadiness and intermittency, characteristic of most turbulent flows. Further, closure models for the unresolved scales are often inadequate, making RANS solutions to be less accurate. T here is the moving average window size.
1 [' G(s)w( s)ds w(x, , t) = u xx t) w' t), where w(x ((x = T T-T
(2)
3

---

Under review as a conference paper at ICLR 2020
Large Eddy Simulation (LES) is alternative approach based on low-pass filtering of the Navier-Stokes equations that solves a part of the multi-scale turbulent flow corresponding to the most energetic scales. In LES, the large scales ally expressed as a convolution product by the filter kemne G. The kernel G is often taken to be a Gaussian kernel. 02 is subdomain of the solution and depends on the filter size (Sagaut 200T).
spatially filtered variable w, which is usu-
f G(xlE) 22
w az (x t) + w (xx t), where w(x,t)
(3)
The key difference between RANS and LES is that RANS is based on time averaging, leading to simpler steady equations, whereas LES based on a spatial filtering process which more accurate but also computationally more expensive.
Hybrid RANS-LES Coupling combines both RANS and LES approaches in order to be able to take advantage of both methods (E. Labourasse 2004] Chaoua 2017). decomposes the flow vari- ables into three parts: mean flow, resolved fluctuations and unre- solved (subgrid) fluctuations. RANS-LES coupling applies the spa- tial filtering operator G1 and the temporal average operator G2 se- quentially. We can define w in discrete form with using w" as an intermediate term,
E/K
W
W
W
"" tx, = -Gi(w)= Σ (((
(4)
Figure Three level spectral decomposition of velocity w, E'k) is the energy spectrum and l is wavenumber.
( t) = G2(w*)
Σ (2(8)w((2;) s=t-T
(5)
T
then d can be defined as the difference between w" and w.
v = ~" w,
w' = w w
(6)
Finally we can have the three-level decomposition of the velocity field.
~ + き
Figure 3 shows this three-level decomposition wavenumber space (E. Labourasse 2004 k the wavenumber, the spatial frequency in the Fourier domain. E(k) is the energy spectrum describing how much kinetic energy is contained in eddies with wavenumber eddies that contain most of the energy. The slope of the spectrum is negative and indicates the transfer of energy from large scales of motion to the small scales. This hybrid approach combines the ease and computational efficiency of RANS with the resolving power of LES to provide technique that is less expensive and more tractable than pure LES.
Small corresponds to large
Turbulent Flow Net We describe TF Net, hybrid deep learning framework based on the multi- level spectral decomposition of hybrid RANS-LÉS Coupling method. We decompose the velocity field into three components of different scales using two scale separation operators, the spatial filter G1 and the temporal filter G2. In traditional CFD, these filters are usually pre-defined, such as the Gaussian spatial filter. In our model, both filters trainable neural networks. The spatial filtering process is realized by applying one convolutional layer with a single 5x5 filter to each input image. The temporal filter implemented as a convolutional layer with single Ix] filter applied every I' images. The motivation for this design is to explicitly guide the DL model to learn the non-linear dynamics of both large and small eddies as relevant to the task of spatio-temporal prediction.
We design three identical encoders to encode the three scale components separately. We use shared decoder to learn the interactions among these three components and generate the final prediction. Each encoder and the decoder can be viewed as U-net without duplicate layers and middle layer in the original architecture (Olaf Ronneberger 2015). The encoder consists of four convolutional layers with double the number of feature channels of the previous layer and stride 2 for hown-nampling. The decoder consists of one output layer and four deconvolutional layers with summation of the corresponding feature channels from the three encoders and the output of the previous layer as input. Figure 2 shows the overall architecture of our hybrid model TF-Net. To generate multiple time- step forecasts, we perform one-step ahead prediction and roll out autoregressively. Furthermore, since the turbulent flow under investigation has zero divergence (V w should be zero everywhere), include | u0l22 regularizer constrain the predictions, leading Con TF-Net.
4

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
Turbulence Modeling Recently, machine learning models, especially DL models have been used to accelerate and improve the simulation of turbulent flows. For example, [Lng et al. (2016); Fang et al. (2018) studied tensor invariant neural networks to learn the Reynolds stress tensor while pre- serving Galilean invariance, but Galilean invariance only applies to flows without external forces. external foree. Most recently, [Kim & Lee (2019) studied
In our case, RBC flow has gravity unsupervised generative modeling of turbulent flows but the model is not able to make real time future predictions given the historic data. RRassi et al.] (2017) applied a Galerkin finite element method with deep neural networks to solve PDEs automatically, what they call "Physics-informed deep learning". Though these methods have shown the ability of deep learning in solving PDEs di- rectly and deriving generalizable solutions, the key limitation of these approaches is that they require explicitly inputs of boundary conditions during inference, which are generally not available in real- time. Arvind Mohan (2019) proposed a purely data- DL model for turbulence, compressed convolutional LSTM, but the model lacks physical constraints and interpretability. Wu et al. (2019) and Tom Beucler (2019) introduced statistical and physical constraints in the loss function to regu- larize the predictions of the model. However, their studies only focused on spatial modeling without temporal dynamics, besides regularization being ad-hoc and difficult to tune the hyper- parameters.
Fluid Animation In parallel, the computer graphics community has also investigated using deep learning to speed up numerical simulations for generating realistic animations of fluids such as water and smoke. For example, [Tompso et al. (2017) used incompressible Euler's equation with cus- tomized Convolutional Neural Network (CNN) to predict velocity update within a finite difference method solver. Chu & Thuerey (017)) propose double CNN networks synthesize high-resolution flow simulation based on reusable space-time regions. Xie et al. (2018) and Donathan Tompson (2177) developed deep learning models in the context of fluid flow animation, where physical con- sistency is less critical. [Seefenn Wiewel (2019) proposed method for the data-driven inference of temporal evolutions of physical functions with deep learning. However, fluid animation emphases on the realism of the simulation rather than the physical consistency of the predictions physics metries and diagnostics of relevance to scientists.
Video Prediction Our work is also related to future video prediction. Conditioning on the ob- served frames, video prediction models are trained to predict future frames, c.g.. Mathieu et al. (215); Finn et al.] (2016); Xue et al. (2016); Villegas et al.) (007);; Chelsea Finn (2016). Many of these models trained on natural videos with complex noisy data from unknown physical pro- cesses. Therefore, it is difficult to explicitly incorporate physical principles into the model. The turbulent flow problem studied in this work is substantially different from natural video prediction because it does not attempt to predict object or camera motions. Instead, our approach aims to em- ulate numerical simulations given noiseless observations from known governing equations. Hence, some of these techniques are perhaps under-suited for our application.
EXPERIMENTS
5.1 DATASET
The dataset for our experiments comes from two dimensional turbulent flow simulated using the Lattice Boltzmann Method (Chirila 2018). We use only the velocity vector fields, where the spatial resolution of each image is 1792 256. Each image has two channels, one is the turbulent flow velocity along direction and the other one is the velocity along direction. The physics parameters relevant to this numerical simulation are: Prandtl number 0.71, Rayleigh number = 2.5 x 108 and the maximum Mach number O.I. We use 1500 images (snapshots in time) for our experiments. The task is to predict the spatiotemporal velocity fields up to 60 steps ahead given 10 initial frames.
We divided each 1792 by 256 image into 7 square sub-regions of size 256 256, then downsample them into 64 x 64 pixels sized images. We use a sliding window approach to generate 9,870 samples of sequences of velocity fields: 6,000 training samples, 1,700 validation samples and 2,170 test samples. The DL model is trained using sack-propagatton through prediction errors accumulated over multiple steps. We use validation set for paraamearreesers tuning based on the average error
5

---

Under review as a conference paper at ICLR 2020
of predictions up six steps ahead. The hyper-parameters tuning range be found in Table 2 in the appendix. All results are averaged over three runs.
5.2 BASELINE
We compare model with series of state-of-the-art baselines for turbulent flow prediction.
ResNet (Kaiming He
2015): ) a 34-layer Residual Network by replacing the final dense layer
with a convolutional layer with two output channels.
ConVI.TM (Xingjian Shi 2015
3-layer Convolutional LSTM model used for spatiotemporal
precipitation nowcasting.
U-Net (Ola Ronneberger 2015): Convolutional neural networks originally developed for image segmentation, also used for video prediction.
GAN: U-net trained with a discriminator like the Generative Neural Networks.
SST (Emmanuel de Bezenac (018); hybrid deep learning model using warping scheme for linear energy equation to predict sea surface temperature, which is also applicable to the linearized momentum equation that governs the velocity fields.
DHPM (Raissi 2018): Deep Hidden Physics Model is to directly approximate the solution of partial differential equations with fully connected networks using space and time as inputs. The model is trained twice on the training set and the test set with boundary conditions.
Here ResNet, ConVLSTM, U-net and GAN pure data-driven spatiotemporal deep learning models for video predictions. SST and DHPM are hybrid techniques that aim to incorporate prior physical knowledge into deep learning for fluid simulation.
5.3 EVALUATION METRICS
Even though Root Mean Square Error (RMSE) a widely accepted metric for quantifying the dif- ferences between model predictions and the ground truth, it is still insufficient to apply the predicted turbulent flows with good RMSE to scientific fields. We need to check whether the predictions are physically meaningful and preserve desired physical quantities, such as Turbulence Kinetic Energy, Divergence and Energy Spectrum. Therefore, we include set of additional metrics for evaluation.
Root Mean Square Error We calculate the RMSE of all predicted values from the ground truth for
each pixel, ( w,/2/N.
Divergence Since we investigate incompressible turbulent flows in this work, which means the divergence, V w, each pixel should be zero, the average of absolute divergence all pixels at each prediction step as an additional evaluation metric.
Turbulence Kinetic Energy In fluid dynamics, turbulence kinetic energy is the mean kinetic energy per unit mass associated with eddies in turbulent flow. Physically, the turbulence kinetic energy is characterised by measured root mean square velocity fluctuations, ((u)2 + (v' 2)/2, where (u )2
+ Et (u(t) (u()) 2)² and t is the time step. We calculate the turbulence kinetic energy for each predicted sample of 60 velocity fields.
Energy Spectrum The energy spectrum of turbulence, E(k), is related to the mean turbulence kinetic energy as l E(k)dk: ((')² (2')3)/2
the wavenumber, the spatial frequency in 2D Fourier domain. We calculate the Energy Spectrum the Fourier transformation of the Turbulence Kinetic Energy fields. The large eddies have low wavenumbers and the small eddies correspond to high wavenumbers. The spectrum tells how much kinetic energy is contained in eddies with wavenumber
RESULTS
Figure shows the growth of RMSE with prediction horizon up 60 time steps ahead. Te-Net consistently outperforms all baselines, and constraining with divergence free regularizer can fur- ther improve the performance. We also found DHPM able to overfit the training set but performs

---

Under review as a conference paper at ICLR 2020
2500
Mean Absalute Divergence
Traat
RMSE
1500
1000
SCT
800
I-ret U.e.
500
COnAESTM
600
SET
20
40
60
50
Predition Step
Predition Step
Figure 4: Root mean square errors of differ- ent models' predictions at varying forecast- ing horizon
Figure 5: Mean absolute divergence of dif- ferent models' predictions at varying fore- casting horizon
Target
Con TF-net
TF-net
U_net
GAN
ResNet
ConvLSTM
SST
DHPM
Figure 6: Turbulence kinetic energy of all models' predictions at the leftmost square field in the original rectangular field with respect to the target.
poorly when tested outside of the training domain. Neither Dropout nor regularization techniques can improve its performance. Also, the warping scheme of the Emmanuel de Bezenac (2018) relies on the simplified linear assumption, which was too limiting for our non-linear problem.
Figure shows the averages of absolute divergence over all pixels at each prediction step. TF-Net has lower di- vergence than other models even without additional di- vergence free constraint for varying prediction step. It is worth mentioning that there a subtle trade-off between RMSE and divergence. Even though explicitly constrain- ing model with the divergence- the divergence of the model predictions, it also has the side effect of smoothing out the small scale eddies, which results in larger RMSE..
Target
Untt
Randle
-03
Energy Spectrum
2014
-1.75
regularizer can reduce
2gm
201
Figure displays the turbulence kinetic energy fields of all models' predictions the leftmost square field in the original rectangular field. Figure 7 shows the energy spectrum of our model and best baseline the left- most square sub-field. We also convert square predicted images back the big rectangular ones and calculate the Energy Spectrum on the entire domain, which can be found in Figure 10 the appendix. While the turbulence kinetic energy of TF-Net, U-net and ResNet appear similar Figure [ however, from the energy spectrum in Figure and Figure TO we can that Te-Net predictions are in fact much closer to the target. Extra divergence free constraint does not affect the energy spectrum of predictions. Thus, unlike other models, TF TF-Net able to generate predictions that are physically consistent with the ground truth.
10P
10 Wave Number
Figure 7: The Energy Spectrum of TF Net, U-net and ResNet on the leftmost square sub-region.
Figure shows the ground truth and the predicted u velocity fields from all models from time step O to 60. We also provide videos of predictions by TF-Net and several best baselines in htt ps www.www....ooo.....o com watchh? w http://atcchv//e///HHHH
and hhtpst: /www. yout respectively. We that the predictions by TF -et model

---

Under review as a conference paper at ICLR 2020
the closest the target based on the shape and the frequency of the motions. GAN is able to gen- erate flows with fine- imeegrained details but physics are not captured in a correct way, which also shows the benefit of developing hybrid models with embedded physics knowledge. U-net is the best per- forming data-driven video prediction baseline. N. Thuerey (2019) also found the U-net architecture quite effective in modeling dynamics flows.
We also performed an additional ablation study of TF-net to understand each component of TF- investigate whether the TF-net has actually learned the flow with different scales. The video, https: / www yoou ube. com wat ch chttch?v-s vv=ysdhuUU che includes the predictions of TF- net, and the outputs of each small U-net while the other two encoders are zeroed out. We can see that the outputs of each small u-net are the flow with different scales. During Inference, we ap- plied the trained Te-net to the entire input domain instead of square sub-regions. We observed that the boundaries between square sub-regions in the previous videos have disappeared. We also did the same experiments on an additional dataset (Rayleigh number 10"). TF TF-Net still consistently outperforms the best two baselines, U-net and ResNet, based all four evaluation metrics. The results are shown in Figure 12
the appendix.
Target
Con TF-net
TF-net
net
GAN
ResNet
ConVLSTM
SST
DHPM
T+1
T+2
T+5
T+10
T+20
Figure 8: Ground truth and predicted velocities by all models. From left to right, constrained TF-Net, TF-Net and all the baselines. From top to bottom, predictions from time + to 7 (suppose T is the time step of the last input frame).
7 DISCUSSION AND FUTURE WORK
We have presented novel hybrid deep learning model, TF Net, that unifies representation learning and turbulence simulation techniques. TF-Net exploits the multi-scale behavior of turbulent flows to design trainable scale-separation operators to model different ranges of scales individually. We provide exhaustive comparisons of TF-Net and baselines and observe significant improvement in both the prediction error and desired physical quantifies, including divergence, turbulence kinetic energy and energy spectrum. We argue that different evaluation metrics are necessary evaluate DL model's prediction performance for physical systems that include both accuracy and physical consistency. A key contribution this work is the skillful combination of state-of of-tee-----rrrr turbulent flow simulation paradigms with deep learning. Future work includes extending these techniques to very high-resolution predictions, 3D turbulent flows and incorporating additional physical variables improve the accuracy and faithfulness of physically-ifformed deep learning models.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Michael Chertkov Daniel Livescu Arvind Mohan, Don Daniel. Compressed convolutional Istm: An efficient deep learning framework to model high fidelity 3d turbulence. 00ti::9903.00000 2019.
Bruno Chaoua. The state of the art of hybrid rans/les modeling for the simulation of turbulent flows 99:279-327, 2017. doi: https://doi.or//1.. 1000000000000000000000 10494-017 017-9888---
Sergey Levine Chelsea Finn, Ian Goodfellow. Unsupervised learning for physical interaction through video prediction. arXiv: 1600.0015774 2016.
D. B. Chirila. Towards lattice boltzmann models for climate sciences: The gelb programming language with application. 2018.
Mengyu Chu and Nils Thuerey. Data-driven synthesis of smoke flows with cnn-based feature de- scriptors. ACM Transactions on Graphics (TOG), 36(4):69, 2017.
P. Sagaut E. Labourasse. Advance in rans-les coupling, a review and an insight on the nlde approach. Archives of Computational Methods in Engineering, 11:199-255 226,, 2004.
Patrick Gallinari Emmanuel de Bezenac, Arthur Pajot. Deep learning for physical processes: Incor- porating prior scientific knowledge. (xXiv:5505.0459) 2018.
Rui Fang. David Sondak, Pavlos Protopapas, and Sauro Succi. Deep learning for turbulent channel flow. arXiv preprint arXiv:/8/2.02244. 2018.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, pp. 64-72, 2016.
Pablo Sprechmann Ken Perlin Jonathan Tompson, Kristofer Schlachter. Accelerating eulerian fluid simulation with convolutional networks. In ICML'17 Proceedings of the 34th International Con- ference Machine Learning, volume 70, pp. 3424-3433, 2017.
Shaoqing Ren Jian Sun Kaiming He, Xiangyu Zhang. Deep residual learning for image recognition. 1rXvv:550.00450 2015.
Junhyuk Kim and Changhoon Lee. Deep unsupervised learning of turbulence for inflow generation at various reynolds numbers. arXiv: 1908. 10515, 2019.
Julia Ling, Andrew Kurzawski, and Jeremy Templeton. Reynolds averaged turbulence modeling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155- 166. 2016.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:151.005440 2015.
George E Karniadakis Maziar Raissi, Paris Perdikaris. Physics- innfrre- neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differ- ential equations. Journal of Computational Physicx, 378:686-707, 2019.
George Em Karniadakis Maziar Raissi. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 15-:145, 2018.
J. M. McDonough. Introductory Lectures on Turbulence. Ancchaniial Textbook Gallery, 2007a. htthttt:: http:////wwwwwwweeeeo http:eeeeteeeeetttooossse.
James M McDonough. Introductory lectures on turbulence: physics, mathematics and modeling. 2007b.
L. Prantl Xiangyu Hu N. Thuerey, K. Weibenow. Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows. 1711v:1810.08217,,20 2019.
Thomas Brox Olaf Ronneberger, Philipp Fischer. U-net: Convolutional networks for biomedical image segmentation. 1rXiv:15/2.03388. 2015.

---

Under review as a conference paper at ICLR 2020
Mare Terracol Pierre Sagaut, Sebastien Deck. Multiscale and Multiresolution Approaches in Turbu- lence. Imperial College Press, 2006.
Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equa- tions. Journal of Machine Learning Research, 2018.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learn- ing (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 17Xiv:1111.10561,, 2017.
Markus Reichstein, Gustau Camps- Valls, Bjom Stevens, Martin Jung, Joachim Denzler, Nuno Carvalhais, and Prabhat. Deep learning and process understanding for data-driven earth sys- tem science. Nature, 566(7743) 204, 2019. doi: 10. 0011008//411586-019990111111 URL
Pierre Sagaut. Large Eddy Simulation for Incompressible Flows. Springer- Verlag Berlin Heidelberg, 2001. doi: 110007978--3-66-04441111
Nils Thuerey Steffen Wiewel, Moritz Becher. Latent-space physics: Towards learning the temporal evolution of fluid flow. Computer Graphics Forum, 38, 2019.
Petros Koumoutsakos Steven Brunton, Bernd Noack. Machine learning for fluid mechanics.
arXiv:1905.11075, 2019.
Stephan Rasp Pierre Gentine Jordan Ott-Pierre Baldi Tom Beucler, Michael Pritchard. Enforcing analytic constraints in neural-networks emulating physical systems. arXiv: /0t.:999990091222. 2019.
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating eulerian fluid simulation with convolutional networks. Proceedings of the 34th International Conference on Machine Learning-Volus Volume 70, pp. 3424-3433. JMLR. org, 2017.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv:/706.08033. 2017.
Jin-Long Wu, Karthik Kashinath, Adrian Albert, Dragos Chirila, Prabhat, and Heng Xiao. Enfore- ing Statistical Constraints in Generative Adversarial Networks for Modeling Chaotic Dynamical Systems, arXiv e-prints, May 2019.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempogan: A temporally coherent, volumetric gan for super-resolution fluid flow. ACM Transactions on Graphics (TOG), 37(4):95, 2018.
Hao Wang Dit- Yan Yeung Wai-kin Wong Wang-chun Woo Xingjian Shi, Zhourong Chen. Convolutional Istm network: A machine learning approach for precipitation nowcasting. arXiv:1506.0422 14, 2015.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances neural information pro- cessing systems, pp. 91-99, 2016.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 ADDITIONAL RESULTS
Table displays the number of parameters, the best number of input frames, the best number of accumulated errors for backpropogation and training time for one epoch on 8 v-100 GPUs for each model. We can conclude that our model has significantly smaller number of parameters than most baselines yet achieves the best performance. About 25 historic images are enough for deep learning models to generate reasonable predictions, and ConvLSTM require large memory and training time, especially when the number of historic input frames is large. Additionally, Table 2 displays the
pyper-parameters tuming range of models.
Models
Te-net
net
GAN
ResNet
ConyLSTM
SST
DHPM
"params(10*6)
15.9
25.0
26.1
21.2
11.8
49.9
2.12
input length
25
25
24
26
27
23
#accumulated errors
4
5
5
time for one epoch(min)
0.39
0.57
0.73
1.68
45.6
0.95
4.591
The number of parameters, the best number of input frames, the best number of accumulated errors for backpropogation and training time for one epoch on 8 v-100 GPUs for each model.
Table 1:
H Hyper-parameters Tuning Range
#Accumulated errors for backpropogation
Moving average
Size of the spatial filter (TF-nct) 3-99
Hidden Dimension (ConvLSTM) 5512
Learning rale
Batch Size
#Input frames
#Layers (ConvLSTM)
window size
(TF-net)
1o-1 ~166
128
1~10
1~0
2~110
1
Table 2: Hyper-parameters tuning ranges
Figure Saareejshooe shows the ground truth and the predicted velocity fields over 60 time steps. Similar to the velocity predictions, we observe that the predictions from TF-Net U-net and GAN generate smooth predictions and miss the details of small scale motion. There is still room for improvement in long-term prediction for all the models.
the closest to the target.
We converted square predicted images back to the big rectangular ones and calculated the Energy Spectrum on the entire domain, as shown in Figure TO we can see that TF-Net predictions are in fact much closer to the target on small wavenumbers and more stable on large wavenumbers.
We only found two types of spatial and temporal filters from all trained TF-Net models, with and without the divergence regularizer. The meaning of these learned filters are yet be explored.
We visualized the learned filters in Figure [T]
We also performed the same experiments on additional dataset (Rayleigh number=10"). TF-Net still consistently outperforms the best two baselines, U-net and ResNet, based on all four evalu- ation metrics. The results are shown in Figure 12
A.2 IMPLEMENTATION DETAILS
We adapt SST Emmanuel de Bezenac 2018) to model non-linear turbulent flow. SST successfully infused deep learning model into the solution of the linear energy equation to predict surface temperature. If we make the assumption that the advection term (w V)u in the momentum equation a linear term (c V)u, where e unknown, then we can use separate models predict u and and the inputs of both parts are the same stacked u and v from previous time steps.
For DHPM (Raissi 2018), we approximate both the velocity field two 6-layer neural networks with 512 neurons per hidden layer and use and a 4-layer neural networks with 512 neurons per hidden layer to represent pressure p and an non-homogeneous term f that encapsulates the influence of temperature and viscosity. During the training, we make sure the outputs of these neural networks satisfy the continuity and momentum equations. It is worth mentioning that the DHPM model is supposed to be trained twice, first the training set then on the initial and boundary conditions of the test set. This model can be formulated as below.
Loss |
xil
+ ||V u
|lu + (uñ V)u 2 -fll
11

---

Under review as a conference paper at ICLR 2020
Target
Con TF-net
TF-net
.et
GAN
ResNet
ConvLSTM
SST
DHPM
T+1
T+3
T+10
T+20
T+60
Figure 9: Ground truth and predicted velocities by models, suppose T is the time step of the last input image.
Target
Con not
Rectet
10:2
CorvLSTM
Energy Spectrum
1037
1035
10
10"
10a
107
Wave Number
Figure 10: Energy Spectrums of all models' predictions on the entire rectangular domain.
Where is NN(x,, y, t), and NN(x, y. û, é, Uzz dz. Bg. 0g. Dexx Gex dyy. Uyy)
12

---

Under review as a conference paper at ICLR 2020
620
A28
933
630
615
A65
-8.0
Figure 11: Learned spatial and temporal filters IF-Net
TF-net
0.0225
-B. TF
Unnet
Ue Restett Target
Mean Absolute Divergence
100
0.0175
80
0.0155
RMSE
60
0.0125
40
0.0075
20
0.0050
30
Predition Step
Predition Step
(a)
(b)
Target
TF-net
Target
10-3
net
Energy Spectrum
10
U-net
ResNet
-
10-1
10a
Wave Number
(c)
Figure 12: The performances of TF-net, U-net and ResNet on an additional dataset(Ra 10000). (a): Root mean square errors of different models' predictions varying forecasting horizon, (b): Mean absolute divergence of models' predictions at varying forecasting horizon, (c): The Energy Spectrums on the entire domain, (d): Turbulence kinetic energy fields of three models' predictions.
13


Reviews:
[{"review_id": "HygdS0Bx9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows a method that combines a convolutional neural network with a multi-scale physical computational fluid dynamics (CFD), in the scope of predicting turbulent flows. The authors proposed a new network, TF-Net, that is based on a multi-scale CFD: a temporal and a spatial filters are learned, prior to 3 separate 'encoder' networks, that are then grouped in a unique 'decoder' part. The method is tested on a synthetic example, showing interesting results and comparing with a large set of baselines.\n\nOverall, I found the paper interesting in that it really starts from modeling and CFD aspects, in order to derive a neural network architecture. However, I am a not totally convinced since (i) the authors said that their goal is to emulate numerical simulations given 'noiseless observations': how can this be used on a real case then? and (ii) the developed (complex) architecture is still not sufficient to constrain the physical problem, as they had to manually include a regularization term (on the divergence) in the loss: what is the purpose of the 'physically-based network' then, and how can this method be extended to other dynamics?\n\nThe paper is quite well written, the method compared with a lot of state-of-the-art methods, and performs well on this noise-free problem.\n\nRemarks/ questions:\n- In this paper, we suppose that the equations are known. What if it is not the case (at least partially)? Would it still be applicable? In the reality, there are other components of the dynamics that might be not fully understood, some noise, ect...\n- As said previously, and also as you mentioned in the introduction, the literature 'focused [...] on regularization being ad-hoc and difficult to tune'. But in the end, you also need a regularization term in your loss to prevent a non-zero divergence. So what is different then?\n- The total size of the image is a rectangle composed of 7 square sub-regions, each of them being used separately as the input of the network. But in the result, we do see clearly some boundary effects. Would it be possible to also learn the 'intermediate' squares, in between the n. 1 and the n. 2 for example, and then to reconstruct the full picture with less boundary effects? Otherwise, I see a limitation: since your work is based on known equations without noise, the goal is only the speedup of the computation. But if the results are too degraded, I don't see how it can be used.\n- Do you think that your model is able to learn the underlying physics? If so, do we have access to the latent variables, that would be the state of your system (see Learning Dynamical Systems from Partial Observations, I. Ayed, https://arxiv.org/pdf/1902.11136.pdf)? Such as p and T. \n- Similarly, is there a way to understand each component of your network? Such as (i) represent the spatial and temporal filters learned; (ii) understand the 3 U-nets: to what inputs do they respond, what are their respective outputs (before the 'decoder')? Is it doing what we wanted, are they all useful?\n- Since you are not using recurrent networks, why don't you automatically predict at 60 frames from the start? \n\nSmall remarks:\n- 'quantifies': few times, did not not mean 'quantities'?\n- 'N. Thuery (2019) also found the U-net architecture is quite' -> a 'that' is missing", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HygAAadJ5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors propose use learned spatio-temporal filtering and a convolutional model to predict the behavior of a turbulent fluid flow. Turbulent flow is a very difficult problem, of great interest for engineers and physical scientists, so the topic of the paper is certainly compelling.\n\nThis paper has several significant issues. The baselines the authors compare to are quite weak. Many of them were designed for other purposes, such as video prediction. The authors claim significant improvements (\"64.2% reduction in divergence\") that I believe are calculated in a (unintentionally) misleading way. In fact, I think they missed the most important baseline---the ground truth simulation itself. Unless they can argue that the learned model is superior to the classical simulation in a significant way, it is hard to see the benefit of using something like TF-NET.\n\n## Specific Comments\n\n* Page 2: The claim \"64.2% reduction in flow divergence, compared to the best baseline\" seems misleading. In Figure 5, the constrained TF-NET is as low as ~590 but the ResNet is between ~610 and ~810. I am guessing that the authors meant a 64.2% reduction in *difference* from the Target model, but this should be clarified.\n* Page 4: Is 'T' supposed to appear in the denominator of Equation 3? I would have expected this to be a normalizing factor resulting from integrating the filter G over the whole domain.\n* Page 4: I would expect filters used for LES are symmetric. Are any symmetry requirements being enforced on the learned filters?\n* Page 4: It's not clear what TF-NET is outputting. I would expect it to output the time derivative of the velocity field, but this should be spelled out explicitly.\n* Page 5: Since you are using incompressible Navier-Stokes equations, I don't think the Mach number is relevant. IIUC this is only relevant for the propogation of shock waves in compressible flows.\n* Page 7: It looks as if the Target model has significant divergence. Why is this? Should we expect this much divergence in the ground truth data?\n* Page 8: In homogeneous isotropic turbulent flow, the energy spectrum is governed by Kolmogorov, so we know what the spectrum should look like. Is there an analytic result for RB convection? If so, could you include that on the plot?\n* Page 8: The lines for U-Net and TF-NET are nearly indistinguishable in Figure 7. Could you change them?\n* Page 8: It seems like it would be worth including the ResNet in the energy spectrum plots as well.\n* What did the learned spatial and temporal filters look like? How do they compare to typical 'hand-chosen' filters?\n* IIUC, the training, validation and test data all have identical Rayleigh number. Does the learned model generalize to higher/lower energy? This seems critical to making this sort of model useful.\n* The paper doesn't describe a tuning process for any of the models. I would expect this to lead to significant improvement. In particular, how do the models' performance change as the weight on the divergence loss term is increased?\n\n### Baselines\n* I think the objective should be to show that TF-NET is superior to the ground-truth method in some way. Can it match the results of the ground truth simulator but do it faster, or with fewer resources?\n* The authors say \"we compare our model with a series of state-of-the-art baselines for turbulent flow prediction.\" However, most of these models are intended for video prediction or other tasks unrelated to turbulent flows. I wouldn't expect any of the baselines considered to do perform well in this context.\n* Since TF-NET gets the benefit of a loss related to divergence, I would have expected the other architectures to get the same treatment. In fact, without this extra loss term, the ResNet architecture is competitive with TF-NET.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lIYP2htr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work targets learning flow fields for two-dimensional Rayleigh-Benard convections inspired by hybrid RANS-LES turbulence models. \n\nInternally, the approach employs three U-nets which are each trained to predict a 32 dimensional feature space from three variants of filtered velocity fields. These three feature spaces are (probably) concatenated and translated to an output velocity via a fourth U-net. While the overall architecture is intuitive, details are missing in the text. \n\nI also assume that the network outputs w for timestep t+1, which is the sole input for the next evaluation of the network? A regular flow solver is not used in conjunction with this network, but the results (i.e. sequences) shown are purely inferred by the network? (I guess G2 uses multiple frames - what is used for T here, btw.?)\n\nThese details, plus specifics of each layer should be written out (e.g. in the appendix), together with operations such as the merging of the three encoder outputs to make the work reproduicble. I hope the authors can also clarify these points in the rebuttal. \n\nFor a future version, I'd also recommend to rephrase equations (2) and (3). The split into w, w-bar and w-tilde is not compatible with figure 2. The figure uses the split from equations (6,7), so it would be good to make this clear via the notation.\n\nAs training data, the model uses single RBC data set produced with a lattice boltzmann method. It's a pity only a single case is shown, as the method claims to learn a general turbulence model. Do the authors have a second data set on which they could demonstrate the method? This would show help to show generality of the approach.\n\nThe RBC test case is used to train a nice range of different methods, from a simple ResNet to approaches from previous work, and the results are evaluated with a good range of turbulence metrics. These evaluations show nice improvements, e.g., the RMSEs over time in figure 4 are consistently lower. Unfortunately, it's not made clear which data is evaluated - is this a single training case, or e.g. averaged for the whole test set? Likewise for figure 6 and 7.\n\nVery minor, but I'd recommend to rephrase the last sentence of the fluid animation discussion. Those works are part of computer science, which arguably also counts as science.\n\nOverall, I found the split into temporally and spatially filtered components of the flow field is an interesting one, and it's nice to see how well this seems to work. The paper certainly does not aim for new insight for deep learning methods in general, but provides an interesting application for turbulent flows that is evaluated with a nice amount of detail. If the authors can address the unclear points mentioned above, and maybe include a second test case that is evaluated on a subset of the different models, I think this paper could be included in the ICLR program.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that this is an interesting paper with good results. The authors' rebuttal response was very helpful. However, given the competitiveness of the submissions this year, the submission did not make it. We encourage the authors to resubmit the work including the new results obtained during the rebuttal.
Decision: Reject