Capturing Extreme Events in Turbulence using an Extreme Variational Autoencoder

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-06 08:31:41

Abstract: Turbulent flows are characterized by intense generation of turbulent kinetic energy through nonlinear physical processes which cascade from the large- to small-scale structures in a forward energy cascade, which is chaotic in nature, and statistically intermittent. Using a recently developed extreme variational autoencoder (XVAE), the turbulent flow fields are replicated to a high order of accuracy. In this extended abstract, we demonstrate XVAE as a powerful alternative to the classical Proper Orthogonal Decomposition (POD) technique for reconstructing large-eddy-simulation (LES) data for scalar temperatures from a buoyant turbulent field at a high Reynolds number of $10^{10}$.

Main paper: 
 introduction: 
 Simulating turbulent flows numerically becomes computationally prohibitive as the Reynolds number increases. Proper Orthogonal Decomposition (POD) has been the most successful statistical method, extracting dominant energetic structures from turbulent flow fields as eigenfunctions of the two-point correlation tensor [Berkooz et al., 1993] . However, POD neglects the intermittency and also extreme turbulence events in the flow. This limitation leads to inaccuracies and information loss in the emulation process.Variational autoencoders [VAEs; Kingma and Welling, 2013] offer an alternative, encoding data inputs as latent variables for generating new output realizations. However, traditional VAEs use Gaussian distributions and do not naturally account for extremes. We introduce a novel framework that incorporates dependent extremes within a VAE engine, allowing for flexible modeling of extremal dependence and efficient uncertainty quantification (UQ). Specifically, we develop a flexible max-infinitely divisible (max-id) process embedded in the VAE's structure (XVAE), enabling a robust emulation of complex turbulent flow systems [Zhang et al., 2023] .In this study, we apply both POD and XVAE to emulate turbulence flow from a high-resolution large-eddy simulation (LES) at the x-z plane of a fixed y level. We also extend the XVAE to allow for the emulation of 3D indexed data so we can model and emulate the evolution of turbulent plumes in all three different directions. We evaluate the methods based on their ability to emulate full data and joint tail behavior, with XVAE showing particular strength in UQ for extreme events. via LES with active scalar transport equations, validated for buoyant plumes [Bhaganagar and Bhimireddy, 2020, Bhimireddy and Bhaganagar, 2021] .The simulation involved a heated, axisymmetric source with diameter D in a quiescent environment.The LES model solves the compressible Euler equations in flux formulation, conserving mass, momentum, and energy; see Bhaganagar and Bhimireddy [2020] for details of the LES methodology and the validation. In this study, the LES coupled with WRF is used to simulate the Mosquito wildland fire (California, September 2022) using realistic boundary conditions from NOAA's HRRR model. The plume is released from a 400m diameter circular source at (39.006 • N, 120.745 • W).The background atmospheric velocity profile, temperature profile and surface heat flux, were obtained from the WRF-LES simulations initiated at 11am on September 09.The LES domain measured 8 km × 8 km × 5 km with a uniform Cartesian grid (198 × 198 × 500 nodes) . Periodic boundary conditions were set on the sides, and constant-pressure at the top. At the center of the bottom boundary, the plume source had a prescribed buoyancy flux of 1.0710 4 m 4 s 3 , with no initial momentum, driven purely by buoyancy. Simulations ran for 50 minutes with data recorded every 30 seconds (i.e., 100 times in total). Planar data were extracted from the x-z plane at y = 99 grid location (i.e., 4 km), the centerline of the plume. 
vae with spatial extremes: 
 The turbulent buoyant plume exhibits high spatial irregularity, temporal volatility, and significant spatial clustering of extremes with anisotropy. Accurate spatio-temporal modeling is challenging due to its high-resolution grid. Therefore, we need to carefully model the tail of the joint distribution while ensuring computational efficiency. Zhang et al. [2023] extended the max-id process, and developed a model to allow for both short-range asymptotic dependence along with mid-range asymptotic independence, and long-range exact independence. They were able to emulate the extreme values in Red Sea sea surface temperature in very high dimensions for the first time.Denote the observed turbulent plume at t by {u t (s) : s ∈ S}, in which S may either be the 3D domain of 8 km × 8 km × 5 km or the 2D centerline; see Figure 1 . In our study, we marginally transform the data so the input data conform to the max-id model assumptionsX t (s) = 1 -u t (s) × 10 5 .(1)Then {X t (s) : s ∈ S, t = 1, . . . , 100} are modeled as follows:EQUATIONwhere t (s) is a noise process with independent Fréchet(µ, τ, 1/α 0 ) marginal distributions, wherex > 0, τ > 0 and α 0 > 0. Then, Y t (s) is constructed using a low-rank representation: for more specific forms of an expPS(α, θ k ) distribution.EQUATIONThe hierarchical construct of our model ( 2)-( 3) allows the embedding of the latent variables Z kt , k = 1, . . . , K in encoded space of a VAE so we can encode and decode the turbulent plume fields while minimizing reconstruction error. The encoder and decoder are multilayer perceptron (MLP) neural networks that optimize parameters to retain maximum information and replicate the joint tail properties of the original field (i.e., to minimize the evidence lower bound). We refer to this VAE, integrated with the max-id process, as XVAE.Once the XVAE is trained and its weights and biases are optimized, posterior simulation of new latent variables Z t for the observations X t at time t can be performed efficiently. Synthetic data X * t can then be generated rapidly by passing these latent variables through the decoder and sampling from the model specified by ( 2)-(3). To transform X * t back to the original scale of the turbulence plume, we apply an inverse marginal transformation: u * t = (1 -X * t )/10 5 , yielding the final emulated output. Refer to Zhang et al. [2023] for an explicit architecture of the XVAE. 
results: 
 For both the 2D and 3D emulation tasks, we pre-specify a dense grid of K knots {s 0 1 , . . . , s 0 K } in the domain S ∈ R 2 or R 3 . The kth basis function is the Gaussian radial basis function, i.e., ω k (s) ∝ exp{-||s -s 0 k || 2 /c} for s ∈ S, k = 1, . . . , K. We follow Algorithm 1 in Zhang et al.[2023] to choose the best number of knots K and shape parameter c, Figure 2 displays emulation results at time 50 using the XVAE for both 2D and 3D LES. Generally, XVAE closely replicate the spatial patterns of turbulence across all times. More importantly, multiple model evaluation metrics, such as the tail RMSE and the spatially-pooled QQ-plot, show good performance in fitting and emulating both the full range of data and the joint tail behavior.In addition, we conduct a comparative study in the 2D case to evaluate the emulation performance of XVAE against POD, commonly used to capture dominant structures in 2D turbulent flows. We compute an extremal dependence measure from spatial extremes literature:EQUATIONin which p ∈ (0, 1) and F i and F j are the marginal distribution functions for u i and u j , respectively.Assuming stationarity and isotropy, we simplify χ ij (u) to χ d (u), with d = ||s i -s j || denoting the distance. The metric χ d (p) is then empirically estimated for each data set (observed or emulated). XVAE aligns closely with observed data at both distances, while POD tends to overestimate extremal dependence at sub-asymptotic levels (p ∈ (0.93, 0.99)), likely due to oversmoothing.To investigate how the marginal distributions at different locations evolve over time, we generate 1,000 emulated samples of {X t : t = 1, . . . , n t } using XVAE. For each location and time point, we thus obtain 1,000 replicates to assess variational Bayes-based marginal uncertainty. Figure 4 shows the marginal distributions of log(X it ) for i = 500, 3861, 17323, 80017, 83014 at t = 1 and 100.The plots reveal how the density shapes change over time at each location, offering sensible UQ for understanding the range and likelihood of turbulence intensity at specific locations and times. 
conclusion: 
 The proposed XVAE method improves the emulation of turbulent buoyant plumes, especially in capturing joint extremal behavior. This approach has significant potential for enhancing predictive models of extreme events. Moreover, the XVAE allows for parameter estimation and UQ within a variational Bayesian framework. This helps in assessing how sensitive the LES is to changes in input parameters and in evaluating the probability of extreme clusters, such as those involving the transport of hazardous or contaminant gases in the atmosphere, essential for improved risk management and more robust decision-making. 
 
 Appendix: Processing Systems (BDU at NeurIPS 2024). Do not distribute. 



Reviews: review: title: title: A paper about using VAEs to forecast turbulent flows with a focus on the role of rare events
main_review: main_review: This submission proposes a VAE extension designed for modeling turbulent flows. An important part of this is sensibly handling non-Gaussianity, since correctly accounting for the role of rare events is important due to non-linear behavior affecting statistical properties of turbulent flows. For this, the authors study a purpose built VAE extension where Gaussianity in latent space is replaced with a significantly more complicated distribution designed to handle rare events.

I am not an expert on the physics / weather modeling side and cannot offer feedback beyond surface level on those aspects of the work. However, I am an expert on the ML side, and I think the the authors' focus on modeling tail behavior through a domain specific non-Gaussian extension of VAEs makes sense. This kind of model would not have been very explored historically in this domain due to the usual objections involving non-convexity of the resulting optimization problems, so it makes sense to explore it now as part of a workshop paper.

The main weakness of the work is its fit to the workshop. While VAEs can in principle be seen as a Bayesian approach, this connection is substantially more shaky than other topics listed in the call for papers. The flavor is much less Bayesian learning and uncertainty, and much more deep representation learning with a carefully constructed latent space. However, the workshop's list of topics includes "Forecasting for dynamical systems" which this paper one could argue is about.

On balance of the tradeoffs I vote to accept.
title: title: Promising XVAE Method Shows Potential, but Requires Enhanced Comparative Analysis, Model Specifications, and Dataset Transparency
main_review: main_review: The paper demonstrates several commendable aspects:
1. The writing is clear and well-structured.
2. The proposed XVAE method is innovative and shows promise.
3. The visualizations are both effective and intuitive.
4. XVAE exhibits superior performance, particularly in capturing extreme events, when compared to the traditional Proper Orthogonal Decomposition (POD) technique.

Areas for Improvement
1. The paper would benefit from a more comprehensive comparative analysis. Include comparisons with recent state-of-the-art models to provide a more current context. The POD technique in 1993 is relatively outdated. Comparing XVAE with more contemporary methods would strengthen the paper's relevance and impact.
2. To enhance reproducibility and clarity, the paper should include the number of knots used in the best-performing model and the shape parameters for the optimal training result. These details are currently absent but are crucial for a thorough understanding of the model's architecture and performance.
3. The paper would be improved by providing a detailed description of the dataset used in the study, including the dataset as an attachment or providing a link to access it.