Optimizing Computationally-Intensive Simulations Using a Biologically-Inspired Acquisition Function and a Fourier Neural Operator Surrogate

Authors: john p lins
wei  liu
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-01-20 07:04:19

Abstract: Computational modeling of physical phenomena has enabled researchers to acquire insight that was previously only observable from costly real-world experiments. The adoption of physics simulations has resulted in numerous advancements in fusion energy, seismic inversion/monitoring, and national defense. However, optimizing simulation studies requires many realizations of the intensive simulations. Manually tweaking control parameters and searching for optimal results can be tedious and inefficient. To tackle such obstacles in simulation studies, we found that differential evolution combined with the covariance matrix adaptation strategy could effectively optimize simulations while simultaneously behaving as an acquisition function to collect samples. The samples collected may be used to train intelligent surrogate models such as a Fourier neural operator (FNO). Once a surrogate is constructed, it could be used to accelerate the sampling of the optimization search space further. This methodology effectively optimized a hydrodynamic simulation modeled by systems of partial differential equations; it may also be extended to simulation optimization in other disciplines.

Main paper: 
 introduction: 
 Idea Optimizing the control parameters of certain physics simulations needs to be approached as a black-box optimization problem when gradient information of the loss landscape cannot be computed from the simulation itself. To work around this issue, we designed a new genetic optimization algorithm in conjunction with constructing a differentiable surrogate model of the simulation using the samples that were already taken; our approach will enable us to efficiently optimize the simulation further and possibly perform gradient-based optimization.Surrogate This idea prompted us to examine the FNO's effectiveness at emulating the real simulation. Executing this task required us to take samples of the simulation. However, if we were to run a set of simulations with randomly or uniformly generated control parameters, then too many samples would crash; that is why we use genetic optimization first, to have a high quality data set.DECAF We propose DECAF (Differential Evolution [with] Covariance Adaptation [followed by optimizing a] Fourier [neural operator surrogate]), a novel solution. Our approach combines differential evolution with covariance matrix adaptation (Igel et al., 2007) to minimize undesirable artifacts while simultaneously behaving as an acquisition function to construct a data set of samples for the FNO surrogate. We then suggest ways to optimize the surrogate further to find a highly optimal set of parameters. We hypothesized that our approach would reduce the number of crashes through the course of sampling and create a high enough quality data set to train an FNO surrogate.Parameters Our control parameters were defined by the artificial coefficient of shear viscosity (c m ), bulk-viscosity dilatation (c b ), and thermal conductivity (c k ), as well as, ceiling values on artificial bulk viscosity (c l ), thermal conductivity (c c ), and diffusivity (c f ). These artificial coefficients are used to enhance numerical stability and fidelity by smoothing shock waves interacting with materials. Some choices of the artificial coefficients can render the simulation highly stiff, in which the time step size becomes drastically small. The minuscule time steps are often associated with non-physical numerical artifacts that can lead to instability in numerical simulations (Cook, 2007) . 
approach: 
 Differential evolution (DE) (Das & Suganthan, 2010 ) is a genetic optimization algorithm that optimizes non-convex loss landscapes. DE adheres to the following sequential process in Fig. 1 .Figure 1 : The sequential process of differential evolutionMutation Vanilla differential evolution (vanilla-DE) generates mutant vectors ⃗ m i ∈ M in the following way. Here, α m is the mutation factor and we set this value to 0.5. M and P define the mutant and current populations respectively.⃗ m i := ⃗ v 1 + α m • ( ⃗ v 2 -⃗ v 3 ) where ⃗ v 1 ̸ = ⃗ v 2 ̸ = ⃗ v 3 and ⃗ v 1 , ⃗ v 2 , ⃗ v 3 ∈ P (1)DE-GM Unlike vanilla differential evolution, our approach of combining the covariance matrix adaptation strategy with differential evolution in this way uses a multivariate Gaussian to generate mutant vectors. We call our approach "Differential Evolution with Gaussian Mutation (DE-GM)."EQUATIONX is a random variable constructed using the mean ⃗ µ P and diagonal covariance entries ⃗ σ 2 P of the entire current population.⃗ m i := X (3) This Gaussian is then sampled to construct a mutant population.Crossover Differential evolution uses crossover to retain some good genetic material from the previous generation. Crossover is performed by randomly swapping some elements of the mutant vectors with elements from the current population vectors to construct trial vectors ∈ T . The trial vectors are later compared to the current population vectors during the selection process.T i,j = M i,j rand(0, 1) ≥ α c P i,j otherwise (4)α c is the crossover factor, we set this value to 0.2.Selection After the test function or simulation has been evaluated, we iterate through the current and trial vectors with their corresponding losses. We select the next population (N ) in the following way.N i = T i loss(T i ) < loss(P i ) P i otherwise (5)Bounds We handled boundary conditions by replacing mutants that step outside the boundary with a random value within the defined boundary. For our simulation, each parameter was bound by its default value scaled up and down by a factor of 10.Convergence We eventually declared that the process has converged when the average loss of the population for the past n iterations is less than the convergence constant α v . When optimizing the test function, we used n = 10, α v = .001, and when optimizing the actual simulation, we were more forgiving and used n = 3, α v = 10.  
de-gm on simulation data: 
 We experimented with a hydrodynamic simulation which modeled the behavior of a water droplet that underwent a hypersonic shock. This particular simulation was modeled by the Navier-Stokes momentum equation (Cook, 2009, eqn. 2), the advection-diffusion equation (Cook, 2009, eqn. 9) , and the energy equation for multicomponent flows (Cook, 2009, eqn. 11) .Objective The optimization objective was to minimize the number of time steps needed to reach a fixed point in time-fewer time steps due to a bigger step size tend to yield fewer undesirable non-physical artifacts. We defined this fixed point in time to be 0.2 µs. The loss landscape is discrete, and the input space is continuous. We also accounted for trials where the choice of control parameters caused the simulation to fail to reach the specified point in time; we declare this phenomenon a 'crash.' When this happened, we assigned the largest possible error to it: 9999999. By doing this, the next population would likely not incorporate parameters that cause crashes. Our population size was four, and we ran six training sessions. The pressure of the third training session is illustrated in Fig. 4 . The first iteration (loss = 31146) showed some non-physical oscillations, potentially Gibbs oscillations (Gottlieb & Shu, 1997) .The last iteration, now optimized, had much fewer time steps (loss = 12793) and showed minimal visual evidence of undesirable non-physical artifacts. 
fourier neural operator: 
 One FNO layer update (Li et al., 2020; Kovachki et al., 2021) consists of passing input function v(x) through both a local linear operator as well as a kernel integral operator; the two are then summed together and passed through a non-linear activation function.EQUATIONF is the fast Fourier transform (FFT). σ is a non-linear activation function-such as ReLU. R θ is a transformation which contains the learnable parameters. ζ removes the higher Fourier modes. W is the local linear operator.Input (5th snapshot) Ground-truth (6th snapshot) FNO prediction (6th snapshot) Not only can the trained FNO be evaluated efficiently, but it is also differentiable with respect to the control parameters-meaning we can further optimize using gradient descent.By constructing a multilayer perceptron (MLP) to identify patterns indicating non-physical oscillations in the FNO output-which we know is possible according to the universal approximation theory (Hornik et al., 1989 )-then we can use the FNO to explore the loss landscape. 
summary: 
 DE-GM can effectively explore simulation settings while minimizing the number of heavily penalized samples and conserving computational resources. Our study found that an accurate FNO surrogate can be constructed from sparse samples in time evolution from the simulation taken with the DE-GM acquisition function. The approach studied here may also be used to optimize other types of numerical experiments in which users can flexibly define optimization objectives.Limitations and Broader Impacts There are no apparent negative societal impacts to this work. A limitation of our study is that training an MLP to approximate the loss from visual data was untested. 
 
 Appendix:  


Decision: Dear Authors,

Thank you for submitting the draft.

Both reviewers agree that the proposed methodology is potentially interesting. However, they do also raise some major points of concern. It is expected that authors will be addressing comments by the reviewers in the final draft. 

regards

AC

Reviews: review: title: title: The proposed approach is potentially of interest for calibrating and tuning numerical simulations across different research fields.
main_review: main_review: The authors propose a methodology using differential evolution combined with covariance matrix adaptation to optimize physical numerical simulations efficiently and then employ the samples collected to train a surrogate model, namely fourier neural operator (FNO). This surrogate may be used to enhance the optimization process by accelerating sampling. Although this approach was successfully applied to optimize a hydrodynamic numerical simulation, it has potential for application in other disciplines requiring physical simulation optimization.

The present study is potentially of interest for calibrating and tuning numerical simulations across different research fields, such as climate science. While the aim of the paper is clearly defined in the manuscript, the text would benefit from further specific details to better understand the results and their implications. The theoretical concepts are concisely presented. The figures and the experiments are in general sound, but the text can better highlight key points (Section 4.1). 
Specific comments:
- Section 2 – Approach. The mutation factor (αm) and the crossover factor (αc) seem to be arbitrarily chosen to 0.5 and 0.2, respectively. A motivation or strategy defining these factors is missing. Moreover, how sensitive are the results presented here to these factors. 
- Section 2 – Approach. How is the loss defined (selection)?
- Section 4 – CMA-DE on Simulation data. Why is the fixed point in time defined as 0.2 µs?  How do the population size and training sessions affect the results?
- Section 5 – FNO. Non-linear activation function “such as ReLU”, is this the function used? “Trained overnight” can be better defined, it is a bit vague.
- Limitations. To me this approach requires to a good extent expert knowledge (e.g., defining control parameters and their likely ranges, or “crash”).
title: title: Promising idea with unclear motivation and limited evaluation
main_review: main_review: **Summary**: The papers propose a method to optimize parameters of physics simulations using samples generated from a genetic optimization algorithm, which is then used to construct an FNO surrogate. The authors evaluate the performance of the FNO and propose further optimization that could be applied to this surrogate via gradient-based algorithms.


**Strengths**
1. Combining differential evolution as a precursor to surrogate models seems like an exciting direction. The authors evaluate their results on computationally intensive tasks such as hydrodynamic simulations.

**Areas of improvement**:

1. The idea of combining multiple techniques lacks clarity and motivation. The choice of CMA-DE and FNO does not seem convincing, as the manuscript lacks details about clear advantages compared with the existing state-of-the-art.
2. There is no evaluation of different sampling strategies or other optimization methods for surrogates, which may have provided better insights into their algorithm choices. There are no details about computational performance analysis.
3. For the CMA-DE and FNO - The CMA-DE method has previously been studied in the literature, but this work doesn't cite any previous publications and claims the technique as a novel [arabas2019, grady2023]. The role of FNO in the work is unclear, and the experiment details are missing.

[arabas2019] Arabas, Jarosław, and Dariusz Jagodziński. "Toward a matrix-free covariance matrix adaptation evolution strategy." IEEE Transactions on Evolutionary Computation 24.1 (2019): 84-98.

[grady2023] Grady, Thomas J., et al. "Model-parallel Fourier neural operators as learned surrogates for large-scale parametric PDEs." Computers & Geosciences (2023): 105402.