An Active Learning Performance Model for Parallel Bayesian Calibration of Expensive Simulations

Authors: özge  sürer
stefan m wild
Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-05 20:57:43

Abstract: Estimating parameters of simulation models based on observed data is an especially challenging task when the computational expense of the model -- necessitated by faithfully capturing the real system -- limits the learning process. When simulation models are expensive to evaluate, emulators are often built to efficiently approximate model outputs during model calibration. Computing-informed active learning, guided by intelligent acquisition functions, can improve data collection for emulators, thereby enhancing the calibration's efficiency. However, the performance of active learning strategies depends on computational factors such as computing environment (e.g., parallel resources available), tradeoffs in (calibration and simulation) algorithm's ability to benefit from parallelism, and the computational expense of the simulation models. In addition to overviewing these considerations, this work provides examples exemplifying the tradeoffs that make such learning difficult.

Main paper: 
 introduction: 
 Simulation models are widely used across engineering and science disciplines to explain complex systems. These models require calibration parameters as input to produce outputs that closely represent reality. However, calibration parameters are often unknown and must be inferred from data. The calibration process becomes particularly challenging when each simulation run is computationally expensive. Bayesian calibration (Sung and Tuo 2024) , which quantifies uncertainty in both parameters and predictions, offers a solution by using emulators such as Gaussian processes (GPs) (Rasmussen and Williams 2005, Gramacy 2020 ) to approximate a simulation model's behavior. Emulators, often built from carefully selected a priori designs, help facilitate the calibration process (see Santner et al. (2018) for a thorough review). However, traditional a priori design methods may not adequately explore the input space, especially in higher-dimensional parameter settings, potentially limiting the emulator's accuracy for calibration. Therefore, careful selection of experimental designs for simulation evaluations is crucial for precise calibration with limited simulation runs. This work explores active learning for sequential simulation data collection to build statistical emulators for calibration. In active learning, an acquisition function assesses the value of evaluating the simulation model at any given parameter. Parameters can be acquired sequentially, either one at a time (fully sequential) or in batches. While fully sequential procedures can yield more accurate, evaluation-efficient results given their use of all information obtained up to that point, batch sequential approaches may achieve high-quality solutions more quickly by evaluating multiple parameters simultaneously in parallel processing environments. However, the performance of active learning is influenced by various computational factors, including the convergence rate of the acquisition function (and its variation with batch size), the time complexity of the acquisition function with different batch and parallelism sizes, and the characteristics of the simulation model (e.g., its parameter dependence and run-time variability). In this work, we propose a performance model to understand the impact of these computational factors, ultimately guiding the selection of the best configuration in practice.In § 2 we present an acquisition function for calibration, which is used to gather experimental data to motivate our experiments in § 4. § 3 introduces the performance model to investigate the effect of computational factors. 
acquisition function for calibration: 
 We consider a simulation model η(•) that takes a parameter θ θ θ ∈ Θ ⊂ R p and returns an output η(θ θ θ), and a data generation mechanism yielding y = η(θ θ θ) + ϵ, ϵ ∼ N (0, σ 2 ). At each stage indexed by t, the simulation data D t = {(θ θ θ i , η(θ θ θ i )) : i = 1, . . . , n t }, collected sequentially, is stored to build a GP emulator. Using a GP emulator, η(θ θ θ)|D t at any θ θ θ follows a normal distribution with mean m t (θ θ θ) and variance s 2 t (θ θ θ). Calibration can have multiple objectives and different acquisition functions are tailored to achieve specific goals. For simplicity, we focus only on enhancing parameter inference by learning the posterior density. To this end, we use the expected integrated variance criterion (EIVAR) (Sürer et al. 2024 , Sürer 2024) , which selects parameters that minimize the aggregated uncertainty of the posterior p(θ θ θ|y). EIVAR at any θ θ θ * is computed asA(θ θ θ * ) = Θ E η(θ θ θ * )|Dt (V [p(θ θ θ|y)| (θ θ θ * , η(θ θ θ * )) ∪ D t ]) dθ θ θ.The derivation of EIVAR follows from Sürer et al. (2024) and can be approximated over a set of (e.g., uniformly distributed) reference parameters Θ ref asEQUATIONwhereτ 2 t (θ θ θ, θ θ θ * ) = cov t (θ θ θ, θ θ θ * ) 2 / s 2 t (θ θ θ * ) + υ t , cov t (θ θ θ, θ θ θ * ) = k t (θ θ θ, θ θ θ * ) -k t (θ θ θ) ⊤ K -1 t k t (θ θ θ * ), and f N evaluates a normal probability density. Here, υ t > 0 is the nugget parameter, k t (•, •) is the kernel function and k t (θ θ θ) consists of the cross-kernel evaluations between θ θ θ and θ θ θ i , for i = 1, . . . , n t . The matrix K t is an n t × n t matrix with (i, j)-th entry k t (θ θ θ i , θ θ θ j ) + υ t 1 i=j for 1 ≤ i, j ≤ n t . Both (Sürer et al. 2024 , Sürer 2024 ) provide a summary of the accuracy of EIVAR and other benchmark methods across different stages, operating under the assumption that the run time for each simulation evaluation is fixed and that acquisition times are negligible. In contrast, this work aims to illustrate how the performance of active learning methods is influenced by various computational factors, including variable run times and acquisition times, thereby providing a more comprehensive understanding of their effectiveness in a multitude of real-world scenarios.We assume access to w workers, which can run w simulation models concurrently, and an additional worker that generates b parameters with the acquisition function. If w = 1, we conduct data collection sequentially, one parameter at a time. For w > 1, updates are performed either synchronously or asynchronously, depending on the batch size b. When b = 1, we generate a set of candidate parameters from the prior and select the one that minimizes (1). For b > 1, rather than solving b × p-dimensional optimization problem, we minimize (1) b times over a candidate set of parameters. At each of the b iterations, we assume a constant value for the unknown simulation output of the selected parameter and then update the emulator. This approach is known as the constant liar strategy in Bayesian optimization (Ginsbourger et al. 2010) . In this context, the constant liar strategy serves to illustrate the proposed batch-sequential procedure. However, this strategy may introduce inaccuracies in predictions because it relies on fixed estimates for unseen parameters, which might not accurately represent the actual simulation outputs. Alternative approaches for batch acquisitions could be employed, each leading to different performance tradeoffs that affect both computational efficiency and the quality of the solutions obtained. 
performance model using active learning: 
 Algorithm 1 presents the pseudocode for a performance model utilizing the sequential data collection described in § 2. The algorithm collects n(b, α) simulation data points to achieve α-level calibration accuracy with batch size b (line 4). To conduct our numerical analysis, we record the time c J j (b, w) at which each job j ended, which is equal to the sum of the start time of job j and the random execution Algorithm 1: Performance model for parallel calibration 1 Input: batch size b, worker size w, number of data to be collected n(b, α) 2 Initialize t = 0; P t (w) = {1, . . . , w}; n t = w 3 c J j (b, w) = s j ∀j = 1, . . . , w; c S t (b, w) = 0 4 while n t < n(b, α) do 5 t ← t + 1; n t ← n t-1 ; P t (w) ← P t-1 (w) We store the end time c S t (b, w) for each stage t (line 6). The acquisition time a(b, t) depends on the stages and batch sizes. In our case, since the time required to build a new emulator increases with the growing size of the simulation data, a(b, t) increases in both batch size and time. The pending list P t (w) maintains the indices of jobs submitted for evaluation that have not yet completed (line 11). This list helps track ongoing simulations, allowing the system to manage and monitor which tasks are still in progress. At each stage, the job in the bth order of the pending list is the last one for which the simulation output was received from the workers. This performance model also allows one to employ problem characteristics (e.g., distributions of the number of simulations required, simulation run times, and acquisition times) to assess the wall time needed to obtain a specific accuracy, and thereby choose an acquisition and computing configuration. 
results and discussion: 
 We acquired 2560 parameters using EIVAR with the well-known Himmelblau function (Surjanovic and Bingham 2013) , experimenting with batch sizes b ∈ {1, 2, 4, 8, 16, 32, 64, 128, 256} and a worker size w = 256. Detailed information about the experimental setup is provided in Appendix A. The accuracy is assessed by the mean absolute deviation (MAD) between the true and estimated posteriors. In a sequential design, performance is typically summarized after receiving new evaluations, as illustrated in the left panel of Figure 1 . The performance degrades with an increasing batch size because more parameters are chosen without incorporating new information from the evaluations of other parameters in the same batch. The performance summary across stages (middle panel) can approximate the wall-clock time required for data collection, assuming that simulation run time is relatively constant and is the dominant computational cost. However, as acquisition times vary across stages and batch sizes (right panel), combining these factors with the computational expense of the simulation models may lead to significant variability in overall performance.To investigate this, we use the progress curves in the left panel of Figure 1 to provide n(b, α) as input to the performance model in Algorithm 1. We evaluate this model under varying simulation and acquisition times. We model random simulation run times using a truncated normal distribution. We apply a linear increase to the acquisition time, varying the slopes and intercepts. For each job j ∈ {1 + b(t -1) : t = 1, . . . , n/b}, the linear acquisition time ȃ + bj/n with ȃ = b is obtained. The α-level corresponds to the highest accuracy, defined as the lowest MAD, achieved with b = 256. For comparison, we determine n(b, α) for other batch sizes (e.g., n(b = 1, α) = 246). Figure 2 illustrates the wall-clock time required to achieve an α-level accuracy across different batch sizes and acquisition times. When acquisition times are relatively longer, larger batch sizes generally reduce the wall-clock time needed to achieve α-level accuracy, as smaller batch sizes require more frequent acquisitions. While smaller batch sizes require fewer simulation evaluations, the wall-clock time increases due to the total acquisition time. The best batch size varies with different acquisition and simulation times. For instance, in the right panel, b = 16 minimizes wall-clock time when the acquisition time is 2 2 , whereas b = 128 is the best in the left panel for the same acquisition time.A key benefit of our performance model is that it allows one to investigate various settings, including when the simulation run times are constant, have multiple modes, or exhibit other variability patterns. Figure 3 examines how variability in simulation run times influences the batch size for minimizing wall-clock time, considering different simulation and acquisition times. As variability increases (right panel), synchronous updates are beneficial only when acquisition times are significantly long. Smaller batch sizes perform better as the ratio of simulation times to acquisition time becomes larger. As a result, the best batch size achieves a balance between minimizing simulation evaluations and limiting acquisition overhead, depending on the relative scales of acquisition and simulation times. 
concluding thoughts: 
 What is the "best" active learning method? This paper shows that this depends on factors such as the computational environment (e.g., available parallelism, method scalability) and problem characteristics (e.g., simulation/acquisition times and their variability). A method that is best in one setting can readily be shown to perform poorly in another. Our proposed performance model is designed to help inform practitioners of the tradeoffs associated with candidate methods for their particular problem and setting. For instance, practitioners could test the candidate methods in a simulated environment using a similar but much less computationally expensive synthetic function with similar characteristics to the target problem. They could also take empirical data on past simulation run times, apply knowledge on the scalability of the approach, and use all of these in our performance model in order to determine actions on the production (very computationally expensive) run. The aim in doing so is to maintain solution quality while being as efficient as possible with computational resources and wall time goals.Future work could broaden the experimental evaluation by incorporating real-world simulations or more complex systems beyond the Himmelblau function. Testing on more intricate models or real calibration scenarios would enhance the robustness and generalizability of the findings, offering insight into the performance of active learning in a broad array of practical applications. Additionally, expanding the scope of acquisition strategies by exploring alternatives such as expected improvement or Thompson sampling, alongside the EIVAR criterion illustrated here, could provide a more thorough comparison. Such an exploration may reveal the potential advantages or limitations of each approach, improving our understanding of the effectiveness of each acquisition strategy across various calibration contexts. One limitation of the performance model is its reliance on the input number n(b, α) of data points to be collected. To address this, a predictive model could further be developed to estimate n(b, α) based on the characteristics of the simulation model, enhancing the performance model's usability and effectiveness. Many of these discussions are detailed in the extended paper by Sürer and Wild (2024) . 
 
 Appendix:  



Reviews: review: title: title: Peer review for An Active Learning Performance Model for Parallel Bayesian Calibration of Expensive Simulations
main_review: main_review: ## Summary
This paper addresses the resource challenge of calibrating simulation models. The authors propose a performance model for active learning in parallel Bayesian calibration. By using Gaussian process emulators and informed acquisition functions, they aim to optimize data collection and improve the efficiency of the calibration process. The paper discusses how computational factors influence the performance of active learning strategies and provides empirical results illustrating the trade-offs involved in selecting batch sizes and acquisition times.

## Big-picture comments
* **Originality**: the paper introduces the performance model to investigate the effect of computation factors, which is an original contribution. However, the paper might benefit from a further discussion on how this approach addresses the gap in prior literature, such as Sürer et al. 2024
* **Study set-up**: the technical execution of this paper as well as the proposed acquisition function and performance model are clearly defined. It might be beneficial to discuss the limitations of the set-up (e.g., constant liar strategy in Bayesian optimization setting is itself a tradeoff between computational efficiency and performance).
* **Result Discussion / Implication**: while the results of the experiment are well-documented and visualized, the paper could benefit from a deeper discussion on the real-world implications of this study and the directions that future research should take.
* **Significance**: the research aims to address a significant problem facing researchers and practitioners in simulations, hence the research carriers real-world implications.

## Questions
1. Is the assumption of constant simulation run times in the performance model justifiable (line 95)?
2. How does the proposed approach compare with other active learning strategies in terms of performance and computational efficiency?
3. What are the specific implications of the proposed performance models? Are there any limitations in this approach?
title: title: Review for An Active Learning Performance Model for Parallel Bayesian Calibration of Expensive Simulation
main_review: main_review: 1. Summary of the Paper
The paper proposes a performance model for active learning in the context of parallel Bayesian calibration of expensive simulations. It addresses the computational challenges in optimizing calibration efficiency using Gaussian Process (GP) emulators for high-cost simulation models. By introducing an acquisition function and a performance model, the authors analyze the trade-offs between batch size, parallelism, and acquisition time, demonstrating how these factors affect the performance of Bayesian calibration in practice.
2. Strengths
• The work tackles an important challenge in simulation-based learning where the cost of running simulations necessitates the use of efficient calibration techniques. The focus on parallelism and active learning makes the contribution particularly relevant in domains involving high-performance computing.
• The authors employ a well-structured set of experiments, including variations in batch sizes and worker configurations, to illustrate the effect of computational factors on calibration performance. The use of both synthetic functions (Himmelblau function) and performance metrics (mean absolute deviation) effectively validates the proposed model.
• The paper is well-organized, with a clear progression from introducing the calibration problem, formulating the acquisition function, and finally presenting the performance model and experiments. This logical structure aids the reader in understanding the flow of ideas.
3. Weaknesses
• While the empirical findings are compelling, the paper lacks deeper theoretical justifications for some of the observed phenomena. For instance, the impact of batch size on the efficiency of active learning is primarily discussed from an empirical perspective. A more rigorous theoretical exploration would strengthen the contribution.
• The work relies on a simulated function (Himmelblau) for the bulk of the experimental evaluation. Though effective for illustrating basic principles, it does not fully capture the complexity of real-world calibration tasks. Incorporating results from real-world simulations or more complex systems would make the findings more generalizable.
• The paper focuses solely on the EIVAR criterion for acquisition. While it is a reasonable choice, the inclusion of alternative acquisition functions, such as expected improvement or Thompson sampling, could provide a more comprehensive comparison and reveal potential advantages or limitations of the selected criterion.
4. Minor Comments
• Typographical errors are minimal, but a final proofreading pass is recommended.
• The paper would benefit from a stronger conclusion that ties together the empirical findings and provides more concrete recommendations for practitioners.